diff --git a/docs/research/colibri-implementation-audit.md b/docs/research/colibri-implementation-audit.md new file mode 100644 index 0000000..4c09f3f --- /dev/null +++ b/docs/research/colibri-implementation-audit.md @@ -0,0 +1,148 @@ +# Colibrì implementation audit + +Research date: 2026-07-15. Primary source: [JustVugg/colibri](https://github.com/JustVugg/colibri) at `main` (README and linked source files). The repository is a model-specific runtime, not a wrapper around llama.cpp. + +## Answer in one paragraph + +Colibrì runs inference in a project-owned, dependency-free C engine (`c/glm.c` for GLM-5.2 and `c/olmoe.c` for OLMoE). Python is used for the one-time FP8/safetensors-to-Colibrì-container conversion and for the standard-library OpenAI HTTP gateway; it is not in the runtime inference path. The engine keeps dense/shared weights resident, while routed MoE experts are stored as individually addressable quantized records on disk and loaded into a per-layer LRU working set. RAM and optional VRAM are hot tiers; disk is a cold immutable backing store. This is local memory/storage tiering on one machine—not distributed expert execution over a network. + +## What performs inference + +- The README explicitly describes a “single C file (`c/glm.c`, ~2,400 lines)” with no BLAS, Python, or GPU requirement; runtime is pure C and Python is conversion-only ([README](https://github.com/JustVugg/colibri#the-idea), [runtime/setup section](https://github.com/JustVugg/colibri#quick-start)). +- The C source declares the GLM MoE forward path, MLA attention, sigmoid router, shared expert, and “expert routed in streaming dal disco (per-expert)” ([`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). It defines its own quantized tensor (`QT`) and expert-slot (`ESlot`) structures, rather than importing GGUF/llama.cpp data structures. +- Optional CUDA and Metal backends are native Colibrì backends. On Windows, CUDA is a separately built `coli_cuda.dll` loaded through `c/backend_loader.c`; the host falls back to CPU if it is absent ([README GPU section](https://github.com/JustVugg/colibri#windows-11-native-no-wsl), [`backend_loader.c`](https://github.com/JustVugg/colibri/blob/main/c/backend_loader.c), [`backend_cuda.cu`](https://github.com/JustVugg/colibri/blob/main/c/backend_cuda.cu)). +- `c/openai_server.py` is only an HTTP adapter. The README says inference remains in the same C engine and that one persistent process owns one mutable KV context ([API section](https://github.com/JustVugg/colibri#openai-compatible-api)). + +## How experts are loaded on one laptop + +The placement policy is a three-tier hierarchy: + +1. Dense attention, embeddings, shared experts, and other always-used weights stay resident in RAM (roughly 9.9 GB int4 for the stated GLM-5.2 setup). +2. Routed experts are separate records (about 19 MB each at int4 in the README's GLM-5.2 example). A per-layer LRU cache holds the currently useful experts in RAM; an optional pinned hot store keeps frequently used experts from eviction. CUDA can make VRAM an additional hot tier. +3. Remaining experts stay on disk (about 370 GB in the stated int4 container). A token routes to top-k experts per MoE layer; cache misses issue bounded background reads, then the loaded records are multiplied before the layer completes. + +The README quantifies the trade-off: 75 layers × 8 experts means approximately 11 GB of cold reads per token, and the reported cold rate is only 0.05–0.1 token/s on a ~1 GB/s disk ([expert layout](https://github.com/JustVugg/colibri#the-idea), [numbers/resource policy](https://github.com/JustVugg/colibri#honest-numbers), [resource policy](https://github.com/JustVugg/colibri#resource-policy)). `PILOT=1` predicts the next layer's routes (reported 71.6% top-8 recall) and prefetches them while the current layer computes ([README router-lookahead](https://github.com/JustVugg/colibri#resource-policy)). Prefill and MTP verification use “batch-union MoE”: each unique expert in a batch is read once and applied to all positions that selected it. + +The learning cache persists expert-use counts in `.coli_usage`, pins hot experts at startup, and can periodically repin them using a session-local LFRU score. This is an adaptive placement policy, not a change to router semantics. The model directory is converted offline one source shard at a time; the original 756 GB FP8 checkpoint need not coexist on disk ([converter and warmup](https://github.com/JustVugg/colibri#quick-start), [cache policy](https://github.com/JustVugg/colibri#resource-policy)). + +## Model format and scope + +Colibrì does **not** consume GGUF. Its converter reads Hugging Face safetensors/config data and writes a Colibrì-specific quantized container/directory (the README calls it an “int4 container” and runs with `COLI_MODEL=/path/to/...`). The C loader and `QT`/`ESlot` types are custom to this repository ([converter](https://github.com/JustVugg/colibri/blob/main/c/tools/convert_fp8_to_int4.py), [`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). Current fidelity is tied to `glm_moe_dsa` (GLM-5.2); OLMoE has a separate implementation. This should be treated as an architectural experiment and source of techniques, not as a drop-in GGUF backend. + +## What is and is not distributed + +There is no peer protocol, tensor RPC, layer hand-off, remote expert service, or multi-host scheduler in the repository. `coli serve` serializes requests through a local process (bounded FIFO queue; optional isolated KV slots), and the README explicitly says concurrent requests queue because the engine owns mutable KV state ([queue/KV section](https://github.com/JustVugg/colibri#openai-compatible-api)). The “distributed-looking” behavior is storage-tier streaming inside one address space: disk I/O overlaps compute, but every expert matmul and the KV state remain on the same laptop. + +## Ideas worth carrying into Meshnet + +1. **Expert-level placement, not only layer-level placement.** For MoE models, advertise and assign individual expert records (or expert groups) independently from dense/layer shards. A node can contribute capacity for hot experts without owning the whole model. +2. **Immutable cold backing + bounded hot cache.** Treat the model artifact as a content-addressed, immutable source; keep a bounded LRU/LFRU cache of resident experts. Placement changes then become cache promotion/eviction rather than model mutation. +3. **Router-aware prefetch.** Add an optional next-seam prefetch hint after layer L predicts likely expert IDs for layer L+1. Hints must be advisory and cancellable; correctness still waits for the router's actual top-k. +4. **Batch-union requests.** During prefill or verification, deduplicate expert IDs across tokens so one transfer serves many positions. This maps naturally to a Meshnet seam batch message. +5. **Persisted usage heat.** Track expert hit/miss/latency histograms and use them for placement recommendations. Keep this separate from billing/reputation and avoid treating historical heat as a correctness signal. +6. **Explicit cold-path telemetry.** Report disk/network service time separately from foreground-visible wait. Colibrì's profile distinguishes overlap; Meshnet should expose the same distinction per activation seam. +7. **Resource planning as a first-class contract.** `coli plan`/`doctor` produce a versioned placement/budget report before loading. Meshnet admission could use an equivalent plan: dense footprint, expert cache budget, KV reserve, bandwidth, and safe concurrency. + +## Follow-up: distributed expert routing + +### The transferable idea + +For an MoE layer, the node that owns and executes that layer's router can select +the token or batch's top-*k* experts, dispatch the same layer input to the +providers that own those experts, then gather and weighted-sum the returned +expert outputs before continuing with the next layer. This is **expert +parallelism**. It is not a responsibility of the route's initial/head node: +every MoE layer has its own router and therefore makes its own selection. + +```text +activation reaches MoE layer L + | + v +L's Shard computes attention + router scores + | + v +top-k expert IDs -> expert-provider groups + | + v +scatter inputs -> run expert(s) -> gather weighted outputs + | + v +complete layer L and continue the Inference Route +``` + +Colibrì proves the useful local analogue: experts are independently addressable +quantized records; its router selects them at execution time; a bounded +RAM/VRAM cache, pinning, and read-ahead decide whether a selected expert comes +from fast memory or its cold disk backing. It does **not** perform the +networked version: all expert execution and KV state remain local to one +process ([Colibrì README: expert layout](https://github.com/JustVugg/colibri#the-idea), +[Colibrì README: server/KV model](https://github.com/JustVugg/colibri#openai-compatible-api), +[`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). + +### Why this is not the first public-network primitive + +Naively making every individual expert independently reachable over a WAN +would cause a scatter/gather at every MoE layer for every decode step. The +Colibrì GLM-5.2 example has 75 MoE layers and selects eight routed experts per +layer; that illustrates the potential fan-out, even though Colibrì satisfies +those selections locally ([Colibrì README: expert layout and cold-path +numbers](https://github.com/JustVugg/colibri#the-idea)). Network latency, +tail-provider delay, failure/retry behavior, and per-expert accounting would +become part of the autoregressive critical path. + +This reinforces ADR-0024's current choice: public Inference Routes use +contiguous layer/pipeline Shards; tensor and expert parallelism are deferred to +trusted composite providers or managed clusters, where the network is +low-latency and one provider can own the collective's operational contract +([ADR-0024: distributed parallelism](../adr/0024-distributed-gguf-runtime.md)). + +### Safe staged adoption + +1. **Local tiered experts inside a contiguous MoE Shard.** Keep a Shard's + expert execution local, but apply Colibrì-style immutable cold storage, + bounded LRU/LFRU caches, hot-expert pinning, batch-union loading, and + router-aware prefetch. +2. **Expert routing within one trusted composite provider.** Let a managed + LAN/cluster expose a single Meshnet provider identity while it handles + expert scatter/gather internally. This is the earliest setting where the + technique should be benchmarked end-to-end. +3. **Public remote expert providers only behind a release gate.** If measured + performance warrants it, expose versioned remote *expert packs* rather than + unconstrained per-expert endpoints. The owning MoE-layer Shard must retain + control of selection and aggregation. + +The public form would require all of the following before it can be routable: + +- content-addressed artifact, quantization, architecture, and runtime-recipe + identity for every expert pack; +- stable ownership, replication, cache residency, and health reports; +- a versioned scatter/gather protocol carrying layer ID, expert IDs, route + session/epoch, token positions, inputs, weights, deadlines, and cancellation; +- batch-union deduplication by provider, bounded fan-out, backpressure, and + straggler/failure policy; +- separate telemetry for cache hit/miss, transfer bytes, overlap, remote + service time, tail latency, and aggregation time; and +- proof that the resulting output, KV isolation, and admission behavior match + the certified whole-model/contiguous-Shard execution. + +The strategy is therefore to borrow Colibrì's **expert-as-movable-artifact and +memory-tiering** idea, while preserving Meshnet's Route Session ownership and +contiguous public layer Shards. Its local cache should be an optimization below +our existing activation seam, not a replacement for the control plane. + +## Important limitations for our design + +- Colibrì's cold path is local NVMe. Network expert fetches add latency, loss, authentication, retries, and Byzantine-data concerns that the project does not solve. +- One mutable KV context and one-at-a-time generation are deliberate constraints; Meshnet needs explicit Route Session/KV ownership and seam backpressure for concurrent users. +- Router lookahead is model-specific and only experimentally measured. It cannot be assumed for arbitrary MoE architectures. +- The custom container and hand-written kernels maximize control but increase maintenance and validation burden. Reusing llama.cpp/GGML remains attractive for a general GGUF lane; Colibrì's expert-cache and planning ideas can sit above that substrate. + +## Source index + +- Repository/README: +- GLM engine and custom tensor/expert structures: +- OLMoE engine: +- FP8→Colibrì int4 converter: +- Optional CUDA backend/loader: +- Local OpenAI gateway: +- Placement planning/doctor implementation: and