COLIBRI RESEARCH

2026-07-16 16:22:58 +02:00
parent 254627629b
commit 737bade989
1 changed files with 148 additions and 0 deletions
--- a/docs/research/colibri-implementation-audit.md
+++ b/docs/research/colibri-implementation-audit.md
@@ -0,0 +1,148 @@
+# Colibrì implementation audit
+
+Research date: 2026-07-15. Primary source: [JustVugg/colibri](https://github.com/JustVugg/colibri) at `main` (README and linked source files). The repository is a model-specific runtime, not a wrapper around llama.cpp.
+
+## Answer in one paragraph
+
+Colibrì runs inference in a project-owned, dependency-free C engine (`c/glm.c` for GLM-5.2 and `c/olmoe.c` for OLMoE). Python is used for the one-time FP8/safetensors-to-Colibrì-container conversion and for the standard-library OpenAI HTTP gateway; it is not in the runtime inference path. The engine keeps dense/shared weights resident, while routed MoE experts are stored as individually addressable quantized records on disk and loaded into a per-layer LRU working set. RAM and optional VRAM are hot tiers; disk is a cold immutable backing store. This is local memory/storage tiering on one machine—not distributed expert execution over a network.
+
+## What performs inference
+
+- The README explicitly describes a “single C file (`c/glm.c`, ~2,400 lines)” with no BLAS, Python, or GPU requirement; runtime is pure C and Python is conversion-only ([README](https://github.com/JustVugg/colibri#the-idea), [runtime/setup section](https://github.com/JustVugg/colibri#quick-start)).
+- The C source declares the GLM MoE forward path, MLA attention, sigmoid router, shared expert, and “expert routed in streaming dal disco (per-expert)” ([`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). It defines its own quantized tensor (`QT`) and expert-slot (`ESlot`) structures, rather than importing GGUF/llama.cpp data structures.
+- Optional CUDA and Metal backends are native Colibrì backends. On Windows, CUDA is a separately built `coli_cuda.dll` loaded through `c/backend_loader.c`; the host falls back to CPU if it is absent ([README GPU section](https://github.com/JustVugg/colibri#windows-11-native-no-wsl), [`backend_loader.c`](https://github.com/JustVugg/colibri/blob/main/c/backend_loader.c), [`backend_cuda.cu`](https://github.com/JustVugg/colibri/blob/main/c/backend_cuda.cu)).
+- `c/openai_server.py` is only an HTTP adapter. The README says inference remains in the same C engine and that one persistent process owns one mutable KV context ([API section](https://github.com/JustVugg/colibri#openai-compatible-api)).
+
+## How experts are loaded on one laptop
+
+The placement policy is a three-tier hierarchy:
+
+1. Dense attention, embeddings, shared experts, and other always-used weights stay resident in RAM (roughly 9.9 GB int4 for the stated GLM-5.2 setup).
+2. Routed experts are separate records (about 19 MB each at int4 in the README's GLM-5.2 example). A per-layer LRU cache holds the currently useful experts in RAM; an optional pinned hot store keeps frequently used experts from eviction. CUDA can make VRAM an additional hot tier.
+3. Remaining experts stay on disk (about 370 GB in the stated int4 container). A token routes to top-k experts per MoE layer; cache misses issue bounded background reads, then the loaded records are multiplied before the layer completes.
+
+The README quantifies the trade-off: 75 layers × 8 experts means approximately 11 GB of cold reads per token, and the reported cold rate is only 0.05–0.1 token/s on a ~1 GB/s disk ([expert layout](https://github.com/JustVugg/colibri#the-idea), [numbers/resource policy](https://github.com/JustVugg/colibri#honest-numbers), [resource policy](https://github.com/JustVugg/colibri#resource-policy)). `PILOT=1` predicts the next layer's routes (reported 71.6% top-8 recall) and prefetches them while the current layer computes ([README router-lookahead](https://github.com/JustVugg/colibri#resource-policy)). Prefill and MTP verification use “batch-union MoE”: each unique expert in a batch is read once and applied to all positions that selected it.
+
+The learning cache persists expert-use counts in `.coli_usage`, pins hot experts at startup, and can periodically repin them using a session-local LFRU score. This is an adaptive placement policy, not a change to router semantics. The model directory is converted offline one source shard at a time; the original 756 GB FP8 checkpoint need not coexist on disk ([converter and warmup](https://github.com/JustVugg/colibri#quick-start), [cache policy](https://github.com/JustVugg/colibri#resource-policy)).
+
+## Model format and scope
+
+Colibrì does **not** consume GGUF. Its converter reads Hugging Face safetensors/config data and writes a Colibrì-specific quantized container/directory (the README calls it an “int4 container” and runs with `COLI_MODEL=/path/to/...`). The C loader and `QT`/`ESlot` types are custom to this repository ([converter](https://github.com/JustVugg/colibri/blob/main/c/tools/convert_fp8_to_int4.py), [`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). Current fidelity is tied to `glm_moe_dsa` (GLM-5.2); OLMoE has a separate implementation. This should be treated as an architectural experiment and source of techniques, not as a drop-in GGUF backend.
+
+## What is and is not distributed
+
+There is no peer protocol, tensor RPC, layer hand-off, remote expert service, or multi-host scheduler in the repository. `coli serve` serializes requests through a local process (bounded FIFO queue; optional isolated KV slots), and the README explicitly says concurrent requests queue because the engine owns mutable KV state ([queue/KV section](https://github.com/JustVugg/colibri#openai-compatible-api)). The “distributed-looking” behavior is storage-tier streaming inside one address space: disk I/O overlaps compute, but every expert matmul and the KV state remain on the same laptop.
+
+## Ideas worth carrying into Meshnet
+
+1. **Expert-level placement, not only layer-level placement.** For MoE models, advertise and assign individual expert records (or expert groups) independently from dense/layer shards. A node can contribute capacity for hot experts without owning the whole model.
+2. **Immutable cold backing + bounded hot cache.** Treat the model artifact as a content-addressed, immutable source; keep a bounded LRU/LFRU cache of resident experts. Placement changes then become cache promotion/eviction rather than model mutation.
+3. **Router-aware prefetch.** Add an optional next-seam prefetch hint after layer L predicts likely expert IDs for layer L+1. Hints must be advisory and cancellable; correctness still waits for the router's actual top-k.
+4. **Batch-union requests.** During prefill or verification, deduplicate expert IDs across tokens so one transfer serves many positions. This maps naturally to a Meshnet seam batch message.
+5. **Persisted usage heat.** Track expert hit/miss/latency histograms and use them for placement recommendations. Keep this separate from billing/reputation and avoid treating historical heat as a correctness signal.
+6. **Explicit cold-path telemetry.** Report disk/network service time separately from foreground-visible wait. Colibrì's profile distinguishes overlap; Meshnet should expose the same distinction per activation seam.
+7. **Resource planning as a first-class contract.** `coli plan`/`doctor` produce a versioned placement/budget report before loading. Meshnet admission could use an equivalent plan: dense footprint, expert cache budget, KV reserve, bandwidth, and safe concurrency.
+
+## Follow-up: distributed expert routing
+
+### The transferable idea
+
+For an MoE layer, the node that owns and executes that layer's router can select
+the token or batch's top-*k* experts, dispatch the same layer input to the
+providers that own those experts, then gather and weighted-sum the returned
+expert outputs before continuing with the next layer. This is **expert
+parallelism**. It is not a responsibility of the route's initial/head node:
+every MoE layer has its own router and therefore makes its own selection.
+
+```text
+activation reaches MoE layer L
+        |
+        v
+L's Shard computes attention + router scores
+        |
+        v
+top-k expert IDs -> expert-provider groups
+        |
+        v
+scatter inputs -> run expert(s) -> gather weighted outputs
+        |
+        v
+complete layer L and continue the Inference Route
+```
+
+Colibrì proves the useful local analogue: experts are independently addressable
+quantized records; its router selects them at execution time; a bounded
+RAM/VRAM cache, pinning, and read-ahead decide whether a selected expert comes
+from fast memory or its cold disk backing. It does **not** perform the
+networked version: all expert execution and KV state remain local to one
+process ([Colibrì README: expert layout](https://github.com/JustVugg/colibri#the-idea),
+[Colibrì README: server/KV model](https://github.com/JustVugg/colibri#openai-compatible-api),
+[`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)).
+
+### Why this is not the first public-network primitive
+
+Naively making every individual expert independently reachable over a WAN
+would cause a scatter/gather at every MoE layer for every decode step. The
+Colibrì GLM-5.2 example has 75 MoE layers and selects eight routed experts per
+layer; that illustrates the potential fan-out, even though Colibrì satisfies
+those selections locally ([Colibrì README: expert layout and cold-path
+numbers](https://github.com/JustVugg/colibri#the-idea)). Network latency,
+tail-provider delay, failure/retry behavior, and per-expert accounting would
+become part of the autoregressive critical path.
+
+This reinforces ADR-0024's current choice: public Inference Routes use
+contiguous layer/pipeline Shards; tensor and expert parallelism are deferred to
+trusted composite providers or managed clusters, where the network is
+low-latency and one provider can own the collective's operational contract
+([ADR-0024: distributed parallelism](../adr/0024-distributed-gguf-runtime.md)).
+
+### Safe staged adoption
+
+1. **Local tiered experts inside a contiguous MoE Shard.** Keep a Shard's
+   expert execution local, but apply Colibrì-style immutable cold storage,
+   bounded LRU/LFRU caches, hot-expert pinning, batch-union loading, and
+   router-aware prefetch.
+2. **Expert routing within one trusted composite provider.** Let a managed
+   LAN/cluster expose a single Meshnet provider identity while it handles
+   expert scatter/gather internally. This is the earliest setting where the
+   technique should be benchmarked end-to-end.
+3. **Public remote expert providers only behind a release gate.** If measured
+   performance warrants it, expose versioned remote *expert packs* rather than
+   unconstrained per-expert endpoints. The owning MoE-layer Shard must retain
+   control of selection and aggregation.
+
+The public form would require all of the following before it can be routable:
+
+- content-addressed artifact, quantization, architecture, and runtime-recipe
+  identity for every expert pack;
+- stable ownership, replication, cache residency, and health reports;
+- a versioned scatter/gather protocol carrying layer ID, expert IDs, route
+  session/epoch, token positions, inputs, weights, deadlines, and cancellation;
+- batch-union deduplication by provider, bounded fan-out, backpressure, and
+  straggler/failure policy;
+- separate telemetry for cache hit/miss, transfer bytes, overlap, remote
+  service time, tail latency, and aggregation time; and
+- proof that the resulting output, KV isolation, and admission behavior match
+  the certified whole-model/contiguous-Shard execution.
+
+The strategy is therefore to borrow Colibrì's **expert-as-movable-artifact and
+memory-tiering** idea, while preserving Meshnet's Route Session ownership and
+contiguous public layer Shards. Its local cache should be an optimization below
+our existing activation seam, not a replacement for the control plane.
+
+## Important limitations for our design
+
+- Colibrì's cold path is local NVMe. Network expert fetches add latency, loss, authentication, retries, and Byzantine-data concerns that the project does not solve.
+- One mutable KV context and one-at-a-time generation are deliberate constraints; Meshnet needs explicit Route Session/KV ownership and seam backpressure for concurrent users.
+- Router lookahead is model-specific and only experimentally measured. It cannot be assumed for arbitrary MoE architectures.
+- The custom container and hand-written kernels maximize control but increase maintenance and validation burden. Reusing llama.cpp/GGML remains attractive for a general GGUF lane; Colibrì's expert-cache and planning ideas can sit above that substrate.
+
+## Source index
+
+- Repository/README: <https://github.com/JustVugg/colibri>
+- GLM engine and custom tensor/expert structures: <https://github.com/JustVugg/colibri/blob/main/c/glm.c>
+- OLMoE engine: <https://github.com/JustVugg/colibri/blob/main/c/olmoe.c>
+- FP8→Colibrì int4 converter: <https://github.com/JustVugg/colibri/blob/main/c/tools/convert_fp8_to_int4.py>
+- Optional CUDA backend/loader: <https://github.com/JustVugg/colibri/tree/main/c>
+- Local OpenAI gateway: <https://github.com/JustVugg/colibri/blob/main/c/openai_server.py>
+- Placement planning/doctor implementation: <https://github.com/JustVugg/colibri/blob/main/c/resource_plan.py> and <https://github.com/JustVugg/colibri/blob/main/c/doctor.py>