149 lines
13 KiB
Markdown
149 lines
13 KiB
Markdown
# Colibrì implementation audit
|
||
|
||
Research date: 2026-07-15. Primary source: [JustVugg/colibri](https://github.com/JustVugg/colibri) at `main` (README and linked source files). The repository is a model-specific runtime, not a wrapper around llama.cpp.
|
||
|
||
## Answer in one paragraph
|
||
|
||
Colibrì runs inference in a project-owned, dependency-free C engine (`c/glm.c` for GLM-5.2 and `c/olmoe.c` for OLMoE). Python is used for the one-time FP8/safetensors-to-Colibrì-container conversion and for the standard-library OpenAI HTTP gateway; it is not in the runtime inference path. The engine keeps dense/shared weights resident, while routed MoE experts are stored as individually addressable quantized records on disk and loaded into a per-layer LRU working set. RAM and optional VRAM are hot tiers; disk is a cold immutable backing store. This is local memory/storage tiering on one machine—not distributed expert execution over a network.
|
||
|
||
## What performs inference
|
||
|
||
- The README explicitly describes a “single C file (`c/glm.c`, ~2,400 lines)” with no BLAS, Python, or GPU requirement; runtime is pure C and Python is conversion-only ([README](https://github.com/JustVugg/colibri#the-idea), [runtime/setup section](https://github.com/JustVugg/colibri#quick-start)).
|
||
- The C source declares the GLM MoE forward path, MLA attention, sigmoid router, shared expert, and “expert routed in streaming dal disco (per-expert)” ([`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). It defines its own quantized tensor (`QT`) and expert-slot (`ESlot`) structures, rather than importing GGUF/llama.cpp data structures.
|
||
- Optional CUDA and Metal backends are native Colibrì backends. On Windows, CUDA is a separately built `coli_cuda.dll` loaded through `c/backend_loader.c`; the host falls back to CPU if it is absent ([README GPU section](https://github.com/JustVugg/colibri#windows-11-native-no-wsl), [`backend_loader.c`](https://github.com/JustVugg/colibri/blob/main/c/backend_loader.c), [`backend_cuda.cu`](https://github.com/JustVugg/colibri/blob/main/c/backend_cuda.cu)).
|
||
- `c/openai_server.py` is only an HTTP adapter. The README says inference remains in the same C engine and that one persistent process owns one mutable KV context ([API section](https://github.com/JustVugg/colibri#openai-compatible-api)).
|
||
|
||
## How experts are loaded on one laptop
|
||
|
||
The placement policy is a three-tier hierarchy:
|
||
|
||
1. Dense attention, embeddings, shared experts, and other always-used weights stay resident in RAM (roughly 9.9 GB int4 for the stated GLM-5.2 setup).
|
||
2. Routed experts are separate records (about 19 MB each at int4 in the README's GLM-5.2 example). A per-layer LRU cache holds the currently useful experts in RAM; an optional pinned hot store keeps frequently used experts from eviction. CUDA can make VRAM an additional hot tier.
|
||
3. Remaining experts stay on disk (about 370 GB in the stated int4 container). A token routes to top-k experts per MoE layer; cache misses issue bounded background reads, then the loaded records are multiplied before the layer completes.
|
||
|
||
The README quantifies the trade-off: 75 layers × 8 experts means approximately 11 GB of cold reads per token, and the reported cold rate is only 0.05–0.1 token/s on a ~1 GB/s disk ([expert layout](https://github.com/JustVugg/colibri#the-idea), [numbers/resource policy](https://github.com/JustVugg/colibri#honest-numbers), [resource policy](https://github.com/JustVugg/colibri#resource-policy)). `PILOT=1` predicts the next layer's routes (reported 71.6% top-8 recall) and prefetches them while the current layer computes ([README router-lookahead](https://github.com/JustVugg/colibri#resource-policy)). Prefill and MTP verification use “batch-union MoE”: each unique expert in a batch is read once and applied to all positions that selected it.
|
||
|
||
The learning cache persists expert-use counts in `.coli_usage`, pins hot experts at startup, and can periodically repin them using a session-local LFRU score. This is an adaptive placement policy, not a change to router semantics. The model directory is converted offline one source shard at a time; the original 756 GB FP8 checkpoint need not coexist on disk ([converter and warmup](https://github.com/JustVugg/colibri#quick-start), [cache policy](https://github.com/JustVugg/colibri#resource-policy)).
|
||
|
||
## Model format and scope
|
||
|
||
Colibrì does **not** consume GGUF. Its converter reads Hugging Face safetensors/config data and writes a Colibrì-specific quantized container/directory (the README calls it an “int4 container” and runs with `COLI_MODEL=/path/to/...`). The C loader and `QT`/`ESlot` types are custom to this repository ([converter](https://github.com/JustVugg/colibri/blob/main/c/tools/convert_fp8_to_int4.py), [`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). Current fidelity is tied to `glm_moe_dsa` (GLM-5.2); OLMoE has a separate implementation. This should be treated as an architectural experiment and source of techniques, not as a drop-in GGUF backend.
|
||
|
||
## What is and is not distributed
|
||
|
||
There is no peer protocol, tensor RPC, layer hand-off, remote expert service, or multi-host scheduler in the repository. `coli serve` serializes requests through a local process (bounded FIFO queue; optional isolated KV slots), and the README explicitly says concurrent requests queue because the engine owns mutable KV state ([queue/KV section](https://github.com/JustVugg/colibri#openai-compatible-api)). The “distributed-looking” behavior is storage-tier streaming inside one address space: disk I/O overlaps compute, but every expert matmul and the KV state remain on the same laptop.
|
||
|
||
## Ideas worth carrying into Meshnet
|
||
|
||
1. **Expert-level placement, not only layer-level placement.** For MoE models, advertise and assign individual expert records (or expert groups) independently from dense/layer shards. A node can contribute capacity for hot experts without owning the whole model.
|
||
2. **Immutable cold backing + bounded hot cache.** Treat the model artifact as a content-addressed, immutable source; keep a bounded LRU/LFRU cache of resident experts. Placement changes then become cache promotion/eviction rather than model mutation.
|
||
3. **Router-aware prefetch.** Add an optional next-seam prefetch hint after layer L predicts likely expert IDs for layer L+1. Hints must be advisory and cancellable; correctness still waits for the router's actual top-k.
|
||
4. **Batch-union requests.** During prefill or verification, deduplicate expert IDs across tokens so one transfer serves many positions. This maps naturally to a Meshnet seam batch message.
|
||
5. **Persisted usage heat.** Track expert hit/miss/latency histograms and use them for placement recommendations. Keep this separate from billing/reputation and avoid treating historical heat as a correctness signal.
|
||
6. **Explicit cold-path telemetry.** Report disk/network service time separately from foreground-visible wait. Colibrì's profile distinguishes overlap; Meshnet should expose the same distinction per activation seam.
|
||
7. **Resource planning as a first-class contract.** `coli plan`/`doctor` produce a versioned placement/budget report before loading. Meshnet admission could use an equivalent plan: dense footprint, expert cache budget, KV reserve, bandwidth, and safe concurrency.
|
||
|
||
## Follow-up: distributed expert routing
|
||
|
||
### The transferable idea
|
||
|
||
For an MoE layer, the node that owns and executes that layer's router can select
|
||
the token or batch's top-*k* experts, dispatch the same layer input to the
|
||
providers that own those experts, then gather and weighted-sum the returned
|
||
expert outputs before continuing with the next layer. This is **expert
|
||
parallelism**. It is not a responsibility of the route's initial/head node:
|
||
every MoE layer has its own router and therefore makes its own selection.
|
||
|
||
```text
|
||
activation reaches MoE layer L
|
||
|
|
||
v
|
||
L's Shard computes attention + router scores
|
||
|
|
||
v
|
||
top-k expert IDs -> expert-provider groups
|
||
|
|
||
v
|
||
scatter inputs -> run expert(s) -> gather weighted outputs
|
||
|
|
||
v
|
||
complete layer L and continue the Inference Route
|
||
```
|
||
|
||
Colibrì proves the useful local analogue: experts are independently addressable
|
||
quantized records; its router selects them at execution time; a bounded
|
||
RAM/VRAM cache, pinning, and read-ahead decide whether a selected expert comes
|
||
from fast memory or its cold disk backing. It does **not** perform the
|
||
networked version: all expert execution and KV state remain local to one
|
||
process ([Colibrì README: expert layout](https://github.com/JustVugg/colibri#the-idea),
|
||
[Colibrì README: server/KV model](https://github.com/JustVugg/colibri#openai-compatible-api),
|
||
[`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)).
|
||
|
||
### Why this is not the first public-network primitive
|
||
|
||
Naively making every individual expert independently reachable over a WAN
|
||
would cause a scatter/gather at every MoE layer for every decode step. The
|
||
Colibrì GLM-5.2 example has 75 MoE layers and selects eight routed experts per
|
||
layer; that illustrates the potential fan-out, even though Colibrì satisfies
|
||
those selections locally ([Colibrì README: expert layout and cold-path
|
||
numbers](https://github.com/JustVugg/colibri#the-idea)). Network latency,
|
||
tail-provider delay, failure/retry behavior, and per-expert accounting would
|
||
become part of the autoregressive critical path.
|
||
|
||
This reinforces ADR-0024's current choice: public Inference Routes use
|
||
contiguous layer/pipeline Shards; tensor and expert parallelism are deferred to
|
||
trusted composite providers or managed clusters, where the network is
|
||
low-latency and one provider can own the collective's operational contract
|
||
([ADR-0024: distributed parallelism](../adr/0024-distributed-gguf-runtime.md)).
|
||
|
||
### Safe staged adoption
|
||
|
||
1. **Local tiered experts inside a contiguous MoE Shard.** Keep a Shard's
|
||
expert execution local, but apply Colibrì-style immutable cold storage,
|
||
bounded LRU/LFRU caches, hot-expert pinning, batch-union loading, and
|
||
router-aware prefetch.
|
||
2. **Expert routing within one trusted composite provider.** Let a managed
|
||
LAN/cluster expose a single Meshnet provider identity while it handles
|
||
expert scatter/gather internally. This is the earliest setting where the
|
||
technique should be benchmarked end-to-end.
|
||
3. **Public remote expert providers only behind a release gate.** If measured
|
||
performance warrants it, expose versioned remote *expert packs* rather than
|
||
unconstrained per-expert endpoints. The owning MoE-layer Shard must retain
|
||
control of selection and aggregation.
|
||
|
||
The public form would require all of the following before it can be routable:
|
||
|
||
- content-addressed artifact, quantization, architecture, and runtime-recipe
|
||
identity for every expert pack;
|
||
- stable ownership, replication, cache residency, and health reports;
|
||
- a versioned scatter/gather protocol carrying layer ID, expert IDs, route
|
||
session/epoch, token positions, inputs, weights, deadlines, and cancellation;
|
||
- batch-union deduplication by provider, bounded fan-out, backpressure, and
|
||
straggler/failure policy;
|
||
- separate telemetry for cache hit/miss, transfer bytes, overlap, remote
|
||
service time, tail latency, and aggregation time; and
|
||
- proof that the resulting output, KV isolation, and admission behavior match
|
||
the certified whole-model/contiguous-Shard execution.
|
||
|
||
The strategy is therefore to borrow Colibrì's **expert-as-movable-artifact and
|
||
memory-tiering** idea, while preserving Meshnet's Route Session ownership and
|
||
contiguous public layer Shards. Its local cache should be an optimization below
|
||
our existing activation seam, not a replacement for the control plane.
|
||
|
||
## Important limitations for our design
|
||
|
||
- Colibrì's cold path is local NVMe. Network expert fetches add latency, loss, authentication, retries, and Byzantine-data concerns that the project does not solve.
|
||
- One mutable KV context and one-at-a-time generation are deliberate constraints; Meshnet needs explicit Route Session/KV ownership and seam backpressure for concurrent users.
|
||
- Router lookahead is model-specific and only experimentally measured. It cannot be assumed for arbitrary MoE architectures.
|
||
- The custom container and hand-written kernels maximize control but increase maintenance and validation burden. Reusing llama.cpp/GGML remains attractive for a general GGUF lane; Colibrì's expert-cache and planning ideas can sit above that substrate.
|
||
|
||
## Source index
|
||
|
||
- Repository/README: <https://github.com/JustVugg/colibri>
|
||
- GLM engine and custom tensor/expert structures: <https://github.com/JustVugg/colibri/blob/main/c/glm.c>
|
||
- OLMoE engine: <https://github.com/JustVugg/colibri/blob/main/c/olmoe.c>
|
||
- FP8→Colibrì int4 converter: <https://github.com/JustVugg/colibri/blob/main/c/tools/convert_fp8_to_int4.py>
|
||
- Optional CUDA backend/loader: <https://github.com/JustVugg/colibri/tree/main/c>
|
||
- Local OpenAI gateway: <https://github.com/JustVugg/colibri/blob/main/c/openai_server.py>
|
||
- Placement planning/doctor implementation: <https://github.com/JustVugg/colibri/blob/main/c/resource_plan.py> and <https://github.com/JustVugg/colibri/blob/main/c/doctor.py>
|