# Colibrì implementation audit

Research date: 2026-07-15. Primary source: [JustVugg/colibri](https://github.com/JustVugg/colibri) at `main` (README and linked source files). The repository is a model-specific runtime, not a wrapper around llama.cpp.

## Answer in one paragraph

Colibrì runs inference in a project-owned, dependency-free C engine (`c/glm.c` for GLM-5.2 and `c/olmoe.c` for OLMoE). Python is used for the one-time FP8/safetensors-to-Colibrì-container conversion and for the standard-library OpenAI HTTP gateway; it is not in the runtime inference path. The engine keeps dense/shared weights resident, while routed MoE experts are stored as individually addressable quantized records on disk and loaded into a per-layer LRU working set. RAM and optional VRAM are hot tiers; disk is a cold immutable backing store. This is local memory/storage tiering on one machine—not distributed expert execution over a network.

## What performs inference

- The README explicitly describes a “single C file (`c/glm.c`, ~2,400 lines)” with no BLAS, Python, or GPU requirement; runtime is pure C and Python is conversion-only ([README](https://github.com/JustVugg/colibri#the-idea), [runtime/setup section](https://github.com/JustVugg/colibri#quick-start)).
- The C source declares the GLM MoE forward path, MLA attention, sigmoid router, shared expert, and “expert routed in streaming dal disco (per-expert)” ([`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). It defines its own quantized tensor (`QT`) and expert-slot (`ESlot`) structures, rather than importing GGUF/llama.cpp data structures.
- Optional CUDA and Metal backends are native Colibrì backends. On Windows, CUDA is a separately built `coli_cuda.dll` loaded through `c/backend_loader.c`; the host falls back to CPU if it is absent ([README GPU section](https://github.com/JustVugg/colibri#windows-11-native-no-wsl), [`backend_loader.c`](https://github.com/JustVugg/colibri/blob/main/c/backend_loader.c), [`backend_cuda.cu`](https://github.com/JustVugg/colibri/blob/main/c/backend_cuda.cu)).
- `c/openai_server.py` is only an HTTP adapter. The README says inference remains in the same C engine and that one persistent process owns one mutable KV context ([API section](https://github.com/JustVugg/colibri#openai-compatible-api)).

## How experts are loaded on one laptop

The placement policy is a three-tier hierarchy:

1. Dense attention, embeddings, shared experts, and other always-used weights stay resident in RAM (roughly 9.9 GB int4 for the stated GLM-5.2 setup).
2. Routed experts are separate records (about 19 MB each at int4 in the README's GLM-5.2 example). A per-layer LRU cache holds the currently useful experts in RAM; an optional pinned hot store keeps frequently used experts from eviction. CUDA can make VRAM an additional hot tier.
3. Remaining experts stay on disk (about 370 GB in the stated int4 container). A token routes to top-k experts per MoE layer; cache misses issue bounded background reads, then the loaded records are multiplied before the layer completes.

The README quantifies the trade-off: 75 layers × 8 experts means approximately 11 GB of cold reads per token, and the reported cold rate is only 0.05–0.1 token/s on a ~1 GB/s disk ([expert layout](https://github.com/JustVugg/colibri#the-idea), [numbers/resource policy](https://github.com/JustVugg/colibri#honest-numbers), [resource policy](https://github.com/JustVugg/colibri#resource-policy)). `PILOT=1` predicts the next layer's routes (reported 71.6% top-8 recall) and prefetches them while the current layer computes ([README router-lookahead](https://github.com/JustVugg/colibri#resource-policy)). Prefill and MTP verification use “batch-union MoE”: each unique expert in a batch is read once and applied to all positions that selected it.

The learning cache persists expert-use counts in `.coli_usage`, pins hot experts at startup, and can periodically repin them using a session-local LFRU score. This is an adaptive placement policy, not a change to router semantics. The model directory is converted offline one source shard at a time; the original 756 GB FP8 checkpoint need not coexist on disk ([converter and warmup](https://github.com/JustVugg/colibri#quick-start), [cache policy](https://github.com/JustVugg/colibri#resource-policy)).

## Model format and scope

Colibrì does **not** consume GGUF. Its converter reads Hugging Face safetensors/config data and writes a Colibrì-specific quantized container/directory (the README calls it an “int4 container” and runs with `COLI_MODEL=/path/to/...`). The C loader and `QT`/`ESlot` types are custom to this repository ([converter](https://github.com/JustVugg/colibri/blob/main/c/tools/convert_fp8_to_int4.py), [`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)). Current fidelity is tied to `glm_moe_dsa` (GLM-5.2); OLMoE has a separate implementation. This should be treated as an architectural experiment and source of techniques, not as a drop-in GGUF backend.

## What is and is not distributed

There is no peer protocol, tensor RPC, layer hand-off, remote expert service, or multi-host scheduler in the repository. `coli serve` serializes requests through a local process (bounded FIFO queue; optional isolated KV slots), and the README explicitly says concurrent requests queue because the engine owns mutable KV state ([queue/KV section](https://github.com/JustVugg/colibri#openai-compatible-api)). The “distributed-looking” behavior is storage-tier streaming inside one address space: disk I/O overlaps compute, but every expert matmul and the KV state remain on the same laptop.

## Ideas worth carrying into Meshnet

1. **Expert-level placement, not only layer-level placement.** For MoE models, advertise and assign individual expert records (or expert groups) independently from dense/layer shards. A node can contribute capacity for hot experts without owning the whole model.
2. **Immutable cold backing + bounded hot cache.** Treat the model artifact as a content-addressed, immutable source; keep a bounded LRU/LFRU cache of resident experts. Placement changes then become cache promotion/eviction rather than model mutation.
3. **Router-aware prefetch.** Add an optional next-seam prefetch hint after layer L predicts likely expert IDs for layer L+1. Hints must be advisory and cancellable; correctness still waits for the router's actual top-k.
4. **Batch-union requests.** During prefill or verification, deduplicate expert IDs across tokens so one transfer serves many positions. This maps naturally to a Meshnet seam batch message.
5. **Persisted usage heat.** Track expert hit/miss/latency histograms and use them for placement recommendations. Keep this separate from billing/reputation and avoid treating historical heat as a correctness signal.
6. **Explicit cold-path telemetry.** Report disk/network service time separately from foreground-visible wait. Colibrì's profile distinguishes overlap; Meshnet should expose the same distinction per activation seam.
7. **Resource planning as a first-class contract.** `coli plan`/`doctor` produce a versioned placement/budget report before loading. Meshnet admission could use an equivalent plan: dense footprint, expert cache budget, KV reserve, bandwidth, and safe concurrency.

## Follow-up: distributed expert routing

### The transferable idea

For an MoE layer, the node that owns and executes that layer's router can select
the token or batch's top-*k* experts, dispatch the same layer input to the
providers that own those experts, then gather and weighted-sum the returned
expert outputs before continuing with the next layer. This is **expert
parallelism**. It is not a responsibility of the route's initial/head node:
every MoE layer has its own router and therefore makes its own selection.

```text
activation reaches MoE layer L
        |
        v
L's Shard computes attention + router scores
        |
        v
top-k expert IDs -> expert-provider groups
        |
        v
scatter inputs -> run expert(s) -> gather weighted outputs
        |
        v
complete layer L and continue the Inference Route
```

Colibrì proves the useful local analogue: experts are independently addressable
quantized records; its router selects them at execution time; a bounded
RAM/VRAM cache, pinning, and read-ahead decide whether a selected expert comes
from fast memory or its cold disk backing. It does **not** perform the
networked version: all expert execution and KV state remain local to one
process ([Colibrì README: expert layout](https://github.com/JustVugg/colibri#the-idea),
[Colibrì README: server/KV model](https://github.com/JustVugg/colibri#openai-compatible-api),
[`c/glm.c`](https://github.com/JustVugg/colibri/blob/main/c/glm.c)).

### Why this is not the first public-network primitive

Naively making every individual expert independently reachable over a WAN
would cause a scatter/gather at every MoE layer for every decode step. The
Colibrì GLM-5.2 example has 75 MoE layers and selects eight routed experts per
layer; that illustrates the potential fan-out, even though Colibrì satisfies
those selections locally ([Colibrì README: expert layout and cold-path
numbers](https://github.com/JustVugg/colibri#the-idea)). Network latency,
tail-provider delay, failure/retry behavior, and per-expert accounting would
become part of the autoregressive critical path.

This reinforces ADR-0024's current choice: public Inference Routes use
contiguous layer/pipeline Shards; tensor and expert parallelism are deferred to
trusted composite providers or managed clusters, where the network is
low-latency and one provider can own the collective's operational contract
([ADR-0024: distributed parallelism](../adr/0024-distributed-gguf-runtime.md)).

### Safe staged adoption

1. **Local tiered experts inside a contiguous MoE Shard.** Keep a Shard's
   expert execution local, but apply Colibrì-style immutable cold storage,
   bounded LRU/LFRU caches, hot-expert pinning, batch-union loading, and
   router-aware prefetch.
2. **Expert routing within one trusted composite provider.** Let a managed
   LAN/cluster expose a single Meshnet provider identity while it handles
   expert scatter/gather internally. This is the earliest setting where the
   technique should be benchmarked end-to-end.
3. **Public remote expert providers only behind a release gate.** If measured
   performance warrants it, expose versioned remote *expert packs* rather than
   unconstrained per-expert endpoints. The owning MoE-layer Shard must retain
   control of selection and aggregation.

The public form would require all of the following before it can be routable:

- content-addressed artifact, quantization, architecture, and runtime-recipe
  identity for every expert pack;
- stable ownership, replication, cache residency, and health reports;
- a versioned scatter/gather protocol carrying layer ID, expert IDs, route
  session/epoch, token positions, inputs, weights, deadlines, and cancellation;
- batch-union deduplication by provider, bounded fan-out, backpressure, and
  straggler/failure policy;
- separate telemetry for cache hit/miss, transfer bytes, overlap, remote
  service time, tail latency, and aggregation time; and
- proof that the resulting output, KV isolation, and admission behavior match
  the certified whole-model/contiguous-Shard execution.

The strategy is therefore to borrow Colibrì's **expert-as-movable-artifact and
memory-tiering** idea, while preserving Meshnet's Route Session ownership and
contiguous public layer Shards. Its local cache should be an optimization below
our existing activation seam, not a replacement for the control plane.

## Important limitations for our design

- Colibrì's cold path is local NVMe. Network expert fetches add latency, loss, authentication, retries, and Byzantine-data concerns that the project does not solve.
- One mutable KV context and one-at-a-time generation are deliberate constraints; Meshnet needs explicit Route Session/KV ownership and seam backpressure for concurrent users.
- Router lookahead is model-specific and only experimentally measured. It cannot be assumed for arbitrary MoE architectures.
- The custom container and hand-written kernels maximize control but increase maintenance and validation burden. Reusing llama.cpp/GGML remains attractive for a general GGUF lane; Colibrì's expert-cache and planning ideas can sit above that substrate.

## Source index

- Repository/README: <https://github.com/JustVugg/colibri>
- GLM engine and custom tensor/expert structures: <https://github.com/JustVugg/colibri/blob/main/c/glm.c>
- OLMoE engine: <https://github.com/JustVugg/colibri/blob/main/c/olmoe.c>
- FP8→Colibrì int4 converter: <https://github.com/JustVugg/colibri/blob/main/c/tools/convert_fp8_to_int4.py>
- Optional CUDA backend/loader: <https://github.com/JustVugg/colibri/tree/main/c>
- Local OpenAI gateway: <https://github.com/JustVugg/colibri/blob/main/c/openai_server.py>
- Placement planning/doctor implementation: <https://github.com/JustVugg/colibri/blob/main/c/resource_plan.py> and <https://github.com/JustVugg/colibri/blob/main/c/doctor.py>
