1.7 KiB
DGR-016 outreach draft
Subject: Narrow llama.cpp hooks for range loading, boundary I/O, and local KV ownership
Hi Georgi and llama.cpp maintainers,
We have been building a distributed GGUF route on top of a Meshnet control plane, and the narrow upstreamable seam is now clear enough to summarize.
We are not asking llama.cpp to own Meshnet routing, billing, relay transport, or any volunteer-network policy. The upstream ask is limited to generic local hooks that make partitioned inference easier to implement and easier to review:
- Range-aware loading and ownership introspection for contiguous layer ranges.
- Architecture-defined boundary input/output using an explicit named-tensor bundle.
- Layer-filtered KV ownership and stable local sequence mapping.
Why we think this is generally useful:
- Nakshatra already demonstrates the value of a narrow layer-worker seam and partial GGUF loading.
- prima.cpp shows the same idea from a different angle with selective loading, local KV, and boundary residual transport.
- Both projects suggest the same conclusion: the missing API is not Meshnet specific, it is a local runtime seam that any layer-partitioned supervisor can use.
The package we would upstream is intentionally split into one concern per patch so review stays small:
- range-aware loading and tensor ownership;
- boundary I/O for intermediate residual state;
- layer-filtered KV and sequence ownership.
If useful, we can send the concrete MRE/test mapping next. We already have deterministic examples covering the loader, boundary contract, and KV/session semantics in the Meshnet tree, and we can trim them into upstream-focused test cases.
Thanks, Meshnet maintainers