44 lines
1.7 KiB
Markdown
44 lines
1.7 KiB
Markdown
# DGR-016 outreach draft
|
|
|
|
Subject: Narrow llama.cpp hooks for range loading, boundary I/O, and local KV ownership
|
|
|
|
Hi Georgi and llama.cpp maintainers,
|
|
|
|
We have been building a distributed GGUF route on top of a Meshnet control
|
|
plane, and the narrow upstreamable seam is now clear enough to summarize.
|
|
|
|
We are not asking llama.cpp to own Meshnet routing, billing, relay transport,
|
|
or any volunteer-network policy. The upstream ask is limited to generic local
|
|
hooks that make partitioned inference easier to implement and easier to review:
|
|
|
|
1. Range-aware loading and ownership introspection for contiguous layer ranges.
|
|
2. Architecture-defined boundary input/output using an explicit named-tensor
|
|
bundle.
|
|
3. Layer-filtered KV ownership and stable local sequence mapping.
|
|
|
|
Why we think this is generally useful:
|
|
|
|
- Nakshatra already demonstrates the value of a narrow layer-worker seam and
|
|
partial GGUF loading.
|
|
- prima.cpp shows the same idea from a different angle with selective loading,
|
|
local KV, and boundary residual transport.
|
|
- Both projects suggest the same conclusion: the missing API is not Meshnet
|
|
specific, it is a local runtime seam that any layer-partitioned supervisor can
|
|
use.
|
|
|
|
The package we would upstream is intentionally split into one concern per patch
|
|
so review stays small:
|
|
|
|
- range-aware loading and tensor ownership;
|
|
- boundary I/O for intermediate residual state;
|
|
- layer-filtered KV and sequence ownership.
|
|
|
|
If useful, we can send the concrete MRE/test mapping next. We already have
|
|
deterministic examples covering the loader, boundary contract, and KV/session
|
|
semantics in the Meshnet tree, and we can trim them into upstream-focused test
|
|
cases.
|
|
|
|
Thanks,
|
|
Meshnet maintainers
|
|
|