91 lines
2.9 KiB
Markdown
91 lines
2.9 KiB
Markdown
# DGR-016 API note: narrow llama.cpp hooks, no Meshnet policy
|
|
|
|
This note is the upstream-facing shape for the collaboration package.
|
|
|
|
## Goal
|
|
|
|
Keep the llama.cpp ask small:
|
|
|
|
- expose generic model-layer hooks that are useful to any local or remote
|
|
layer-worker setup;
|
|
- keep Meshnet routing, session ownership, billing, and relay transport out of
|
|
llama.cpp;
|
|
- preserve one patch per concern so the series rebases cleanly on the pinned
|
|
upstream commit.
|
|
|
|
## Concern 1: range-aware loading and authoritative tensor ownership
|
|
|
|
Requested surface:
|
|
|
|
- accept a contiguous `[start_layer, end_layer)` range;
|
|
- expose whether the worker owns embeddings, final norm, and final head;
|
|
- make the loaded range authoritative from the model state, not from CLI
|
|
claims;
|
|
- allow unowned tensors to be absent rather than fabricated.
|
|
|
|
Why this is upstreamable:
|
|
|
|
- it is generic loader and introspection plumbing;
|
|
- it helps any local partitioned inference mode;
|
|
- it does not require any Meshnet identity, route, or transport type.
|
|
|
|
Minimal examples/tests:
|
|
|
|
- `tests/test_gguf_ownership.py`
|
|
- `tests/test_llama_worker_build.py`
|
|
|
|
## Concern 2: architecture boundary input/output
|
|
|
|
Requested surface:
|
|
|
|
- accept a versioned boundary bundle carrying one or more named tensors;
|
|
- support an unnormalized residual stream as the intermediate handoff;
|
|
- keep final norm, LM head, and sampling on the tail shard only;
|
|
- keep the bundle format explicit about name, shape, dtype, byte order, and
|
|
fragments.
|
|
|
|
Why this is upstreamable:
|
|
|
|
- it matches both dense Llama and other certified adapter families;
|
|
- it does not assume Meshnet or any specific wire protocol;
|
|
- it gives a stable ABI for a layer-worker boundary.
|
|
|
|
Minimal examples/tests:
|
|
|
|
- `tests/test_boundary_adapter.py`
|
|
- `tests/test_native_shard_protocol.py`
|
|
|
|
## Concern 3: layer-filtered KV and session mapping
|
|
|
|
Requested surface:
|
|
|
|
- let the worker own KV only for its layer range;
|
|
- map a stable session/context identifier to the local sequence;
|
|
- allow cache miss, stale epoch, truncate, release, and eviction semantics;
|
|
- reject incompatible cache recipes rather than trying to heal them silently.
|
|
|
|
Why this is upstreamable:
|
|
|
|
- it is a local sequence/KV API, not a network scheduler;
|
|
- it is useful to any supervisor that needs one process per layer range;
|
|
- it keeps session semantics outside llama.cpp while still making the worker
|
|
stateful in a controlled way.
|
|
|
|
Minimal examples/tests:
|
|
|
|
- `tests/test_hot_kv_state.py`
|
|
- `tests/test_kv_cache_distributed.py`
|
|
|
|
## Suggested patch split
|
|
|
|
Keep the series narrow and independently reviewable against the exact pinned
|
|
commit `b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac`:
|
|
|
|
1. `range-aware-loading` and ownership introspection.
|
|
2. `boundary-input-output` and named tensor bundle handoff.
|
|
3. `layer-filtered-kv` and sequence ownership.
|
|
|
|
The current Meshnet worker scaffold remains a project-owned wrapper and is not
|
|
part of the upstream ask.
|
|
|