neuron-tai/.scratch/distributed-gguf-runtime/evidence/DGR-016/api-note.md

DGR-016 API note: narrow llama.cpp hooks, no Meshnet policy

This note is the upstream-facing shape for the collaboration package.

Goal

Keep the llama.cpp ask small:

• expose generic model-layer hooks that are useful to any local or remote layer-worker setup;
• keep Meshnet routing, session ownership, billing, and relay transport out of llama.cpp;
• preserve one patch per concern so the series rebases cleanly on the pinned upstream commit.

Concern 1: range-aware loading and authoritative tensor ownership

Requested surface:

• accept a contiguous [start_layer, end_layer) range;
• expose whether the worker owns embeddings, final norm, and final head;
• make the loaded range authoritative from the model state, not from CLI claims;
• allow unowned tensors to be absent rather than fabricated.

Why this is upstreamable:

• it is generic loader and introspection plumbing;
• it helps any local partitioned inference mode;
• it does not require any Meshnet identity, route, or transport type.

Minimal examples/tests:

• tests/test_gguf_ownership.py
• tests/test_llama_worker_build.py

Concern 2: architecture boundary input/output

Requested surface:

• accept a versioned boundary bundle carrying one or more named tensors;
• support an unnormalized residual stream as the intermediate handoff;
• keep final norm, LM head, and sampling on the tail shard only;
• keep the bundle format explicit about name, shape, dtype, byte order, and fragments.

Why this is upstreamable:

• it matches both dense Llama and other certified adapter families;
• it does not assume Meshnet or any specific wire protocol;
• it gives a stable ABI for a layer-worker boundary.

Minimal examples/tests:

• tests/test_boundary_adapter.py
• tests/test_native_shard_protocol.py

Concern 3: layer-filtered KV and session mapping

Requested surface:

• let the worker own KV only for its layer range;
• map a stable session/context identifier to the local sequence;
• allow cache miss, stale epoch, truncate, release, and eviction semantics;
• reject incompatible cache recipes rather than trying to heal them silently.

Why this is upstreamable:

• it is a local sequence/KV API, not a network scheduler;
• it is useful to any supervisor that needs one process per layer range;
• it keeps session semantics outside llama.cpp while still making the worker stateful in a controlled way.

Minimal examples/tests:

• tests/test_hot_kv_state.py
• tests/test_kv_cache_distributed.py

Suggested patch split

Keep the series narrow and independently reviewable against the exact pinned commit b3c9d1b846cc80a6360adb6aeaa4fcd8c4c8dcac:

1. range-aware-loading and ownership introspection.
2. boundary-input-output and named tensor bundle handoff.
3. layer-filtered-kv and sequence ownership.

The current Meshnet worker scaffold remains a project-owned wrapper and is not part of the upstream ask.