neuron-tai/.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md

# Ralph execution context: Performant Concurrent Distributed GGUF Runtime

Status: authoritative context for every fresh Ralph iteration
Last updated: 2026-07-13

## Mandatory startup sequence

Before changing code, every Ralph agent must:

1. Read this file completely.
2. Read the selected issue under `.scratch/distributed-gguf-runtime/issues/`.
3. Read `docs/adr/0024-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
4. Read `.claude/memory/MEMORY.md` and root `CONTEXT.md` for current project vocabulary and constraints.
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
6. Read the evidence/handoff directories for every declared dependency.
7. Inspect `git status` and preserve all pre-existing working-tree changes.

A fresh Ralph iteration has no conversational memory. These files are the context contract.

## Story sizing and interruption rule

Each story is intended to fit one focused Ralph context. Before implementation, estimate whether every acceptance criterion can be completed and verified in the current iteration.

If the story is too large, an external dependency is unavailable, or the context/provider limit prevents completion:

- Do not weaken criteria.
- Do not mark the issue done or set `passes: true`.
- Avoid leaving an unverified cross-cutting partial implementation when a smaller safe spike is possible.
- Write `evidence/<TASK-ID>/DECOMPOSITION.md` or `BLOCKED.md` with the exact blocker, current verified state, proposed child stories, dependency graph and rollback/continuation instructions.
- Stop for supervised review.

If interrupted after code changes, record every changed file, command result and unresolved invariant so the next fresh loop can verify rather than guess.

## Product objective

Build performant, concurrent distributed inference that combines consumer machines to serve top open models that exceed one node's RAM/VRAM.

A distributed demo is not success. The product must provide:

- Useful measured prefill and decode speed.
- Multiple concurrent Route Sessions.
- No KV/token cross-talk.
- Bounded memory, queues, cancellation and failures.
- Real execution on every participating node.
- A model-fit or performance advantage over the current Transformers/safetensors route.

## Critical-path architecture

```text
Existing Meshnet control plane
        |
Versioned Protobuf over gRPC/HTTP2
        |
Project-owned standalone C++ Shard worker
        |
Small exact-commit llama.cpp patch stack
```

Meshnet remains the only control plane and owns:

- Tracker registration, Coverage Map, route selection and route epochs.
- Route Sessions and Activation Seams.
- Direct/relay routing.
- Capability admission.
- Cancellation, Generation Telemetry and backpressure.
- Billing, validation and per-node work attribution.

Do not introduce another scheduler/control plane from vLLM, Nakshatra, prima.cpp, llama-gguf, GPUStack or another project.

## Runtime decisions that are not open

1. Public-network Shards are contiguous transformer layer ranges.
2. llama.cpp/GGML is the native GGUF execution substrate.
3. The project owns a small standalone worker and a narrow pinned llama.cpp patch stack.
4. The native Shard protocol is Protocol Buffers over gRPC/HTTP2.
5. One long-lived bidirectional stream serves one Route Session Activation Seam.
6. The public activation boundary is a versioned named-tensor bundle.
7. Hot KV State remains local to the node serving the Shard.
8. `(Route Session ID, route epoch)` maps to an isolated llama sequence or bounded context.
9. Concurrency uses continuous batching of compatible active sessions inside each node.
10. Transformers/safetensors remains the correctness and performance baseline.
11. vLLM may be an optional complete managed provider and concept donor; it is not forked into public Shards.
12. Tensor/expert collectives are deferred to a trusted composite provider, not public WAN routes.
13. Unsupported architectures/backends remain registered-but-dark until real certification passes.
14. Alpha failure retries from token zero; unverified KV is never migrated silently.
15. Model artifacts must remain on mounted-drive storage and never under `/home`.

Changing one of these requires an explicit ADR update and human review, not an incidental story implementation.

## Performance discipline

GGUF performance is a hypothesis. Never write “GGUF is faster” without measurements.

DGR-001 locks controlled benchmark lanes and thresholds. DGR-014 enforces the final distributed comparison.

Always distinguish:

- Weight quantization from activation/compute/KV dtype.
- Runtime/kernel gains from quantization/model-fit gains.
- Single-request latency from aggregate concurrency throughput.
- Synthetic unit coverage from real distributed acceptance.

Required metrics where applicable:

```text
TTFT
prefill tokens/sec
decode tokens/sec
aggregate throughput
p50/p95 latency
seam bytes and latency
queue and batch occupancy
RSS and VRAM
KV pressure
output-quality drift
failures and cleanup
```

Do not weaken or move performance thresholds after seeing implementation results.

## Transport discipline

Do not invent a raw TCP protocol, new WebSocket protocol, QUIC layer or bespoke binary control format.

The `.proto` schema is the semantic contract. Direct transport uses gRPC. Existing relay infrastructure may carry the same serialized protobuf frames as opaque binary.

Protocol requirements:

- Schema/version negotiation.
- Request/work ID.
- Route Session ID and route epoch.
- Exact Model Artifact/runtime recipe fingerprint.
- Shard range and effective overlap-safe start.
- Prefill/decode/release/cancel phases.
- Position/token range and idempotency step.
- Named tensors with shape, dtype, byte order and bounded fragments.
- Compression/checksum.
- Cache expectation/result.
- Deadlines, cancellation, flow control and structured status.

Avoid per-token channel creation and unbounded unary payloads. Generated code and build tooling must be reproducible; do not require manual copying.

## Native runtime discipline

Reuse llama.cpp for GGUF, mmap, kernels, architecture graphs, tokenizer, KV, sequences and heterogeneous backends.

The project patch stack is limited to:

- Range-aware tensor registration/loading.
- Endpoint-specific embedding/final head ownership.
- Architecture-defined intermediate input/output.
- Intermediate output before final norm/head.
- Layer-filtered KV and session mapping.

Do not place Meshnet routing, transport, billing or authentication inside llama.cpp. Keep patches numbered, scoped, pinned and upstreamable.

Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate adapter after the dense release gate. Do not generalize through unchecked tensor-name substitutions.

## Existing code seams to inspect first

- `packages/node/meshnet_node/model_backend.py` — backend abstraction.
- `packages/node/meshnet_node/torch_server.py` — reference ranged execution and session behavior.
- `packages/node/meshnet_node/activation_compression.py` — current activation framing/compression.
- `packages/node/meshnet_node/route_session_benchmark.py` — existing benchmark infrastructure.
- `packages/tracker/meshnet_tracker/server.py` — registration, route and proxy behavior.
- `packages/tracker/meshnet_tracker/capability.py` — fail-closed capability admission.
- `tests/test_real_model_backend.py` — real backend coverage.
- `tests/test_tracker_routing.py` — route/session behavior.
- `tests/test_tracker_capability_admission.py` — recipe admission.
- `tests/test_route_session_benchmark.py` and `tests/test_manual_route_benchmark.py` — benchmark patterns.
- `docs/adr/0008-binary-activation-wire-format.md` — existing wire compatibility.
- `docs/adr/0012-start-layer-overlapping-shards.md` — effective start semantics.
- `docs/adr/0022-sharded-per-node-kv-cache.md` — Hot KV State contract.
- `docs/adr/0023-model-agnostic-node-capability-admission.md` — certification/admission.

Do not edit generated `build/`, `__pycache__`, egg-info, Ralph logs or unrelated scratch features.

## Planned source layout

Use these paths unless current code inspection proves a better project-consistent location. If changed, document the reason in task evidence.

```text
packages/node/native/
  proto/shard_runtime.proto
  cmake/
  llama/
    UPSTREAM_COMMIT
    patches/
  gguf_worker/
  tests/

packages/node/meshnet_node/
  native_protocol/
  gguf_backend.py
  runtime_recipe.py

.scratch/distributed-gguf-runtime/evidence/<TASK-ID>/
  README.md
  commands.txt
  results.json or other machine-readable evidence
```

Generated protobuf/C++ build outputs belong in build directories unless packaging explicitly requires checked-in generated Python modules. The story must document the generation command and version.

## Story output map

| Story | Required durable outputs |
|---|---|
| DGR-001 | benchmark harness/tests; `evidence/DGR-001/performance-contract.json`; raw/summary benchmark evidence |
| DGR-002 | `packages/node/native/proto/shard_runtime.proto`; reproducible Python/C++ generation/build wiring; protocol round-trip/compatibility tests; `evidence/DGR-002/` |
| DGR-003 | exact runtime-recipe/fingerprint implementation and admission tests; `evidence/DGR-003/` |
| DGR-004 | exact upstream pin, numbered patch series, reproducible fetch/apply/build smoke; `evidence/DGR-004/` |
| DGR-005 | dense-Llama range ownership loader and memory evidence; `evidence/DGR-005/` |
| DGR-006 | architecture boundary adapter/parity tests and results; `evidence/DGR-006/` |
| DGR-007 | concurrent session/KV manager, isolation/cleanup tests; `evidence/DGR-007/` |
| DGR-008 | standalone C++ gRPC worker, fake-model integration tests, lifecycle evidence; `evidence/DGR-008/` |
| DGR-009 | Meshnet backend/registration/relay integration and tests; `evidence/DGR-009/` |
| DGR-010 | real local two-process commands, raw metrics and parity report; `evidence/DGR-010/` |
| DGR-011 | two-machine configuration, commands, hardware/network manifest and raw results; `evidence/DGR-011/` |
| DGR-012 | continuous scheduler/admission implementation and 1/2/4/8 concurrency report; `evidence/DGR-012/` |
| DGR-013 | failure/cancel/restart test matrix and resource-cleanup evidence; `evidence/DGR-013/` |
| DGR-014 | immutable final comparison against DGR-001 thresholds and ship/stop recommendation; `evidence/DGR-014/` |
| DGR-015 | Qwen3-family adapter, architecture-specific parity/admission/performance evidence; `evidence/DGR-015/` |
| DGR-016 | narrow upstream patches/tests, design note and human-ready outreach package; `evidence/DGR-016/` |

## Dependency handoff rule

For every dependency listed by Ralph:

1. Confirm its `passes` state in `prd.json`.
2. Read `.scratch/distributed-gguf-runtime/evidence/<DEPENDENCY-ID>/README.md`.
3. Verify referenced source paths and commands still exist.
4. Do not repeat completed work unless verification exposes a concrete defect.
5. If dependency evidence is missing or contradictory, stop and repair the dependency instead of guessing.

## Testing and hardware rules

Default tests must be deterministic, GPU-free, model-download-free and API-credit-free.

Real model tests require:

```text
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1
```

On this machine:

- Use `.venv-rocm` for real Radeon 8060S ROCm execution.
- The default Python 3.14 `.venv` is unsuitable for real ROCm inference.
- Resolve model storage through the machine-specific `.env.<hostname>` configuration.
- Never download model artifacts under `/home`.
- Real acceptance must exercise actual Tracker-routed CPU/GPU computation; synthetic workers are only unit tests.

Record exact:

- Model/revision and Artifact hash.
- Quantization and runtime recipe.
- Host/hardware/backend/driver.
- Commands and environment names without secrets.
- Raw output and metrics.
- Whether the evidence is synthetic, local-real, or multi-machine-real.

## Worktree and commit discipline

This repository may contain pre-existing changes from research or another feature.

- Inspect `git status` before editing.
- Never reset, checkout over, stash, delete or reformat unrelated changes.
- Stage only files belonging to the selected story.
- Exclude `.ralph-tui`, iteration logs, caches, generated builds, FUSE artifacts and unrelated scratch work.
- Keep one scoped commit per completed story when the supervising loop requests commits.
- Do not modify `passes` for another story.

## Mandatory finish/handoff sequence

Before emitting `<promise>COMPLETE</promise>`:

1. Verify every acceptance criterion with real command output or file evidence.
2. Run story-specific gates and repository quality gates.
3. Write `.scratch/distributed-gguf-runtime/evidence/<TASK-ID>/README.md` containing:
   - Summary of changes.
   - Exact files changed.
   - Commands run and their real results.
   - Performance/correctness evidence.
   - Known limitations and deferred work.
   - Compatibility or migration notes.
   - Clear handoff for dependent stories.
4. Save machine-readable evidence beside it when the story produces metrics or schemas.
5. Update the source issue status to `done` only after all gates pass.
6. Preserve failures honestly. Never fabricate model, benchmark, test or hardware output.

## Authoritative references

Active decisions:

- `.scratch/distributed-gguf-runtime/README.md`
- `.scratch/distributed-gguf-runtime/implementation-strategy.md`
- `.scratch/distributed-gguf-runtime/architecture.md`
- `docs/adr/0024-distributed-gguf-runtime.md`
- `.scratch/distributed-gguf-runtime/PRD.md`
- `.scratch/distributed-gguf-runtime/prd.json`

Source research:

- `docs/research/distributed-gguf-landscape.md`
- `docs/research/distributed-gguf-github-followup.md`
- `docs/research/vllm-distributed-gguf-assessment.md`

If historical notes conflict with these files, the active decisions above win.