neuron-tai/.scratch/distributed-gguf-runtime/RALPH-CONTEXT.md

Ralph execution context: Performant Concurrent Distributed GGUF Runtime

Status: authoritative context for every fresh Ralph iteration Last updated: 2026-07-13

Mandatory startup sequence

Before changing code, every Ralph agent must:

1. Read this file completely.
2. Read the selected issue under .scratch/distributed-gguf-runtime/issues/.
3. Read .scratch/distributed-gguf-runtime/GLM-5.2-MAX-ALPHA-ROADMAP.md, .scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md, and the relevant part of architecture.md.
4. Read .claude/memory/MEMORY.md and root CONTEXT.md for current project vocabulary and constraints.
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
6. Read the evidence/handoff directories for every declared dependency.
7. Inspect git status and preserve all pre-existing working-tree changes.

A fresh Ralph iteration has no conversational memory. These files are the context contract.

Story sizing and interruption rule

Each story is intended to fit one focused Ralph context. Before implementation, estimate whether every acceptance criterion can be completed and verified in the current iteration.

If the story is too large, an external dependency is unavailable, or the context/provider limit prevents completion:

• Do not weaken criteria.
• Do not mark the issue done or set passes: true.
• Avoid leaving an unverified cross-cutting partial implementation when a smaller safe spike is possible.
• Write evidence/<TASK-ID>/DECOMPOSITION.md or BLOCKED.md with the exact blocker, current verified state, proposed child stories, dependency graph and rollback/continuation instructions.
• Stop for supervised review.

If interrupted after code changes, record every changed file, command result and unresolved invariant so the next fresh loop can verify rather than guess.

Product objective

Build performant, concurrent distributed inference that combines consumer machines to serve top open models that exceed one node's RAM/VRAM.

The alpha target is the exact pinned GLM-5.2 UD-IQ1_S artifact served with reasoning_effort=max across physical consumer machines. Dense Llama is a structural fixture. Synthetic workers, dense-attention compatibility fallback, a smaller model, or a single host cannot satisfy target alpha. The immutable target contract and resource envelope are in GLM-5.2-MAX-ALPHA-ROADMAP.md.

A distributed demo is not success. The product must provide:

• Useful measured prefill and decode speed.
• Multiple concurrent Route Sessions.
• No KV/token cross-talk.
• Bounded memory, queues, cancellation and failures.
• Real execution on every participating node.
• A model-fit or performance advantage over the current Transformers/safetensors route.

Critical-path architecture

Existing Meshnet control plane
        |
Versioned Protobuf over gRPC/HTTP2
        |
Project-owned standalone C++ Shard worker
        |
Small exact-commit llama.cpp patch stack

Meshnet remains the only control plane and owns:

• Tracker registration, Coverage Map, route selection and route epochs.
• Route Sessions and Activation Seams.
• Direct/relay routing.
• Capability admission.
• Cancellation, Generation Telemetry and backpressure.
• Billing, validation and per-node work attribution.

Do not introduce another scheduler/control plane from vLLM, Nakshatra, prima.cpp, llama-gguf, GPUStack or another project.

Runtime decisions that are not open

1. Public-network Shards are contiguous transformer layer ranges.
2. llama.cpp/GGML is the native GGUF execution substrate.
3. The project owns a small standalone worker and a narrow pinned llama.cpp patch stack.
4. The native Shard protocol is Protocol Buffers over gRPC/HTTP2.
5. One long-lived bidirectional stream serves one Route Session Activation Seam.
6. The public activation boundary is a versioned named-tensor bundle.
7. Hot KV State remains local to the node serving the Shard.
8. (Route Session ID, route epoch) maps to an isolated llama sequence or bounded context.
9. Concurrency uses continuous batching of compatible active sessions inside each node.
10. Transformers/safetensors remains the correctness and performance baseline.
11. vLLM may be an optional complete managed provider and concept donor; it is not forked into public Shards.
12. Tensor/expert collectives are deferred to a trusted composite provider, not public WAN routes.
13. Unsupported architectures/backends remain registered-but-dark until real certification passes.
14. Alpha failure retries from token zero; unverified KV is never migrated silently.
15. Model artifacts must remain on mounted-drive storage and never under /home.
16. Unified system RAM and integrated-GPU memory are one physical pool and must never be double-counted for admission.
17. Alpha requires native GLM-5.2 MoE, DSA, and IndexShare semantics; MTP/speculative decoding and 1M-context certification are post-alpha.

Changing one of these requires an explicit ADR update and human review, not an incidental story implementation.

Performance discipline

GGUF performance is a hypothesis. Never write “GGUF is faster” without measurements.

DGR-001 locks controlled benchmark lanes and thresholds. DGR-014 enforces the final distributed comparison.

Always distinguish:

• Weight quantization from activation/compute/KV dtype.
• Runtime/kernel gains from quantization/model-fit gains.
• Single-request latency from aggregate concurrency throughput.
• Synthetic unit coverage from real distributed acceptance.

Required metrics where applicable:

TTFT
prefill tokens/sec
decode tokens/sec
aggregate throughput
p50/p95 latency
seam bytes and latency
queue and batch occupancy
RSS and VRAM
KV pressure
output-quality drift
failures and cleanup

Do not weaken or move performance thresholds after seeing implementation results.

Transport discipline

Do not invent a raw TCP protocol, new WebSocket protocol, QUIC layer or bespoke binary control format.

The .proto schema is the semantic contract. Direct transport uses gRPC. Existing relay infrastructure may carry the same serialized protobuf frames as opaque binary.

Protocol requirements:

• Schema/version negotiation.
• Request/work ID.
• Route Session ID and route epoch.
• Exact Model Artifact/runtime recipe fingerprint.
• Shard range and effective overlap-safe start.
• Prefill/decode/release/cancel phases.
• Position/token range and idempotency step.
• Named tensors with shape, dtype, byte order and bounded fragments.
• Compression/checksum.
• Cache expectation/result.
• Deadlines, cancellation, flow control and structured status.

Avoid per-token channel creation and unbounded unary payloads. Generated code and build tooling must be reproducible; do not require manual copying.

Native runtime discipline

Reuse llama.cpp for GGUF, mmap, kernels, architecture graphs, tokenizer, KV, sequences and heterogeneous backends.

The project patch stack is limited to:

• Range-aware tensor registration/loading.
• Endpoint-specific embedding/final head ownership.
• Architecture-defined intermediate input/output.
• Intermediate output before final norm/head.
• Layer-filtered KV and session mapping.

Do not place Meshnet routing, transport, billing or authentication inside llama.cpp. Keep patches numbered, scoped, pinned and upstreamable.

Dense Llama is first only as a cheap range/boundary fixture. GLM-5.2 is the explicit product adapter and alpha target immediately afterward. Qwen3/Qwen3-MoE is post-alpha. Do not generalize through unchecked tensor-name substitutions.

Existing code seams to inspect first

• packages/node/meshnet_node/model_backend.py — backend abstraction.
• packages/node/meshnet_node/torch_server.py — reference ranged execution and session behavior.
• packages/node/meshnet_node/activation_compression.py — current activation framing/compression.
• packages/node/meshnet_node/route_session_benchmark.py — existing benchmark infrastructure.
• packages/tracker/meshnet_tracker/server.py — registration, route and proxy behavior.
• packages/tracker/meshnet_tracker/capability.py — fail-closed capability admission.
• tests/test_real_model_backend.py — real backend coverage.
• tests/test_tracker_routing.py — route/session behavior.
• tests/test_tracker_capability_admission.py — recipe admission.
• tests/test_route_session_benchmark.py and tests/test_manual_route_benchmark.py — benchmark patterns.
• docs/adr/0008-binary-activation-wire-format.md — existing wire compatibility.
• docs/adr/0012-start-layer-overlapping-shards.md — effective start semantics.
• docs/adr/0022-sharded-per-node-kv-cache.md — Hot KV State contract.
• docs/adr/0023-model-agnostic-node-capability-admission.md — certification/admission.

Do not edit generated build/, __pycache__, egg-info, Ralph logs or unrelated scratch features.

Planned source layout

Use these paths unless current code inspection proves a better project-consistent location. If changed, document the reason in task evidence.

packages/node/native/
  proto/shard_runtime.proto
  cmake/
  llama/
    UPSTREAM_COMMIT
    patches/
  gguf_worker/
  tests/

packages/node/meshnet_node/
  native_protocol/
  gguf_backend.py
  runtime_recipe.py

.scratch/distributed-gguf-runtime/evidence/<TASK-ID>/
  README.md
  commands.txt
  results.json or other machine-readable evidence

Generated protobuf/C++ build outputs belong in build directories unless packaging explicitly requires checked-in generated Python modules. The story must document the generation command and version.

Story output map

Story	Required durable outputs
DGR-001	benchmark harness/tests; evidence/DGR-001/performance-contract.json; raw/summary benchmark evidence
DGR-002	packages/node/native/proto/shard_runtime.proto; reproducible Python/C++ generation/build wiring; protocol round-trip/compatibility tests; evidence/DGR-002/
DGR-003	exact runtime-recipe/fingerprint implementation and admission tests; evidence/DGR-003/
DGR-004	exact upstream pin, numbered patch series, reproducible fetch/apply/build smoke; evidence/DGR-004/
DGR-005	dense-Llama range ownership loader and memory evidence; evidence/DGR-005/
DGR-006	architecture boundary adapter/parity tests and results; evidence/DGR-006/
DGR-007	concurrent session/KV manager, isolation/cleanup tests; evidence/DGR-007/
DGR-008	standalone C++ gRPC worker, fake-model integration tests, lifecycle evidence; evidence/DGR-008/
DGR-009	Meshnet backend/registration/relay integration and tests; evidence/DGR-009/
DGR-010	real local two-process commands, raw metrics and parity report; evidence/DGR-010/
DGR-011	two-machine configuration, commands, hardware/network manifest and raw results; evidence/DGR-011/
DGR-012	continuous scheduler/admission implementation and 1/2/4/8 concurrency report; evidence/DGR-012/
DGR-013	failure/cancel/restart test matrix and resource-cleanup evidence; evidence/DGR-013/
DGR-014	immutable final comparison against DGR-001 thresholds and ship/stop recommendation; evidence/DGR-014/
DGR-015	Qwen3-family adapter, architecture-specific parity/admission/performance evidence; evidence/DGR-015/
DGR-016	narrow upstream patches/tests, design note and human-ready outreach package; evidence/DGR-016/
DGR-017	exact GLM-5.2/GGUF target manifest, resource planner, immutable alpha contract and upstream status; evidence/DGR-017/
DGR-018	verified whole-model UD-IQ1_S oracle with native GLM semantic evidence; evidence/DGR-018/
DGR-019	explicit range-owned GLM MoE/MLA/DSA/IndexShare adapter, fixtures and parity; evidence/DGR-019/
DGR-020	real multi-node GLM-5.2 Max target evidence and immutable alpha/stop verdict; evidence/DGR-020/

Dependency handoff rule

For every dependency listed by Ralph:

1. Confirm its passes state in prd.json.
2. Read .scratch/distributed-gguf-runtime/evidence/<DEPENDENCY-ID>/README.md.
3. Verify referenced source paths and commands still exist.
4. Do not repeat completed work unless verification exposes a concrete defect.
5. If dependency evidence is missing or contradictory, stop and repair the dependency instead of guessing.

Testing and hardware rules

Default tests must be deterministic, GPU-free, model-download-free and API-credit-free.

Real model tests require:

MESHNET_ENABLE_REAL_INFERENCE_TESTS=1

On this machine:

• Use .venv-rocm for real Radeon 8060S ROCm execution.
• The default Python 3.14 .venv is unsuitable for real ROCm inference.
• Resolve model storage through the machine-specific .env.<hostname> configuration.
• Never download model artifacts under /home.
• Real acceptance must exercise actual Tracker-routed CPU/GPU computation; synthetic workers are only unit tests.

Record exact:

• Model/revision and Artifact hash.
• Quantization and runtime recipe.
• Host/hardware/backend/driver.
• Commands and environment names without secrets.
• Raw output and metrics.
• Whether the evidence is synthetic, local-real, or multi-machine-real.

Worktree and commit discipline

This repository may contain pre-existing changes from research or another feature.

• Inspect git status before editing.
• Never reset, checkout over, stash, delete or reformat unrelated changes.
• Stage only files belonging to the selected story.
• Exclude .ralph-tui, iteration logs, caches, generated builds, FUSE artifacts and unrelated scratch work.
• Keep one scoped commit per completed story when the supervising loop requests commits.
• Do not modify passes for another story.

Mandatory finish/handoff sequence

Before emitting <promise>COMPLETE</promise>:

1. Verify every acceptance criterion with real command output or file evidence.
2. Run story-specific gates and repository quality gates.
3. Write .scratch/distributed-gguf-runtime/evidence/<TASK-ID>/README.md containing:
   • Summary of changes.
   • Exact files changed.
   • Commands run and their real results.
   • Performance/correctness evidence.
   • Known limitations and deferred work.
   • Compatibility or migration notes.
   • Clear handoff for dependent stories.
4. Save machine-readable evidence beside it when the story produces metrics or schemas.
5. Update the source issue status to done only after all gates pass.
6. Preserve failures honestly. Never fabricate model, benchmark, test or hardware output.

Authoritative references

Active decisions:

• .scratch/distributed-gguf-runtime/README.md
• .scratch/distributed-gguf-runtime/implementation-strategy.md
• .scratch/distributed-gguf-runtime/architecture.md
• .scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md
• .scratch/distributed-gguf-runtime/PRD.md
• .scratch/distributed-gguf-runtime/prd.json

Source research:

• docs/research/distributed-gguf-landscape.md
• docs/research/distributed-gguf-github-followup.md
• docs/research/vllm-distributed-gguf-assessment.md

If historical notes conflict with these files, the active decisions above win.