docs: define implementation-ready distributed GGUF roadmap
This commit is contained in:
Dobromir Popov
@@ -1,309 +1,41 @@
|
||||
# Ralph execution context: Performant Concurrent Distributed GGUF Runtime
|
||||
# Ralph context: Distributed GGUF Runtime
|
||||
|
||||
Status: authoritative context for every fresh Ralph iteration
|
||||
Last updated: 2026-07-13
|
||||
> **Specification status:** planning artifacts only. No distributed GGUF runtime is implemented by this materialization, no story has completion credit, and legacy files remain for the DGR-017 audit. `prd.json` is authoritative.
|
||||
|
||||
## Mandatory startup sequence
|
||||
## Mandatory startup for every fresh story
|
||||
|
||||
Before changing code, every Ralph agent must:
|
||||
1. Read this file and authoritative `prd.json` completely.
|
||||
2. Read the generated source issue named in the selected story description.
|
||||
3. Read every dependency evidence README; legacy DGR-001..016 evidence is provenance only.
|
||||
4. Read `docs/adr/0024-distributed-gguf-runtime.md`, root `CONTEXT.md`, `.claude/memory/MEMORY.md`, and relevant live source/tests.
|
||||
5. Inspect `git status`; preserve unrelated work. Never infer implementation from planning text or old pass states.
|
||||
6. If blocked or oversized, keep `passes: false` and write an honest `BLOCKED.md`/`DECOMPOSITION.md`; never weaken criteria or fabricate evidence.
|
||||
|
||||
1. Read this file completely.
|
||||
2. Read the selected issue under `.scratch/distributed-gguf-runtime/issues/`.
|
||||
3. Read `docs/adr/0024-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
|
||||
4. Read `.claude/memory/MEMORY.md` and root `CONTEXT.md` for current project vocabulary and constraints.
|
||||
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
|
||||
6. Read the evidence/handoff directories for every declared dependency.
|
||||
7. Inspect `git status` and preserve all pre-existing working-tree changes.
|
||||
## Locked scope
|
||||
|
||||
A fresh Ralph iteration has no conversational memory. These files are the context contract.
|
||||
- Existing Meshnet Tracker routing, load balancing, billing, telemetry, relay, and provider semantics are backend-agnostic and are **not redesigned**. GGUF contributes exact compatibility, range/capacity, queue/load, seam-cost, health/reliability, and certification inputs only.
|
||||
- The data plane is a standalone project-owned C++ Shard worker with gRPC/Protobuf and a project-owned `ShardEngine` boundary.
|
||||
- llama.cpp is fetched at one exact commit into an ignored workspace from an in-repo manifest, then a numbered minimal patch stack is applied. There is no submodule, vendored tree, or permanent-fork dependency.
|
||||
- llama.cpp owns DeepSeek V4 graphs, mHC, MoE, attention, hash routing, and kernels. Meshnet adds only range-ownership hooks, typed boundary/local-state adapters, worker integration, and parity/certification.
|
||||
- Quantization and placement are dynamic recipe inputs. The 2–4 and 10+ stage layouts are certification scenarios, never product constants.
|
||||
- Per-shard Hot KV and V4 CSA/HCA/SWA/indexer/compressor state remain local and keyed by route session/epoch. The WAN seam carries the typed mHC 4×4096 residual boundary, positions, token-ID sideband where required, and schema/cache expectations—not per-layer caches.
|
||||
- Route changes use cache miss plus re-prefill/restart. There is no WAN KV or V4 auxiliary-cache migration.
|
||||
- CPU/CUDA/ROCm/Vulkan/Metal compile lanes are planned; only exact real-hardware-certified backend/model/recipe lanes may be advertised.
|
||||
- Alpha requires correctness and the pre-locked useful-speed gate. MTP is reserved and off for alpha; its ownership contract, implementation, and benchmark are required before beta.
|
||||
|
||||
## Story sizing and interruption rule
|
||||
## Target identities
|
||||
|
||||
Each story is intended to fit one focused Ralph context. Before implementation, estimate whether every acceptance criterion can be completed and verified in the current iteration.
|
||||
- DeepSeek V4 official target SHA: `60d8d70770c6776ff598c94bb586a859a38244f1`.
|
||||
- llama.cpp V4 support lineage began at PR 24162 / merge `8c146a8366304c871efc26057cc90370ccf58dad`; DGR-027 later pins one exact validated current commit.
|
||||
- V4 scope: 43 main layers plus MTP; mHC 4×4096 boundary; 256 routed + 1 shared experts with six routed active; token IDs required for the first three hash-routed layers.
|
||||
- Exact split-GGUF artifacts are provisioned to mounted-drive storage with a complete hashed manifest and resumable verification; no model artifact may be placed under `/home`.
|
||||
|
||||
If the story is too large, an external dependency is unavailable, or the context/provider limit prevents completion:
|
||||
## Control/data-plane contract
|
||||
|
||||
- Do not weaken criteria.
|
||||
- Do not mark the issue done or set `passes: true`.
|
||||
- Avoid leaving an unverified cross-cutting partial implementation when a smaller safe spike is possible.
|
||||
- Write `evidence/<TASK-ID>/DECOMPOSITION.md` or `BLOCKED.md` with the exact blocker, current verified state, proposed child stories, dependency graph and rollback/continuation instructions.
|
||||
- Stop for supervised review.
|
||||
Meshnet continues to own registration, coverage, existing route selection/load balancing, route epochs/sessions, direct/relay behavior, capability admission, cancellation, telemetry, billing, validation, and attribution. The GGUF adapter exposes measured inputs to those existing mechanisms. Direct seams use long-lived gRPC streams; relay seams carry byte-identical protobuf frames opaquely.
|
||||
|
||||
If interrupted after code changes, record every changed file, command result and unresolved invariant so the next fresh loop can verify rather than guess.
|
||||
The project-owned `ShardEngine` hides llama.cpp internals. A worker loads one exact artifact/recipe/range identity. Default tests use fake/tiny fixtures. Real runs are opt-in, preserve raw metrics, and never download models under `/home`.
|
||||
|
||||
## Product objective
|
||||
## Evidence and completion
|
||||
|
||||
Build performant, concurrent distributed inference that combines consumer machines to serve top open models that exceed one node's RAM/VRAM.
|
||||
|
||||
A distributed demo is not success. The product must provide:
|
||||
|
||||
- Useful measured prefill and decode speed.
|
||||
- Multiple concurrent Route Sessions.
|
||||
- No KV/token cross-talk.
|
||||
- Bounded memory, queues, cancellation and failures.
|
||||
- Real execution on every participating node.
|
||||
- A model-fit or performance advantage over the current Transformers/safetensors route.
|
||||
|
||||
## Critical-path architecture
|
||||
|
||||
```text
|
||||
Existing Meshnet control plane
|
||||
|
|
||||
Versioned Protobuf over gRPC/HTTP2
|
||||
|
|
||||
Project-owned standalone C++ Shard worker
|
||||
|
|
||||
Small exact-commit llama.cpp patch stack
|
||||
```
|
||||
|
||||
Meshnet remains the only control plane and owns:
|
||||
|
||||
- Tracker registration, Coverage Map, route selection and route epochs.
|
||||
- Route Sessions and Activation Seams.
|
||||
- Direct/relay routing.
|
||||
- Capability admission.
|
||||
- Cancellation, Generation Telemetry and backpressure.
|
||||
- Billing, validation and per-node work attribution.
|
||||
|
||||
Do not introduce another scheduler/control plane from vLLM, Nakshatra, prima.cpp, llama-gguf, GPUStack or another project.
|
||||
|
||||
## Runtime decisions that are not open
|
||||
|
||||
1. Public-network Shards are contiguous transformer layer ranges.
|
||||
2. llama.cpp/GGML is the native GGUF execution substrate.
|
||||
3. The project owns a small standalone worker and a narrow pinned llama.cpp patch stack.
|
||||
4. The native Shard protocol is Protocol Buffers over gRPC/HTTP2.
|
||||
5. One long-lived bidirectional stream serves one Route Session Activation Seam.
|
||||
6. The public activation boundary is a versioned named-tensor bundle.
|
||||
7. Hot KV State remains local to the node serving the Shard.
|
||||
8. `(Route Session ID, route epoch)` maps to an isolated llama sequence or bounded context.
|
||||
9. Concurrency uses continuous batching of compatible active sessions inside each node.
|
||||
10. Transformers/safetensors remains the correctness and performance baseline.
|
||||
11. vLLM may be an optional complete managed provider and concept donor; it is not forked into public Shards.
|
||||
12. Tensor/expert collectives are deferred to a trusted composite provider, not public WAN routes.
|
||||
13. Unsupported architectures/backends remain registered-but-dark until real certification passes.
|
||||
14. Alpha failure retries from token zero; unverified KV is never migrated silently.
|
||||
15. Model artifacts must remain on mounted-drive storage and never under `/home`.
|
||||
|
||||
Changing one of these requires an explicit ADR update and human review, not an incidental story implementation.
|
||||
|
||||
## Performance discipline
|
||||
|
||||
GGUF performance is a hypothesis. Never write “GGUF is faster” without measurements.
|
||||
|
||||
DGR-001 locks controlled benchmark lanes and thresholds. DGR-014 enforces the final distributed comparison.
|
||||
|
||||
Always distinguish:
|
||||
|
||||
- Weight quantization from activation/compute/KV dtype.
|
||||
- Runtime/kernel gains from quantization/model-fit gains.
|
||||
- Single-request latency from aggregate concurrency throughput.
|
||||
- Synthetic unit coverage from real distributed acceptance.
|
||||
|
||||
Required metrics where applicable:
|
||||
|
||||
```text
|
||||
TTFT
|
||||
prefill tokens/sec
|
||||
decode tokens/sec
|
||||
aggregate throughput
|
||||
p50/p95 latency
|
||||
seam bytes and latency
|
||||
queue and batch occupancy
|
||||
RSS and VRAM
|
||||
KV pressure
|
||||
output-quality drift
|
||||
failures and cleanup
|
||||
```
|
||||
|
||||
Do not weaken or move performance thresholds after seeing implementation results.
|
||||
|
||||
## Transport discipline
|
||||
|
||||
Do not invent a raw TCP protocol, new WebSocket protocol, QUIC layer or bespoke binary control format.
|
||||
|
||||
The `.proto` schema is the semantic contract. Direct transport uses gRPC. Existing relay infrastructure may carry the same serialized protobuf frames as opaque binary.
|
||||
|
||||
Protocol requirements:
|
||||
|
||||
- Schema/version negotiation.
|
||||
- Request/work ID.
|
||||
- Route Session ID and route epoch.
|
||||
- Exact Model Artifact/runtime recipe fingerprint.
|
||||
- Shard range and effective overlap-safe start.
|
||||
- Prefill/decode/release/cancel phases.
|
||||
- Position/token range and idempotency step.
|
||||
- Named tensors with shape, dtype, byte order and bounded fragments.
|
||||
- Compression/checksum.
|
||||
- Cache expectation/result.
|
||||
- Deadlines, cancellation, flow control and structured status.
|
||||
|
||||
Avoid per-token channel creation and unbounded unary payloads. Generated code and build tooling must be reproducible; do not require manual copying.
|
||||
|
||||
## Native runtime discipline
|
||||
|
||||
Reuse llama.cpp for GGUF, mmap, kernels, architecture graphs, tokenizer, KV, sequences and heterogeneous backends.
|
||||
|
||||
The project patch stack is limited to:
|
||||
|
||||
- Range-aware tensor registration/loading.
|
||||
- Endpoint-specific embedding/final head ownership.
|
||||
- Architecture-defined intermediate input/output.
|
||||
- Intermediate output before final norm/head.
|
||||
- Layer-filtered KV and session mapping.
|
||||
|
||||
Do not place Meshnet routing, transport, billing or authentication inside llama.cpp. Keep patches numbered, scoped, pinned and upstreamable.
|
||||
|
||||
Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate adapter after the dense release gate. Do not generalize through unchecked tensor-name substitutions.
|
||||
|
||||
## Existing code seams to inspect first
|
||||
|
||||
- `packages/node/meshnet_node/model_backend.py` — backend abstraction.
|
||||
- `packages/node/meshnet_node/torch_server.py` — reference ranged execution and session behavior.
|
||||
- `packages/node/meshnet_node/activation_compression.py` — current activation framing/compression.
|
||||
- `packages/node/meshnet_node/route_session_benchmark.py` — existing benchmark infrastructure.
|
||||
- `packages/tracker/meshnet_tracker/server.py` — registration, route and proxy behavior.
|
||||
- `packages/tracker/meshnet_tracker/capability.py` — fail-closed capability admission.
|
||||
- `tests/test_real_model_backend.py` — real backend coverage.
|
||||
- `tests/test_tracker_routing.py` — route/session behavior.
|
||||
- `tests/test_tracker_capability_admission.py` — recipe admission.
|
||||
- `tests/test_route_session_benchmark.py` and `tests/test_manual_route_benchmark.py` — benchmark patterns.
|
||||
- `docs/adr/0008-binary-activation-wire-format.md` — existing wire compatibility.
|
||||
- `docs/adr/0012-start-layer-overlapping-shards.md` — effective start semantics.
|
||||
- `docs/adr/0022-sharded-per-node-kv-cache.md` — Hot KV State contract.
|
||||
- `docs/adr/0023-model-agnostic-node-capability-admission.md` — certification/admission.
|
||||
|
||||
Do not edit generated `build/`, `__pycache__`, egg-info, Ralph logs or unrelated scratch features.
|
||||
|
||||
## Planned source layout
|
||||
|
||||
Use these paths unless current code inspection proves a better project-consistent location. If changed, document the reason in task evidence.
|
||||
|
||||
```text
|
||||
packages/node/native/
|
||||
proto/shard_runtime.proto
|
||||
cmake/
|
||||
llama/
|
||||
UPSTREAM_COMMIT
|
||||
patches/
|
||||
gguf_worker/
|
||||
tests/
|
||||
|
||||
packages/node/meshnet_node/
|
||||
native_protocol/
|
||||
gguf_backend.py
|
||||
runtime_recipe.py
|
||||
|
||||
.scratch/distributed-gguf-runtime/evidence/<TASK-ID>/
|
||||
README.md
|
||||
commands.txt
|
||||
results.json or other machine-readable evidence
|
||||
```
|
||||
|
||||
Generated protobuf/C++ build outputs belong in build directories unless packaging explicitly requires checked-in generated Python modules. The story must document the generation command and version.
|
||||
|
||||
## Story output map
|
||||
|
||||
| Story | Required durable outputs |
|
||||
|---|---|
|
||||
| DGR-001 | benchmark harness/tests; `evidence/DGR-001/performance-contract.json`; raw/summary benchmark evidence |
|
||||
| DGR-002 | `packages/node/native/proto/shard_runtime.proto`; reproducible Python/C++ generation/build wiring; protocol round-trip/compatibility tests; `evidence/DGR-002/` |
|
||||
| DGR-003 | exact runtime-recipe/fingerprint implementation and admission tests; `evidence/DGR-003/` |
|
||||
| DGR-004 | exact upstream pin, numbered patch series, reproducible fetch/apply/build smoke; `evidence/DGR-004/` |
|
||||
| DGR-005 | dense-Llama range ownership loader and memory evidence; `evidence/DGR-005/` |
|
||||
| DGR-006 | architecture boundary adapter/parity tests and results; `evidence/DGR-006/` |
|
||||
| DGR-007 | concurrent session/KV manager, isolation/cleanup tests; `evidence/DGR-007/` |
|
||||
| DGR-008 | standalone C++ gRPC worker, fake-model integration tests, lifecycle evidence; `evidence/DGR-008/` |
|
||||
| DGR-009 | Meshnet backend/registration/relay integration and tests; `evidence/DGR-009/` |
|
||||
| DGR-010 | real local two-process commands, raw metrics and parity report; `evidence/DGR-010/` |
|
||||
| DGR-011 | two-machine configuration, commands, hardware/network manifest and raw results; `evidence/DGR-011/` |
|
||||
| DGR-012 | continuous scheduler/admission implementation and 1/2/4/8 concurrency report; `evidence/DGR-012/` |
|
||||
| DGR-013 | failure/cancel/restart test matrix and resource-cleanup evidence; `evidence/DGR-013/` |
|
||||
| DGR-014 | immutable final comparison against DGR-001 thresholds and ship/stop recommendation; `evidence/DGR-014/` |
|
||||
| DGR-015 | Qwen3-family adapter, architecture-specific parity/admission/performance evidence; `evidence/DGR-015/` |
|
||||
| DGR-016 | narrow upstream patches/tests, design note and human-ready outreach package; `evidence/DGR-016/` |
|
||||
|
||||
## Dependency handoff rule
|
||||
|
||||
For every dependency listed by Ralph:
|
||||
|
||||
1. Confirm its `passes` state in `prd.json`.
|
||||
2. Read `.scratch/distributed-gguf-runtime/evidence/<DEPENDENCY-ID>/README.md`.
|
||||
3. Verify referenced source paths and commands still exist.
|
||||
4. Do not repeat completed work unless verification exposes a concrete defect.
|
||||
5. If dependency evidence is missing or contradictory, stop and repair the dependency instead of guessing.
|
||||
|
||||
## Testing and hardware rules
|
||||
|
||||
Default tests must be deterministic, GPU-free, model-download-free and API-credit-free.
|
||||
|
||||
Real model tests require:
|
||||
|
||||
```text
|
||||
MESHNET_ENABLE_REAL_INFERENCE_TESTS=1
|
||||
```
|
||||
|
||||
On this machine:
|
||||
|
||||
- Use `.venv-rocm` for real Radeon 8060S ROCm execution.
|
||||
- The default Python 3.14 `.venv` is unsuitable for real ROCm inference.
|
||||
- Resolve model storage through the machine-specific `.env.<hostname>` configuration.
|
||||
- Never download model artifacts under `/home`.
|
||||
- Real acceptance must exercise actual Tracker-routed CPU/GPU computation; synthetic workers are only unit tests.
|
||||
|
||||
Record exact:
|
||||
|
||||
- Model/revision and Artifact hash.
|
||||
- Quantization and runtime recipe.
|
||||
- Host/hardware/backend/driver.
|
||||
- Commands and environment names without secrets.
|
||||
- Raw output and metrics.
|
||||
- Whether the evidence is synthetic, local-real, or multi-machine-real.
|
||||
|
||||
## Worktree and commit discipline
|
||||
|
||||
This repository may contain pre-existing changes from research or another feature.
|
||||
|
||||
- Inspect `git status` before editing.
|
||||
- Never reset, checkout over, stash, delete or reformat unrelated changes.
|
||||
- Stage only files belonging to the selected story.
|
||||
- Exclude `.ralph-tui`, iteration logs, caches, generated builds, FUSE artifacts and unrelated scratch work.
|
||||
- Keep one scoped commit per completed story when the supervising loop requests commits.
|
||||
- Do not modify `passes` for another story.
|
||||
|
||||
## Mandatory finish/handoff sequence
|
||||
|
||||
Before emitting `<promise>COMPLETE</promise>`:
|
||||
|
||||
1. Verify every acceptance criterion with real command output or file evidence.
|
||||
2. Run story-specific gates and repository quality gates.
|
||||
3. Write `.scratch/distributed-gguf-runtime/evidence/<TASK-ID>/README.md` containing:
|
||||
- Summary of changes.
|
||||
- Exact files changed.
|
||||
- Commands run and their real results.
|
||||
- Performance/correctness evidence.
|
||||
- Known limitations and deferred work.
|
||||
- Compatibility or migration notes.
|
||||
- Clear handoff for dependent stories.
|
||||
4. Save machine-readable evidence beside it when the story produces metrics or schemas.
|
||||
5. Update the source issue status to `done` only after all gates pass.
|
||||
6. Preserve failures honestly. Never fabricate model, benchmark, test or hardware output.
|
||||
|
||||
## Authoritative references
|
||||
|
||||
Active decisions:
|
||||
|
||||
- `.scratch/distributed-gguf-runtime/README.md`
|
||||
- `.scratch/distributed-gguf-runtime/implementation-strategy.md`
|
||||
- `.scratch/distributed-gguf-runtime/architecture.md`
|
||||
- `docs/adr/0024-distributed-gguf-runtime.md`
|
||||
- `.scratch/distributed-gguf-runtime/PRD.md`
|
||||
- `.scratch/distributed-gguf-runtime/prd.json`
|
||||
|
||||
Source research:
|
||||
|
||||
- `docs/research/distributed-gguf-landscape.md`
|
||||
- `docs/research/distributed-gguf-github-followup.md`
|
||||
- `docs/research/vllm-distributed-gguf-assessment.md`
|
||||
|
||||
If historical notes conflict with these files, the active decisions above win.
|
||||
Each story writes `/run/media/popov/d/DEV/repos/d-popov.com/AI/.claude/worktrees/distributed-gguf-runtime/.scratch/distributed-gguf-runtime/evidence/<DGR-ID>/README.md` with exact files, commands/results, limitations, identities, and dependent-story handoff. Only `prd.json` may record `passes`; all 55 are false now. Generated Markdown cannot override it. One scoped commit per story is expected during future execution, but this plan materialization is intentionally uncommitted.
|
||||
|
||||
Reference in New Issue
Block a user