documentation revision
This commit is contained in:
Dobromir Popov
@@ -8,7 +8,7 @@
|
||||
|
||||
## 1. Mission / where we are
|
||||
|
||||
neuron-tai is a volunteer-GPU, pipeline-parallel LLM inference network with a working routing layer and a **broken money/trust path**. Three independent audits agreed: unauthenticated gossip, free-credit faucet, double-pay risks, ephemeral bans, and node self-reported accounting undermine alpha release. The owner locked alpha scope (single settlement tracker, open node join, devnet mock-USDT, carried-forward reputation) and a fraud/verification design (TOPLOC adoption, adaptive audits, on-demand hop bisection, persisted graduated reputation, tracker-authoritative accounting). **Research and planning artifacts are complete** (ADRs 0016–0019, 22 issue files, README index). Next: implement Bucket 1 blockers test-first.
|
||||
neuron-tai is a volunteer-GPU, pipeline-parallel LLM inference network with a working routing layer. Pre-release audits found the money/trust path was not alpha-ready; **Bucket 1 alpha blockers are implemented** (see `.scratch/alpha-hardening/README.md`). Remaining launch gates: issue **21** (human calibration run), post-alpha Bucket 2 (12–15), and active scratch tracks (NCA, perf, distributed GGUF).
|
||||
|
||||
---
|
||||
|
||||
@@ -42,7 +42,7 @@ Point to artifacts — do not re-derive from this handoff.
|
||||
| Path | What it contains |
|
||||
|---|---|
|
||||
| `.scratch/alpha-hardening/README.md` | Issue/ADR index + implementation order |
|
||||
| `.scratch/alpha-hardening/issues/` | 22 work items (Buckets 1–3) |
|
||||
| `.scratch/alpha-hardening/issues/` | 25 work items (Buckets 1–3 + perf follow-ups) |
|
||||
| `.scratch/alpha-hardening/research-verifiable-inference.md` | SOTA research, layered alpha scheme (§8), build-vs-adopt (§9) |
|
||||
| `docs/adr/0016–0019` | Alpha scope, auth, fraud, multi-tracker design |
|
||||
| `docs/agents/issue-tracker.md` | Issue file conventions |
|
||||
|
||||
@@ -16,7 +16,7 @@ Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8
|
||||
|
||||
**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay stake-free until it's done, then move to real staking once thresholds derive from their own hardware.
|
||||
|
||||
**Current gap (confirmed 2026-07-06 by code read):** none of the three pieces below exist yet.
|
||||
**Current gap (historical — closed 2026-07-06):** the three engineering pieces below were missing when this issue was filed; all are now implemented and unit-tested. Remaining work is the human calibration run on the live hired-VPS fleet.
|
||||
|
||||
1. `verify_activation_proofs()` (`packages/validator/meshnet_validator/audit.py:94-127`) returns a **plain bool** — no raw TOPLOC divergence/distance value is ever computed or surfaced. Every "done" fraud-detection issue (06–10) currently runs on a guessed threshold baked into that bool, not a calibrated one.
|
||||
2. Fleet dispatch exists but is the wrong shape: `_handle_benchmark_hop_penalty` / `_handle_benchmark_results` (`packages/tracker/meshnet_tracker/server.py:2998-3104`, from the old US-030 latency work) targets pinned 1–3-node *routes* and measures latency, not TOPLOC divergence across *every* registered node.
|
||||
|
||||
@@ -1,121 +0,0 @@
|
||||
# ADR-0020: Lean Native Distributed GGUF Runtime
|
||||
|
||||
Status: Accepted
|
||||
Date: 2026-07-13
|
||||
|
||||
## Context
|
||||
|
||||
The project currently uses Transformers/safetensors as its real model execution backend. This provides broad architecture coverage and a correctness reference, but reported and observed consumer CPU/GPU inference performance motivates evaluating llama.cpp/GGML and quantized GGUF.
|
||||
|
||||
The product objective is not merely local GGUF serving. It is performant concurrent inference for top open models whose weights do not fit on one consumer node. The project already owns the Tracker, Inference Route, Route Session, Activation Seam, local Hot KV State, relay/direct transport, cancellation, telemetry, billing, and capability admission.
|
||||
|
||||
Research audited llama.cpp RPC, GPUStack/llama-box, Nakshatra, prima.cpp, llama-gguf, LiGGUF, vLLM and its GGUF plugin, Petals, exo, and related projects. No repository provides the complete public-network contract. llama.cpp is the strongest GGUF execution substrate. vLLM has mature managed-cluster parallelism and scheduling concepts but its PP/TP/EP runtime assumes a static trusted distributed world and is unsuitable as the public Shard runtime.
|
||||
|
||||
The project must remain lean and avoid combining several half-integrated inference control planes.
|
||||
|
||||
## Decision
|
||||
|
||||
### Primary native runtime
|
||||
|
||||
Use llama.cpp/GGML through one standalone C++ Shard worker and a small exact-commit patch stack.
|
||||
|
||||
The patch scope is limited to:
|
||||
|
||||
- Range-aware GGUF tensor ownership/loading.
|
||||
- Architecture-defined intermediate boundary input/output.
|
||||
- Intermediate output before tail normalization/head.
|
||||
- Layer-filtered KV and external session-to-sequence mapping.
|
||||
|
||||
Meshnet networking, routing, admission, billing, telemetry, and work evidence stay outside llama.cpp.
|
||||
|
||||
Nakshatra, prima.cpp, llama-gguf, LiGGUF, and historical GPUStack are source/test donors only. Their repositories are not runtime dependencies.
|
||||
|
||||
### Distributed parallelism
|
||||
|
||||
The first public-network primitive is layer/pipeline parallelism through contiguous Shards in an Inference Route.
|
||||
|
||||
Per-node continuous batching combines decode steps from compatible active Route Sessions. Multiple complete routes provide data parallelism.
|
||||
|
||||
Tensor and expert parallel collectives may later operate inside one trusted composite node or managed cluster represented as one provider. They are not public WAN routing primitives.
|
||||
|
||||
### Transport
|
||||
|
||||
Use gRPC over HTTP/2 with Protocol Buffers for the native Python/C++ Shard data plane.
|
||||
|
||||
- One long-lived bidirectional stream per Route Session Activation Seam.
|
||||
- Deadlines, cancellation, flow control, TLS/authentication hooks, structured status, and generated schemas.
|
||||
- Bounded chunks for prefill and a small decode fast path.
|
||||
- Existing relay infrastructure may carry the same versioned protobuf frames as opaque binary when direct connectivity is unavailable.
|
||||
- OpenAI client APIs remain HTTP/SSE; existing Tracker APIs remain unchanged.
|
||||
|
||||
The boundary payload is a versioned named-tensor bundle because architecture boundaries may require more than one tensor.
|
||||
|
||||
### vLLM
|
||||
|
||||
Do not fork vLLM for public distributed Shards and do not transplant PagedAttention, Torch process groups, or the vLLM GGUF plugin into the llama.cpp worker.
|
||||
|
||||
Allow unmodified vLLM as an optional whole-model backend or managed TP/PP/EP cluster represented as one logical provider.
|
||||
|
||||
Adapt only small control-plane concepts:
|
||||
|
||||
- Named intermediate bundles.
|
||||
- Continuous batching and request ownership.
|
||||
- Versioned cache-transfer compatibility fingerprints.
|
||||
- Explicit transfer failure/abort lifecycle.
|
||||
- Load telemetry and fair tie-breaking.
|
||||
|
||||
### Benchmark gate
|
||||
|
||||
GGUF performance is a hypothesis. Before expensive native work, compare the current Transformers/safetensors recipe with whole-model llama.cpp on controlled model, hardware, prompt, context, output, sampling, concurrency, memory, and quality lanes.
|
||||
|
||||
Later distributed release gates use thresholds locked before implementation results are known. The native track stops if llama.cpp/GGUF offers neither a meaningful performance benefit nor a meaningful model-fit benefit at useful speed.
|
||||
|
||||
### Concurrency
|
||||
|
||||
A native worker must isolate `(Route Session ID, route epoch)` through a llama sequence or bounded context and must not serialize all generations behind one global serving sequence.
|
||||
|
||||
The node admits sessions against weight/KV/scratch budgets, batches compatible decode steps, prevents prefill starvation, applies backpressure, and exposes queue/batch/KV telemetry.
|
||||
|
||||
### Architecture certification
|
||||
|
||||
Dense Llama-family is first. Qwen3/Qwen3-MoE is a separate explicit adapter. Every architecture/backend/recipe remains registered-but-dark until a real distributed forward, parity test, concurrency test, and capability admission pass.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
### Fork vLLM for the public mesh
|
||||
|
||||
Rejected because extracting its PP/TP/EP stages requires replacing static process groups, rank lifecycle, scheduler, request ownership, cache layout, failure behavior, and hardware assumptions. This would create a large difficult fork while discarding much of vLLM's core architecture.
|
||||
|
||||
### llama.cpp RPC as the public protocol
|
||||
|
||||
Rejected because it exposes coordinator-owned raw GGML devices, not independent Shards. Its trust, security, failure, cache, and per-node accounting model is unsuitable for arbitrary volunteer nodes.
|
||||
|
||||
### Adopt Nakshatra or prima.cpp wholesale
|
||||
|
||||
Rejected because their repositories, build reproducibility, session/concurrency semantics, architecture coverage, protocol identity, and control planes do not satisfy the project contract. Their partial-loading and boundary work remains valuable evidence.
|
||||
|
||||
### Build a custom GGUF engine
|
||||
|
||||
Rejected because llama.cpp already provides the parser, kernels, architecture graphs, KV, tokenizer, and heterogeneous backends. Reimplementing these would spread effort and increase correctness risk.
|
||||
|
||||
### Invent a custom transport
|
||||
|
||||
Rejected. gRPC/HTTP2 already provides mature streaming, flow control, deadlines, cancellation, TLS, and cross-language schema generation.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The critical path contains Meshnet, one standalone worker, and one small pinned llama.cpp patch stack.
|
||||
- Transformers/safetensors remains the correctness reference and fallback for unsupported architectures.
|
||||
- Whole-model llama.cpp and vLLM managed clusters remain useful optional provider types.
|
||||
- The first milestone emphasizes controlled benchmark, parity, concurrent KV, and real two-machine evidence rather than a large-model demo.
|
||||
- Upstream collaboration with llama.cpp targets generic local hooks only; the project remains able to ship a narrow pinned fork if upstream acceptance takes time.
|
||||
- QUIC, public tensor parallelism, disaggregated prefill, speculative decode, route repair, and KV migration remain deferred until the core route passes release gates.
|
||||
|
||||
## Verification gates
|
||||
|
||||
1. Controlled safetensors-versus-GGUF performance contract.
|
||||
2. Two-process local range parity.
|
||||
3. Four-session concurrent KV isolation.
|
||||
4. Real two-machine execution using both Shards.
|
||||
5. End-to-end performance/fit advantage over the current distributed route.
|
||||
6. Separate Qwen3-family architecture certification.
|
||||
@@ -9,7 +9,7 @@ Before changing code, every Ralph agent must:
|
||||
|
||||
1. Read this file completely.
|
||||
2. Read the selected issue under `.scratch/distributed-gguf-runtime/issues/`.
|
||||
3. Read `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
|
||||
3. Read `docs/adr/0024-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
|
||||
4. Read `.claude/memory/MEMORY.md` and root `CONTEXT.md` for current project vocabulary and constraints.
|
||||
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
|
||||
6. Read the evidence/handoff directories for every declared dependency.
|
||||
@@ -296,7 +296,7 @@ Active decisions:
|
||||
- `.scratch/distributed-gguf-runtime/README.md`
|
||||
- `.scratch/distributed-gguf-runtime/implementation-strategy.md`
|
||||
- `.scratch/distributed-gguf-runtime/architecture.md`
|
||||
- `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md`
|
||||
- `docs/adr/0024-distributed-gguf-runtime.md`
|
||||
- `.scratch/distributed-gguf-runtime/PRD.md`
|
||||
- `.scratch/distributed-gguf-runtime/prd.json`
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ Transformers/safetensors remains the correctness baseline. vLLM remains an optio
|
||||
- [Current architecture](architecture.md)
|
||||
- [PRD](PRD.md)
|
||||
- [Ralph backlog](prd.json)
|
||||
- [ADR-0020](ADR-0020-distributed-gguf-runtime.md)
|
||||
- [ADR-0024](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
- [Milestones](milestones.md)
|
||||
- [Issues](issues/)
|
||||
- [Distributed GGUF research](../../docs/research/distributed-gguf-landscape.md)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Distributed GGUF Decision Framework
|
||||
|
||||
> **Superseded for active implementation decisions.** The grill was resolved on 2026-07-13. Use [implementation-strategy.md](implementation-strategy.md), [architecture.md](architecture.md), [ADR-0020](ADR-0020-distributed-gguf-runtime.md), and [prd.json](prd.json). This file remains as historical decision rationale.
|
||||
> **Superseded for active implementation decisions.** The grill was resolved on 2026-07-13. Use [implementation-strategy.md](implementation-strategy.md), [architecture.md](architecture.md), [ADR-0024](../../docs/adr/0024-distributed-gguf-runtime.md), and [prd.json](prd.json). This file remains as historical decision rationale.
|
||||
|
||||
This framework is for grilling open decisions. It keeps decisions tied to project vocabulary and implementation gates instead of vague "distributed inference" language.
|
||||
|
||||
|
||||
@@ -56,4 +56,4 @@ As a runtime engineer, I need a controlled baseline so that GGUF work proceeds f
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -56,4 +56,4 @@ As a node developer, I need a battle-proven streaming protocol so that Python an
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -54,4 +54,4 @@ As the Tracker, I need exact compatibility identity so that only numerically and
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -58,4 +58,4 @@ As a maintainer, I need a small auditable fork boundary so that upstream updates
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -58,4 +58,4 @@ As a node, I need to map only my assigned dense-Llama Shard so that aggregate co
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -58,4 +58,4 @@ As a Shard, I need to consume and emit the correct transformer boundary state so
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -57,4 +57,4 @@ As a client, I need concurrent Route Sessions to retain independent per-Shard ca
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -62,4 +62,4 @@ As a node runtime, I need one supervised native process so that llama.cpp intern
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -58,4 +58,4 @@ As the existing node service, I need a GGUF Shard backend adapter so that the Tr
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -59,4 +59,4 @@ As a release engineer, I need real local distributed parity before involving net
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -59,4 +59,4 @@ As a consumer-hardware operator, I need two physical machines to execute one GGU
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -60,4 +60,4 @@ As a node operator, I need active sessions batched safely so that concurrency in
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -59,4 +59,4 @@ As a client, I need failures to be bounded and explicit so that distributed spee
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -62,4 +62,4 @@ As the product owner, I need an end-to-end comparison so that the native runtime
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -58,4 +58,4 @@ As a client seeking top models, I need a separately certified MoE-capable archit
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
@@ -57,4 +57,4 @@ As a maintainer, I need narrow upstreamable proposals so that our patch burden c
|
||||
- [PRD](../PRD.md)
|
||||
- [Implementation strategy](../implementation-strategy.md)
|
||||
- [Current architecture](../architecture.md)
|
||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
||||
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||
|
||||
Reference in New Issue
Block a user