documentation revision
This commit is contained in:
Dobromir Popov
@@ -7,4 +7,4 @@
|
|||||||
- [Alpha hardening navigation](alpha-hardening-navigation.md) — locked fraud/auth decisions, Bucket-1 order, handoff pointers
|
- [Alpha hardening navigation](alpha-hardening-navigation.md) — locked fraud/auth decisions, Bucket-1 order, handoff pointers
|
||||||
- **Node capability admission** — `.scratch/node-capability-admission/` (P0 plan: generic doctor/real-forward validation, fail-closed readiness, tracker admission gate; [PRD](../../.scratch/node-capability-admission/PRD.md), [README](../../.scratch/node-capability-admission/README.md), ADR-0023)
|
- **Node capability admission** — `.scratch/node-capability-admission/` (P0 plan: generic doctor/real-forward validation, fail-closed readiness, tracker admission gate; [PRD](../../.scratch/node-capability-admission/PRD.md), [README](../../.scratch/node-capability-admission/README.md), ADR-0023)
|
||||||
- **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning.
|
- **Distributed relay performance** — relay `/rpc` requester sockets are persistent per Route Session and Activation Seam as of 2026-07-10; `request_id` remains unique per activation while `X-Meshnet-Session` remains stable for KV state. Next low-risk priorities: persistent direct/loopback HTTP, seam byte/latency telemetry, then trace-driven zstd tuning.
|
||||||
- **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md).
|
- **Distributed GGUF direction** — benchmark-gated native runtime: compare controlled Transformers/safetensors and whole-model llama.cpp lanes before expensive work; ship only for measured speed or model-fit advantage. Public parallelism is contiguous Shards in an Inference Route; concurrency comes from per-node continuous batching across isolated Route Sessions, while tensor/expert collectives stay inside optional trusted composite providers. Native data plane uses versioned Protobuf over long-lived gRPC/HTTP2 seam streams, with existing relay carrying the same opaque frames when needed. llama.cpp/GGML remains the substrate behind a project-owned standalone worker and small pinned fork; vLLM is an optional complete managed provider and concept donor, not a fork. Nakshatra, `prima.cpp`, `llama-gguf`, LiGGUF and historical GPUStack are source/test donors only. Active plan: [README](../../.scratch/distributed-gguf-runtime/README.md), [architecture](../../.scratch/distributed-gguf-runtime/architecture.md), [PRD](../../.scratch/distributed-gguf-runtime/PRD.md), [Ralph backlog](../../.scratch/distributed-gguf-runtime/prd.json). ADR: [0024](../../docs/adr/0024-distributed-gguf-runtime.md). Research: [landscape](../../docs/research/distributed-gguf-landscape.md), [GitHub follow-up](../../docs/research/distributed-gguf-github-followup.md), [vLLM](../../docs/research/vllm-distributed-gguf-assessment.md).
|
||||||
|
|||||||
@@ -20,9 +20,9 @@ Active workstream (started 2026-07-04): alpha hardening of the money/trust path.
|
|||||||
|
|
||||||
**Launch-readiness grilling (2026-07-06):** Locked launch plan — devnet dev/test run now, then **real mainnet SOL/USDT** (not devnet, not a new public token) for the first cohort: friends (API clients) + hired VPS/VPC hosts (our own test infra, not third-party volunteers — stake-free, risk-free if something breaks, not a long-term topology). Pricing: clients are the only party spending real money; nodes only accumulate off-chain credit and get paid in batches (30min dev / 24h later) — a failed distribution leaves funds parked, not lost, so mainnet-vs-devnet mixups are lower-risk than initially assumed. TAI token: do NOT issue/list now — ADR-0002 already locks listing behind $50k volume + 25 nodes/15 wallets plus an unresolved securities-review gate; only a dormant mainnet mint (cheap, ~few $ SOL) for name/branding reservation is in scope, bundled with treasury-key work, not before it. Treasury custody: bare keypair file (current runbook 02) is not acceptable for real funds — plan is **free native SPL multisig** (`spl-token create-multisig`, no protocol fee unlike Squads' 0.5 SOL), 2-of-3 signers, at least one cold/offline, others one-per-hired-VPS-provider to avoid correlated compromise (not yet built — ops task, no issue filed). Stake/slash asymmetry (registry/slash is a local Python adapter per ADR-0007, not on-chain) accepted for now since hired hosts are our own infra and friends aren't node operators — revisit before opening to real third-party node operators. A mainnet-vs-devnet boot guardrail was proposed and explicitly declined by the owner given the safe-by-default money flow above.
|
**Launch-readiness grilling (2026-07-06):** Locked launch plan — devnet dev/test run now, then **real mainnet SOL/USDT** (not devnet, not a new public token) for the first cohort: friends (API clients) + hired VPS/VPC hosts (our own test infra, not third-party volunteers — stake-free, risk-free if something breaks, not a long-term topology). Pricing: clients are the only party spending real money; nodes only accumulate off-chain credit and get paid in batches (30min dev / 24h later) — a failed distribution leaves funds parked, not lost, so mainnet-vs-devnet mixups are lower-risk than initially assumed. TAI token: do NOT issue/list now — ADR-0002 already locks listing behind $50k volume + 25 nodes/15 wallets plus an unresolved securities-review gate; only a dormant mainnet mint (cheap, ~few $ SOL) for name/branding reservation is in scope, bundled with treasury-key work, not before it. Treasury custody: bare keypair file (current runbook 02) is not acceptable for real funds — plan is **free native SPL multisig** (`spl-token create-multisig`, no protocol fee unlike Squads' 0.5 SOL), 2-of-3 signers, at least one cold/offline, others one-per-hired-VPS-provider to avoid correlated compromise (not yet built — ops task, no issue filed). Stake/slash asymmetry (registry/slash is a local Python adapter per ADR-0007, not on-chain) accepted for now since hired hosts are our own infra and friends aren't node operators — revisit before opening to real third-party node operators. A mainnet-vs-devnet boot guardrail was proposed and explicitly declined by the owner given the safe-by-default money flow above.
|
||||||
|
|
||||||
**Two new issues from this session, both `ready-for-agent`:**
|
**Two new issues from this session:**
|
||||||
- **21 — Honest-noise calibration corpus** (`.scratch/alpha-hardening/issues/21-honest-noise-calibration-corpus.md`) rescoped from "prod gate" to a **hard alpha-release blocker**. Confirmed by code read: `verify_activation_proofs()` (`packages/validator/meshnet_validator/audit.py:94-127`) returns bool only, no raw divergence value; fleet-dispatch exists but wrong shape (`server.py:2998-3104`, pinned routes + latency, not full-fleet + TOPLOC divergence); storage wrong shape (`registry_events` has no divergence/hardware columns). Three-part build: (1) surface raw TOPLOC distance from audit.py, (2) extend dispatch to hit every registered node with fixed prompt/seed, (3) new SQLite table keyed by node+GPU+dtype. Small-fleet exception granted (N = actual hired-VPS fleet size). Hired VPS hosts stay stake-free until this closes.
|
- **21 — Honest-noise calibration corpus** — `Status: ready-for-human` (engineering done 2026-07-06; blocked on human fleet calibration run before mainnet launch).
|
||||||
- **23 — Dynamic HF-benchmarked pricing** (`.scratch/alpha-hardening/issues/23-dynamic-hf-pricing.md`), high priority but not a release blocker. Pricing today is 100% static (`DEFAULT_PRICE_PER_1K_TOKENS = 0.02`, `billing.py:21`; `model_presets.json` has no per-model price). Target: 80% of cheapest comparable provider on `https://huggingface.co/inference/models` (per-provider-per-model marketplace, `?search=` query param works, no confirmed JSON API — plain scrape attempted first, escalate to headless browser only if the table isn't in raw HTML). Human-verified `hf_aliases` + `hf_verified_match_note` (params/quantization) per model, not auto-discovered matching. Reuses the `_settlement_loop` daemon-thread pattern for a daily refresh; falls back silently to the static default on any failure.
|
- **23 — Dynamic HF-benchmarked pricing** — `Status: done` (see `23-dynamic-hf-pricing_completed.md`).
|
||||||
|
|
||||||
Both are already migrated into `.scratch/alpha-hardening/prd.json` (AH-021 updated, AH-023 added) and the README index — ready for Ralph to pick up unattended.
|
Both are already migrated into `.scratch/alpha-hardening/prd.json` (AH-021 updated, AH-023 added) and the README index — ready for Ralph to pick up unattended.
|
||||||
|
|
||||||
|
|||||||
@@ -6,7 +6,9 @@ metadata:
|
|||||||
type: project
|
type: project
|
||||||
---
|
---
|
||||||
|
|
||||||
# Project Status (2026-07-02)
|
# Project Status (2026-07-13)
|
||||||
|
|
||||||
|
> Doc reconciliation 2026-07-13: `docs/PRD.md`, ADR numbering (0024 = distributed GGUF), and `docs/issues/` statuses updated to match `docs/prd.json` (US-001…US-035 done). Post-035 work remains in issues 36–48 and `.scratch/`.
|
||||||
|
|
||||||
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
|
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
|
||||||
|
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
|
|
||||||
## 1. Mission / where we are
|
## 1. Mission / where we are
|
||||||
|
|
||||||
neuron-tai is a volunteer-GPU, pipeline-parallel LLM inference network with a working routing layer and a **broken money/trust path**. Three independent audits agreed: unauthenticated gossip, free-credit faucet, double-pay risks, ephemeral bans, and node self-reported accounting undermine alpha release. The owner locked alpha scope (single settlement tracker, open node join, devnet mock-USDT, carried-forward reputation) and a fraud/verification design (TOPLOC adoption, adaptive audits, on-demand hop bisection, persisted graduated reputation, tracker-authoritative accounting). **Research and planning artifacts are complete** (ADRs 0016–0019, 22 issue files, README index). Next: implement Bucket 1 blockers test-first.
|
neuron-tai is a volunteer-GPU, pipeline-parallel LLM inference network with a working routing layer. Pre-release audits found the money/trust path was not alpha-ready; **Bucket 1 alpha blockers are implemented** (see `.scratch/alpha-hardening/README.md`). Remaining launch gates: issue **21** (human calibration run), post-alpha Bucket 2 (12–15), and active scratch tracks (NCA, perf, distributed GGUF).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -42,7 +42,7 @@ Point to artifacts — do not re-derive from this handoff.
|
|||||||
| Path | What it contains |
|
| Path | What it contains |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `.scratch/alpha-hardening/README.md` | Issue/ADR index + implementation order |
|
| `.scratch/alpha-hardening/README.md` | Issue/ADR index + implementation order |
|
||||||
| `.scratch/alpha-hardening/issues/` | 22 work items (Buckets 1–3) |
|
| `.scratch/alpha-hardening/issues/` | 25 work items (Buckets 1–3 + perf follow-ups) |
|
||||||
| `.scratch/alpha-hardening/research-verifiable-inference.md` | SOTA research, layered alpha scheme (§8), build-vs-adopt (§9) |
|
| `.scratch/alpha-hardening/research-verifiable-inference.md` | SOTA research, layered alpha scheme (§8), build-vs-adopt (§9) |
|
||||||
| `docs/adr/0016–0019` | Alpha scope, auth, fraud, multi-tracker design |
|
| `docs/adr/0016–0019` | Alpha scope, auth, fraud, multi-tracker design |
|
||||||
| `docs/agents/issue-tracker.md` | Issue file conventions |
|
| `docs/agents/issue-tracker.md` | Issue file conventions |
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ Research anchor: `.scratch/alpha-hardening/research-verifiable-inference.md` §8
|
|||||||
|
|
||||||
**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay stake-free until it's done, then move to real staking once thresholds derive from their own hardware.
|
**Launch context (why this is buildable now, not a research project):** first-launch nodes are hired VPS/VPC hosts under our own direct control (test infrastructure we pay for, not third-party volunteers) — not a long-term topology, but risk-free for calibration purposes since there's no external party to dispute a bad reading. Friends are client-side users of the API in this phase, not node operators. Run the calibration pass against this small, fully-controlled fleet first; hired hosts stay stake-free until it's done, then move to real staking once thresholds derive from their own hardware.
|
||||||
|
|
||||||
**Current gap (confirmed 2026-07-06 by code read):** none of the three pieces below exist yet.
|
**Current gap (historical — closed 2026-07-06):** the three engineering pieces below were missing when this issue was filed; all are now implemented and unit-tested. Remaining work is the human calibration run on the live hired-VPS fleet.
|
||||||
|
|
||||||
1. `verify_activation_proofs()` (`packages/validator/meshnet_validator/audit.py:94-127`) returns a **plain bool** — no raw TOPLOC divergence/distance value is ever computed or surfaced. Every "done" fraud-detection issue (06–10) currently runs on a guessed threshold baked into that bool, not a calibrated one.
|
1. `verify_activation_proofs()` (`packages/validator/meshnet_validator/audit.py:94-127`) returns a **plain bool** — no raw TOPLOC divergence/distance value is ever computed or surfaced. Every "done" fraud-detection issue (06–10) currently runs on a guessed threshold baked into that bool, not a calibrated one.
|
||||||
2. Fleet dispatch exists but is the wrong shape: `_handle_benchmark_hop_penalty` / `_handle_benchmark_results` (`packages/tracker/meshnet_tracker/server.py:2998-3104`, from the old US-030 latency work) targets pinned 1–3-node *routes* and measures latency, not TOPLOC divergence across *every* registered node.
|
2. Fleet dispatch exists but is the wrong shape: `_handle_benchmark_hop_penalty` / `_handle_benchmark_results` (`packages/tracker/meshnet_tracker/server.py:2998-3104`, from the old US-030 latency work) targets pinned 1–3-node *routes* and measures latency, not TOPLOC divergence across *every* registered node.
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ Before changing code, every Ralph agent must:
|
|||||||
|
|
||||||
1. Read this file completely.
|
1. Read this file completely.
|
||||||
2. Read the selected issue under `.scratch/distributed-gguf-runtime/issues/`.
|
2. Read the selected issue under `.scratch/distributed-gguf-runtime/issues/`.
|
||||||
3. Read `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
|
3. Read `docs/adr/0024-distributed-gguf-runtime.md` and the relevant part of `architecture.md`.
|
||||||
4. Read `.claude/memory/MEMORY.md` and root `CONTEXT.md` for current project vocabulary and constraints.
|
4. Read `.claude/memory/MEMORY.md` and root `CONTEXT.md` for current project vocabulary and constraints.
|
||||||
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
|
5. Inspect the current implementation and tests; do not assume historical scratch text describes live code.
|
||||||
6. Read the evidence/handoff directories for every declared dependency.
|
6. Read the evidence/handoff directories for every declared dependency.
|
||||||
@@ -296,7 +296,7 @@ Active decisions:
|
|||||||
- `.scratch/distributed-gguf-runtime/README.md`
|
- `.scratch/distributed-gguf-runtime/README.md`
|
||||||
- `.scratch/distributed-gguf-runtime/implementation-strategy.md`
|
- `.scratch/distributed-gguf-runtime/implementation-strategy.md`
|
||||||
- `.scratch/distributed-gguf-runtime/architecture.md`
|
- `.scratch/distributed-gguf-runtime/architecture.md`
|
||||||
- `.scratch/distributed-gguf-runtime/ADR-0020-distributed-gguf-runtime.md`
|
- `docs/adr/0024-distributed-gguf-runtime.md`
|
||||||
- `.scratch/distributed-gguf-runtime/PRD.md`
|
- `.scratch/distributed-gguf-runtime/PRD.md`
|
||||||
- `.scratch/distributed-gguf-runtime/prd.json`
|
- `.scratch/distributed-gguf-runtime/prd.json`
|
||||||
|
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ Transformers/safetensors remains the correctness baseline. vLLM remains an optio
|
|||||||
- [Current architecture](architecture.md)
|
- [Current architecture](architecture.md)
|
||||||
- [PRD](PRD.md)
|
- [PRD](PRD.md)
|
||||||
- [Ralph backlog](prd.json)
|
- [Ralph backlog](prd.json)
|
||||||
- [ADR-0020](ADR-0020-distributed-gguf-runtime.md)
|
- [ADR-0024](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
- [Milestones](milestones.md)
|
- [Milestones](milestones.md)
|
||||||
- [Issues](issues/)
|
- [Issues](issues/)
|
||||||
- [Distributed GGUF research](../../docs/research/distributed-gguf-landscape.md)
|
- [Distributed GGUF research](../../docs/research/distributed-gguf-landscape.md)
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Distributed GGUF Decision Framework
|
# Distributed GGUF Decision Framework
|
||||||
|
|
||||||
> **Superseded for active implementation decisions.** The grill was resolved on 2026-07-13. Use [implementation-strategy.md](implementation-strategy.md), [architecture.md](architecture.md), [ADR-0020](ADR-0020-distributed-gguf-runtime.md), and [prd.json](prd.json). This file remains as historical decision rationale.
|
> **Superseded for active implementation decisions.** The grill was resolved on 2026-07-13. Use [implementation-strategy.md](implementation-strategy.md), [architecture.md](architecture.md), [ADR-0024](../../docs/adr/0024-distributed-gguf-runtime.md), and [prd.json](prd.json). This file remains as historical decision rationale.
|
||||||
|
|
||||||
This framework is for grilling open decisions. It keeps decisions tied to project vocabulary and implementation gates instead of vague "distributed inference" language.
|
This framework is for grilling open decisions. It keeps decisions tied to project vocabulary and implementation gates instead of vague "distributed inference" language.
|
||||||
|
|
||||||
|
|||||||
@@ -56,4 +56,4 @@ As a runtime engineer, I need a controlled baseline so that GGUF work proceeds f
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -56,4 +56,4 @@ As a node developer, I need a battle-proven streaming protocol so that Python an
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -54,4 +54,4 @@ As the Tracker, I need exact compatibility identity so that only numerically and
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -58,4 +58,4 @@ As a maintainer, I need a small auditable fork boundary so that upstream updates
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -58,4 +58,4 @@ As a node, I need to map only my assigned dense-Llama Shard so that aggregate co
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -58,4 +58,4 @@ As a Shard, I need to consume and emit the correct transformer boundary state so
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -57,4 +57,4 @@ As a client, I need concurrent Route Sessions to retain independent per-Shard ca
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -62,4 +62,4 @@ As a node runtime, I need one supervised native process so that llama.cpp intern
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -58,4 +58,4 @@ As the existing node service, I need a GGUF Shard backend adapter so that the Tr
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -59,4 +59,4 @@ As a release engineer, I need real local distributed parity before involving net
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -59,4 +59,4 @@ As a consumer-hardware operator, I need two physical machines to execute one GGU
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -60,4 +60,4 @@ As a node operator, I need active sessions batched safely so that concurrency in
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -59,4 +59,4 @@ As a client, I need failures to be bounded and explicit so that distributed spee
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -62,4 +62,4 @@ As the product owner, I need an end-to-end comparison so that the native runtime
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -58,4 +58,4 @@ As a client seeking top models, I need a separately certified MoE-capable archit
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
@@ -57,4 +57,4 @@ As a maintainer, I need narrow upstreamable proposals so that our patch burden c
|
|||||||
- [PRD](../PRD.md)
|
- [PRD](../PRD.md)
|
||||||
- [Implementation strategy](../implementation-strategy.md)
|
- [Implementation strategy](../implementation-strategy.md)
|
||||||
- [Current architecture](../architecture.md)
|
- [Current architecture](../architecture.md)
|
||||||
- [Architecture decision](../ADR-0020-distributed-gguf-runtime.md)
|
- [Architecture decision](../../docs/adr/0024-distributed-gguf-runtime.md)
|
||||||
|
|||||||
10
CONTEXT-MAP.md
Normal file
10
CONTEXT-MAP.md
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
# Context map
|
||||||
|
|
||||||
|
Multi-context layout is not yet split. Use the root domain vocabulary:
|
||||||
|
|
||||||
|
- **[CONTEXT.md](../CONTEXT.md)** — ubiquitous language for the distributed inference network
|
||||||
|
- **`docs/adr/`** — system-wide architectural decisions
|
||||||
|
- **`.scratch/<feature>/`** — active feature plans and issues
|
||||||
|
- **`.claude/memory/MEMORY.md`** — agent session index and current workstreams
|
||||||
|
|
||||||
|
Per-context `src/<context>/docs/adr/` ADRs will be added when bounded contexts graduate out of the monorepo packages layout.
|
||||||
16
docs/PRD.md
16
docs/PRD.md
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done (base program US-001…US-035 complete; see `docs/prd.json`. Post-035 work lives in `docs/issues/36+` and `.scratch/`. Payment/settlement superseded by ADR-0015; fraud by ADR-0018.)
|
||||||
|
|
||||||
# Distributed Inference Network — PRD
|
# Distributed Inference Network — PRD
|
||||||
|
|
||||||
@@ -8,7 +8,7 @@ Running large language models requires expensive dedicated hardware that most pe
|
|||||||
|
|
||||||
## Solution
|
## Solution
|
||||||
|
|
||||||
A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in SOL or USDC; node operators earn our native token. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
|
A volunteer GPU network where anyone can share their GPU by running a single command and immediately start earning tokens. Nodes each load a shard of a large model; a tracker routes inference requests through the optimal chain of nodes whose shards collectively cover all layers. Developers access the network through an OpenAI-compatible API — a one-line change from any existing LLM integration. Clients pay in **USDT** (alpha: devnet mock-USDT; production: mainnet USDT). Node operators earn USDT payouts from the custodial treasury (ADR-0015); the TAI reward token (ADR-0002) remains deferred. Everything is auto-configured: GPU detection, shard download, wallet creation, and network registration happen automatically on first start.
|
||||||
|
|
||||||
## User Stories
|
## User Stories
|
||||||
|
|
||||||
@@ -76,8 +76,8 @@ The codebase is organized as a Python monorepo with the following top-level pack
|
|||||||
- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement)
|
- `packages/contracts` — Solana L2 smart contracts (stake, slash, strike, ban, settlement)
|
||||||
- `packages/p2p` — P2P gossip layer and shard swarm seeding
|
- `packages/p2p` — P2P gossip layer and shard swarm seeding
|
||||||
|
|
||||||
### Inference engine (ADR-0001)
|
### Inference engine (ADR-0001; native GGUF path ADR-0024)
|
||||||
PyTorch with a Petals-style shard pipeline. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.
|
PyTorch with a Petals-style shard pipeline remains the current production backend. A benchmark-gated llama.cpp/GGUF native path is planned in ADR-0024. Each node independently loads its assigned shard from local disk. At inference time, only activation tensors (~8 KB per layer boundary per token) travel between nodes — no model weights cross the network during serving.
|
||||||
|
|
||||||
### Inference route execution
|
### Inference route execution
|
||||||
The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response.
|
The gateway receives a client request, asks the tracker for an inference route (ordered list of node endpoints covering all layers), opens a persistent TCP session to the first node in the route, streams activation tensors through each node in sequence, and returns the final logits as a streaming chat completion response.
|
||||||
@@ -91,11 +91,11 @@ The gateway receives a client request, asks the tracker for an inference route (
|
|||||||
6. Register with tracker (wallet, hardware profile, shard, endpoint)
|
6. Register with tracker (wallet, hardware profile, shard, endpoint)
|
||||||
7. Begin accepting inference connections
|
7. Begin accepting inference connections
|
||||||
|
|
||||||
### Payment flow
|
### Payment flow (ADR-0015 supersedes ADR-0002 settlement mechanics)
|
||||||
Clients pre-fund an API key with SOL/USDC. The gateway records per-request compute attribution. A settlement transaction runs on Solana L2 at the end of each epoch: client balance is debited, node operators receive our native token proportional to layers served, validators receive a reward share. Solana contracts are the authoritative source for all stake, slash, strike, and ban state (ADR-0002).
|
Clients pre-fund an API key with USDT. The tracker meters each request against the off-chain ledger. Periodic settlement batches USDT payouts from the custodial treasury to node operators proportional to work units. Fraud penalties forfeit pending balance (ADR-0018); strike/ban state persists in the tracker registry. TAI token emission remains deferred (ADR-0002 roadmap).
|
||||||
|
|
||||||
### Fraud detection (ADR-0003)
|
### Fraud detection (ADR-0018; historical ADR-0003)
|
||||||
Validators re-run ~5% of completed requests. If a node's output diverges beyond floating-point tolerance from the reference, the validator submits a slash transaction on-chain. Strike count increments. At the configured strike threshold, the wallet is banned on-chain. New wallets complete N unpaid jobs before earning begins.
|
Validators re-run ~5% of completed requests with TOPLOC activation verification. Caught cheaters forfeit pending balance and receive strikes; three strikes bans the wallet. Probation (first N unpaid jobs) remains the anti-sybil re-entry cost.
|
||||||
|
|
||||||
### Tracker architecture (ADR-0004)
|
### Tracker architecture (ADR-0004)
|
||||||
Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state.
|
Centralized tracker service (HTTP + WebSocket) for fast routing. Nodes gossip state via a lightweight P2P layer so the node client can discover routes during tracker outages. Solana is the authoritative source of truth for all incentive-relevant state.
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
# PyTorch over llama.cpp for the inference engine
|
# PyTorch over llama.cpp for the inference engine
|
||||||
|
|
||||||
|
> **Runtime direction update (2026-07-13):** PyTorch/safetensors remains the current production backend and correctness reference. A benchmark-gated native GGUF path is defined in [ADR-0024](0024-distributed-gguf-runtime.md); it does not replace this ADR until release gates pass.
|
||||||
|
|
||||||
We started with llama.cpp RPC as the distributed backend (following kyuz0/amd-strix-halo-toolboxes), but switched to PyTorch with a Petals-style shard pipeline. llama.cpp RPC requires the primary node to load the full model and distribute weights over the network at every session start — for a 70B model that's ~70GB over LAN per launch, making tracker-driven node rebalancing prohibitively expensive. PyTorch/Petals lets each node load its shard independently from local disk; only activations (~8KB per layer boundary per token) cross the network at inference time. PyTorch also has same-day support for new model architectures, training support (required for the planned torrent-style fine-tuning feature), and is the engine Petals itself uses for this exact use case.
|
We started with llama.cpp RPC as the distributed backend (following kyuz0/amd-strix-halo-toolboxes), but switched to PyTorch with a Petals-style shard pipeline. llama.cpp RPC requires the primary node to load the full model and distribute weights over the network at every session start — for a 70B model that's ~70GB over LAN per launch, making tracker-driven node rebalancing prohibitively expensive. PyTorch/Petals lets each node load its shard independently from local disk; only activations (~8KB per layer boundary per token) cross the network at inference time. PyTorch also has same-day support for new model architectures, training support (required for the planned torrent-style fine-tuning feature), and is the engine Petals itself uses for this exact use case.
|
||||||
|
|
||||||
## Considered Options
|
## Considered Options
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
# Optimistic trust with stake slashing and strike-based bans
|
# Optimistic trust with stake slashing and strike-based bans
|
||||||
|
|
||||||
|
> **Settlement update (2026-07-04):** Alpha uses pending-balance forfeiture instead of stake slashing ([ADR-0015](0015-usdt-custodial-settlement.md)). Fraud detection, TOPLOC audits, and persisted reputation are specified in [ADR-0018](0018-fraud-detection-verification-and-reputation.md). The text below is the historical prototype design.
|
||||||
|
|
||||||
All inference responses are trusted by default. Validators re-run a random sample (~5%) of requests on reference nodes and compare outputs. Nodes that fail are slashed (stake reduced). Enough strikes result in a permanent on-chain ban.
|
All inference responses are trusted by default. Validators re-run a random sample (~5%) of requests on reference nodes and compare outputs. Nodes that fail are slashed (stake reduced). Enough strikes result in a permanent on-chain ban.
|
||||||
|
|
||||||
For the prototype, the gateway emits validation events after completed requests. A validation event records the session id, model preset, request messages, observed output, and the route metadata for each node that served the request. The validator samples events with a configurable rate and deterministic seed for tests. Sampled events are re-run against a trusted reference node/reference function; string outputs must match exactly for stub models, while future tensor/model outputs use a configurable floating-point tolerance.
|
For the prototype, the gateway emits validation events after completed requests. A validation event records the session id, model preset, request messages, observed output, and the route metadata for each node that served the request. The validator samples events with a configurable rate and deterministic seed for tests. Sampled events are re-run against a trusted reference node/reference function; string outputs must match exactly for stub models, while future tensor/model outputs use a configurable floating-point tolerance.
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw
|
# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw
|
||||||
|
|
||||||
## Status: Accepted (chat/streaming/styles implemented); routing flaw documented, fix pending
|
## Status: Accepted (chat/streaming/styles and mixed-topology routing fix implemented)
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
@@ -94,7 +94,7 @@ head + full-model downstream is a topology the planner never had to handle befor
|
|||||||
prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to
|
prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to
|
||||||
equal the correct continuation layer.
|
equal the correct continuation layer.
|
||||||
|
|
||||||
### Required fix (not yet implemented)
|
### Required fix (implemented 2026-07-07 — commits `518c259`, `e44abc9`, `1ecc599`; see ADR-0021)
|
||||||
|
|
||||||
1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute
|
1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute
|
||||||
from `start_layer = e + 1` regardless of the downstream node's own `shard_start`
|
from `start_layer = e + 1` regardless of the downstream node's own `shard_start`
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
# ADR-0022: Sharded per-node generation cache for distributed PyTorch routes
|
# ADR-0022: Sharded per-node generation cache for distributed PyTorch routes
|
||||||
|
|
||||||
## Status: Accepted
|
## Status: Superseded — see [0022-sharded-per-node-kv-cache.md](0022-sharded-per-node-kv-cache.md)
|
||||||
|
|
||||||
|
> Draft alternate header names (`X-Meshnet-Cache-Mode`, `X-Meshnet-Seq-Len`) were not implemented. The accepted wire protocol and implementation use `X-Meshnet-Cache` and `X-Meshnet-Past-Len` per the linked ADR.
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,12 @@
|
|||||||
# ADR-0020: Lean Native Distributed GGUF Runtime
|
# ADR-0024: Lean Native Distributed GGUF Runtime
|
||||||
|
|
||||||
Status: Accepted
|
Status: Accepted
|
||||||
Date: 2026-07-13
|
Date: 2026-07-13
|
||||||
|
|
||||||
|
> **Numbering note:** ADR-0020 is reserved for dashboard chat streaming and mixed-topology routing (`docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md`). This record was originally drafted as ADR-0020 in `.scratch/distributed-gguf-runtime/` and renumbered to avoid the collision.
|
||||||
|
>
|
||||||
|
> **Relation to ADR-0001:** PyTorch/safetensors remains the correctness reference and current production backend. This ADR defines a benchmark-gated native GGUF path; it does not revoke ADR-0001 until release gates pass.
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
The project currently uses Transformers/safetensors as its real model execution backend. This provides broad architecture coverage and a correctness reference, but reported and observed consumer CPU/GPU inference performance motivates evaluating llama.cpp/GGML and quantized GGUF.
|
The project currently uses Transformers/safetensors as its real model execution backend. This provides broad architecture coverage and a correctness reference, but reported and observed consumer CPU/GPU inference performance motivates evaluating llama.cpp/GGML and quantized GGUF.
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done
|
||||||
|
|
||||||
# 01 — Monorepo scaffold + single-node smoke test
|
# 01 — Monorepo scaffold + single-node smoke test
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done
|
||||||
|
|
||||||
# 02 — Two-node shard pipeline
|
# 02 — Two-node shard pipeline
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done
|
||||||
|
|
||||||
# 03 — Tracker: node registration + route selection
|
# 03 — Tracker: node registration + route selection
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done
|
||||||
|
|
||||||
# 04 — Node client startup flow (`meshnet-node start`)
|
# 04 — Node client startup flow (`meshnet-node start`)
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done
|
||||||
|
|
||||||
# 05 — OpenAI-compatible gateway
|
# 05 — OpenAI-compatible gateway
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done (on-chain registry mechanics superseded — probation/ban enforcement uses tracker registry + ADR-0015/0018)
|
||||||
|
|
||||||
# 08 — Node probationary period + ban enforcement
|
# 08 — Node probationary period + ban enforcement
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done
|
||||||
|
|
||||||
# 09 — P2P shard swarm
|
# 09 — P2P shard swarm
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
Status: ready-for-agent
|
Status: done
|
||||||
|
|
||||||
# 10 — `meshnet` Python SDK
|
# 10 — `meshnet` Python SDK
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,6 @@
|
|||||||
# US-019 — Binary data plane and optional peer weight transfer
|
# US-019 — Binary data plane and optional peer weight transfer
|
||||||
|
|
||||||
Status: needs-triage
|
Status: done (design parking lot; binary activation path shipped in US-011/US-019)
|
||||||
Priority: Low
|
|
||||||
Stage: Design parking lot
|
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +1,6 @@
|
|||||||
# US-036 — Streamed chat completions over the relay RPC path
|
# US-036 — Streamed chat completions over the relay RPC path
|
||||||
|
|
||||||
Status: planned
|
Status: done (implemented — `_stream_relayed_frames` in `server.py`; verify on public NAT relay before friends-test)
|
||||||
Priority: Critical (blocks public friends-test deployment)
|
|
||||||
Stage: Designed
|
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# US-042 — GGUF/llama.cpp node backend
|
# US-042 — GGUF/llama.cpp node backend
|
||||||
|
|
||||||
Status: planned
|
Status: planned
|
||||||
Priority: High (unlocks big MoE models on volunteer hardware — the pool's core value)
|
Priority: High (whole-model GGUF shortcut; distributed path in [ADR-0024](../adr/0024-distributed-gguf-runtime.md))
|
||||||
Stage: Draft design
|
Stage: Draft design
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
Status: ready-for-agent
|
Status: planned
|
||||||
|
|
||||||
# US-020 - Memory budget, shard slots, and dropout relocation hardening
|
# US-048 — Memory budget, shard slots, and dropout relocation hardening
|
||||||
|
|
||||||
|
> Renumbered from duplicate slot `20` (which belongs to tracker-node-hardening / US-020 in `docs/prd.json`).
|
||||||
|
|
||||||
## Goal
|
## Goal
|
||||||
|
|
||||||
@@ -64,3 +66,4 @@ The current runtime still effectively has one active backend shard per node. A n
|
|||||||
|
|
||||||
- 2026-06-30: Created after implementing the initial registration plumbing in commit `f1e4ed6` (`--memory`, `--max-shards`, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.
|
- 2026-06-30: Created after implementing the initial registration plumbing in commit `f1e4ed6` (`--memory`, `--max-shards`, tracker validation). This issue captures the remaining end-to-end behavior so it does not conflict with US-013.
|
||||||
- 2026-06-30: Implementation decision: `max_loaded_shards` is currently a validated and exposed capacity field, but multi-range assignment remains reserved because `TorchNodeServer` serves one active backend shard. The tracker therefore emits at most one active range per node while exposing `vram_bytes`, `ram_bytes`, `max_loaded_shards`, quantization, throughput, and computed `max_assignable_layers` in inspection endpoints.
|
- 2026-06-30: Implementation decision: `max_loaded_shards` is currently a validated and exposed capacity field, but multi-range assignment remains reserved because `TorchNodeServer` serves one active backend shard. The tracker therefore emits at most one active range per node while exposing `vram_bytes`, `ram_bytes`, `max_loaded_shards`, quantization, throughput, and computed `max_assignable_layers` in inspection endpoints.
|
||||||
|
- 2026-07-13: Renumbered from `docs/issues/20-memory-budget-…` to resolve duplicate issue slot 20.
|
||||||
Reference in New Issue
Block a user