docs: consolidate all docs under docs/ — single source of truth
Move issues (01–29) and PRD from .scratch/distributed-inference-network/ into docs/issues/ and docs/. Update ralph_progress.py DEFAULT_PRD path and rewrite docs/agents/issue-tracker.md to reflect the new layout. The distributed_inference_network.egg-info/docs/ mirror is a build artifact already covered by *.egg-info/ in .gitignore — not committed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
53
docs/issues/13-coverage-first-shard-assignment.md
Normal file
53
docs/issues/13-coverage-first-shard-assignment.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# US-013 — Coverage-first tracker shard assignment
|
||||
|
||||
Upgrade the tracker's route selection to a coverage-first, speed-weighted bin-packing algorithm. The tracker maintains a live coverage map per model, issues rebalance directives when gaps appear, and assigns shard ranges to nodes based on declared VRAM, quantization, and benchmark throughput.
|
||||
|
||||
## Context
|
||||
|
||||
Per ADR-0009:
|
||||
- **Coverage gap** = any layer range in a model with zero serving nodes → model is unroutable
|
||||
- **Coverage-first**: fill all gaps before adding redundancy to covered ranges
|
||||
- **Speed-weighted bin-packing**: fill fastest nodes to their VRAM limit first; cascade to next-fastest
|
||||
- **Continuous rebalancing**: triggered every 30s and on every node join/leave event
|
||||
- **Rebalance directive**: `LOAD_SHARD` or `DROP_SHARD` instruction sent from tracker to node
|
||||
|
||||
Node registration now includes capability data:
|
||||
```json
|
||||
{
|
||||
"wallet": "...",
|
||||
"endpoint": "http://10.0.0.5:7000",
|
||||
"vram_bytes": 25769803776,
|
||||
"ram_bytes": 137438953472,
|
||||
"quantizations": ["nf4", "int8", "bfloat16"],
|
||||
"benchmark_tokens_per_sec": 12.4,
|
||||
"benchmark_model": "openai-community/gpt2"
|
||||
}
|
||||
```
|
||||
|
||||
Model preset metadata (stored in tracker config) includes `bytes_per_layer` at each quantization level and `total_layers`. The tracker can compute max assignable layers for any node.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Tracker accepts `vram_bytes`, `ram_bytes`, `quantizations`, `benchmark_tokens_per_sec` in the node registration payload
|
||||
- `GET /v1/coverage/<model_preset>` returns the coverage map: list of `{start_layer, end_layer, node_count}` for each assigned range
|
||||
- A model is only routable when all layer ranges have `node_count >= 1`
|
||||
- When a new node registers, the tracker assigns it to the highest-priority uncovered range (or expands the most-loaded range if fully covered)
|
||||
- When a node disconnects (heartbeat timeout), the tracker marks affected ranges as reduced-coverage and, if any reach 0, issues `LOAD_SHARD` directives to idle nodes within 30 seconds
|
||||
- The shard assignment respects VRAM: `assigned_layers <= floor((vram_bytes * 0.8) / bytes_per_layer_at_quant)`
|
||||
- Speed weighting: given two idle nodes both capable of covering a gap, the tracker assigns the wider sub-range to the faster node
|
||||
- An integration test: start tracker with a model preset, register three nodes with different VRAM, assert the coverage map shows 100% coverage and the largest VRAM node received the widest shard range
|
||||
- A second integration test: kill the node covering the middle layer range; assert tracker issues a `LOAD_SHARD` directive to an idle node and coverage recovers
|
||||
- `python -m pytest` passes
|
||||
- Commit only this story's changes
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Coverage map data structure: `dict[str, list[tuple[int, int, int]]]` keyed by model preset
|
||||
- Bin-packing is run in-process on the tracker (not a separate service); it's triggered by registration/heartbeat events and a 30s periodic timer
|
||||
- Rebalance directives are delivered as responses to node heartbeat POSTs (node polls tracker every 10s anyway) — no new push channel needed
|
||||
- `bytes_per_layer` is read from a `model_presets.json` config file; add GPT-2 (12 layers, ~30MB/layer bfloat16) as the test preset
|
||||
- The current tracker `_select_route` function is extended, not replaced — backward compat with stub nodes that don't send VRAM data (default to 8GB / bfloat16 if omitted)
|
||||
|
||||
## Comments
|
||||
|
||||
- 2026-06-30: Follow-up capacity hardening lives in `20-memory-budget-shard-slots-and-dropout-relocation.md`. US-013 remains the base coverage-first assignment and dropout rebalance story; US-020 owns operator `--memory`, `--max-shards`, shard-slot enforcement, and relocation limit hardening so the two scopes do not conflict.
|
||||
Reference in New Issue
Block a user