2b439e8a5f
Issue files (.scratch/issues/20-29): retrospective specs for all work done in the current sprint — hardening, route-timeout, start-layer protocol, heartbeat stats, availability map, rolling RPM, smart assignment, throughput routing, routing tests, relay outbound client. ADRs (docs/adr/0011-0014): 0011 — Auto-shard from memory budget and tracker network assignment 0012 — X-Meshnet-Start-Layer overlapping shard execution protocol 0013 — Rolling RPM statistics, smart assignment scoring, throughput routing 0014 — Relay outbound client for NAT/internet pipeline hops prd.json: US-020 through US-029 added, all marked done. ralph_progress.py now shows 29/29 complete (100%). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
74 lines
2.8 KiB
Markdown
74 lines
2.8 KiB
Markdown
# ADR-0011: Auto-shard from memory budget and tracker-managed network assignment
|
|
|
|
## Status: Accepted
|
|
|
|
## Context
|
|
|
|
Early node startup required explicit `--shard-start` and `--shard-end` flags. This is
|
|
fine for expert operators but a barrier to new participants who don't know how many layers
|
|
their GPU can hold. Two improvements were needed:
|
|
|
|
1. **Auto-detect shard range**: fetch `num_hidden_layers` from the model's `config.json`
|
|
and compute how many layers fit in available VRAM.
|
|
2. **Network-aware assignment**: instead of each node picking its own shard, the tracker
|
|
knows the current coverage map and can tell the node which gap to fill.
|
|
|
|
## Decisions
|
|
|
|
### 1. Layer count from HuggingFace config
|
|
|
|
`AutoConfig.from_pretrained(model_id)` downloads only `config.json` (~1 KB, no weights).
|
|
`cfg.num_hidden_layers` gives the total layer count. The node uses this to set
|
|
`shard_end = num_layers - 1` when no explicit range is given.
|
|
|
|
A curated `MODEL_CATALOG` in `model_catalog.py` provides layer counts for common models
|
|
without any network call — HuggingFace is only hit for uncatalogued repos.
|
|
|
|
### 2. VRAM-aware shard sizing
|
|
|
|
`hardware.detect_hardware()` returns `vram_mb`. The node sends this to
|
|
`/v1/network/assign?device=cuda&vram_mb=<n>&hf_repo=<repo>`. The tracker responds with
|
|
a `{shard_start, shard_end}` gap that fits within the reported VRAM budget using the
|
|
`bytes_per_layer` table from the model preset.
|
|
|
|
When the tracker has no registered nodes for the model yet, `gap_found: false` is
|
|
returned and the node defaults to the full model.
|
|
|
|
### 3. --memory override
|
|
|
|
`--memory MB` allows overriding the detected VRAM. Useful for CPU nodes (which report 0
|
|
VRAM) that want to serve a specific slice using system RAM.
|
|
|
|
### 4. Tracker network assignment endpoint
|
|
|
|
`GET /v1/network/assign` replaces the old `GET /v1/nodes/assign`. It accepts
|
|
`device`, `vram_mb`, and optionally `hf_repo`. It returns:
|
|
|
|
```json
|
|
{
|
|
"hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
|
|
"shard_start": 12,
|
|
"shard_end": 23,
|
|
"num_layers": 24,
|
|
"gap_found": true,
|
|
"price_per_token": 0.0
|
|
}
|
|
```
|
|
|
|
`price_per_token` is reserved at 0.0 for future billing integration.
|
|
|
|
## Alternatives rejected
|
|
|
|
**Fixed shard table per model**: would require updating the code for every new model.
|
|
HuggingFace config fetch is more general.
|
|
|
|
**Node computes its own gap**: requires the node to know the full coverage map. The
|
|
tracker already has this; having the tracker compute the assignment is cleaner.
|
|
|
|
## Consequences
|
|
|
|
- Nodes can join the network with a single command: `meshnet-node start --tracker <url>`
|
|
- The tracker is now the authoritative source for shard assignment
|
|
- VRAM budgets are advisory — nodes can still pin a range with explicit flags
|
|
- `price_per_token: 0.0` is a stable protocol field; future billing sets it to a real value
|