Files

Dobromir Popov 2b439e8a5f docs: add US-020–029 issue files, ADR 0011–0014, update prd.json to 29/29

Issue files (.scratch/issues/20-29): retrospective specs for all work
done in the current sprint — hardening, route-timeout, start-layer
protocol, heartbeat stats, availability map, rolling RPM, smart
assignment, throughput routing, routing tests, relay outbound client.

ADRs (docs/adr/0011-0014):
  0011 — Auto-shard from memory budget and tracker network assignment
  0012 — X-Meshnet-Start-Layer overlapping shard execution protocol
  0013 — Rolling RPM statistics, smart assignment scoring, throughput routing
  0014 — Relay outbound client for NAT/internet pipeline hops

prd.json: US-020 through US-029 added, all marked done. ralph_progress.py
now shows 29/29 complete (100%).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-30 22:15:41 +03:00

2.8 KiB

Raw Blame History

ADR-0011: Auto-shard from memory budget and tracker-managed network assignment

Status: Accepted

Context

Early node startup required explicit --shard-start and --shard-end flags. This is fine for expert operators but a barrier to new participants who don't know how many layers their GPU can hold. Two improvements were needed:

Auto-detect shard range: fetch num_hidden_layers from the model's config.json and compute how many layers fit in available VRAM.
Network-aware assignment: instead of each node picking its own shard, the tracker knows the current coverage map and can tell the node which gap to fill.

Decisions

1. Layer count from HuggingFace config

AutoConfig.from_pretrained(model_id) downloads only config.json (~1 KB, no weights). cfg.num_hidden_layers gives the total layer count. The node uses this to set shard_end = num_layers - 1 when no explicit range is given.

A curated MODEL_CATALOG in model_catalog.py provides layer counts for common models without any network call — HuggingFace is only hit for uncatalogued repos.

2. VRAM-aware shard sizing

hardware.detect_hardware() returns vram_mb. The node sends this to /v1/network/assign?device=cuda&vram_mb=<n>&hf_repo=<repo>. The tracker responds with a {shard_start, shard_end} gap that fits within the reported VRAM budget using the bytes_per_layer table from the model preset.

When the tracker has no registered nodes for the model yet, gap_found: false is returned and the node defaults to the full model.

3. --memory override

--memory MB allows overriding the detected VRAM. Useful for CPU nodes (which report 0 VRAM) that want to serve a specific slice using system RAM.

4. Tracker network assignment endpoint

GET /v1/network/assign replaces the old GET /v1/nodes/assign. It accepts device, vram_mb, and optionally hf_repo. It returns:

{
  "hf_repo": "Qwen/Qwen2.5-0.5B-Instruct",
  "shard_start": 12,
  "shard_end": 23,
  "num_layers": 24,
  "gap_found": true,
  "price_per_token": 0.0
}

price_per_token is reserved at 0.0 for future billing integration.

Alternatives rejected

Fixed shard table per model: would require updating the code for every new model. HuggingFace config fetch is more general.

Node computes its own gap: requires the node to know the full coverage map. The tracker already has this; having the tracker compute the assignment is cleaner.

Consequences

Nodes can join the network with a single command: meshnet-node start --tracker <url>
The tracker is now the authoritative source for shard assignment
VRAM budgets are advisory — nodes can still pin a range with explicit flags
price_per_token: 0.0 is a stable protocol field; future billing sets it to a real value

2.8 KiB Raw Blame History