2b439e8a5f
Issue files (.scratch/issues/20-29): retrospective specs for all work done in the current sprint — hardening, route-timeout, start-layer protocol, heartbeat stats, availability map, rolling RPM, smart assignment, throughput routing, routing tests, relay outbound client. ADRs (docs/adr/0011-0014): 0011 — Auto-shard from memory budget and tracker network assignment 0012 — X-Meshnet-Start-Layer overlapping shard execution protocol 0013 — Rolling RPM statistics, smart assignment scoring, throughput routing 0014 — Relay outbound client for NAT/internet pipeline hops prd.json: US-020 through US-029 added, all marked done. ralph_progress.py now shows 29/29 complete (100%). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2.6 KiB
ADR-0012: X-Meshnet-Start-Layer protocol for overlapping shard execution
Status: Accepted
Context
The greedy route-selection algorithm picks a minimal set of nodes whose shard ranges collectively cover all model layers. This is exact when shard ranges are disjoint (node A: 0–11, node B: 12–23). But two nodes with overlapping ranges can also cover the full model (node A: 0–15, node B: 10–23).
Without coordination, node B would re-run layers 10–15 on top of an activation tensor that already has those layers applied — producing silently wrong output.
The question is: who resolves the overlap, and how?
Options considered
A. Tracker injects start_layer per hop (chosen)
The tracker knows the full route when it builds X-Meshnet-Route. It computes
covered_up_to as it walks the route and sets start_layer = covered_up_to + 1
for each subsequent hop. The head node forwards this per-hop in
X-Meshnet-Start-Layer. No peer-to-peer negotiation needed.
B. Each node negotiates with the next
Node A would tell node B "I ran layers 0–15, you start from 16". This requires
node A to know node B's shard range, which means an extra tracker lookup or
exposing shard metadata in the activation wire protocol.
C. Strict non-overlapping enforcement
Reject any route that contains overlapping nodes. Simpler but limits redundancy:
two nodes with the same shard can't form a route even if their combined coverage
is complete.
Decision
Option A. The tracker is already the central coordinator; it already knows every
node's shard range. Injecting start_layer at route-build time costs nothing and
keeps the node implementation simple.
Wire protocol
X-Meshnet-Route (JSON array, injected by tracker into the first-hop request):
[
{"endpoint": "http://node-b:7002", "start_layer": 12, "relay_addr": null},
{"endpoint": "http://node-c:7003", "start_layer": 20}
]
X-Meshnet-Start-Layer (integer header, forwarded by head node to each downstream hop):
X-Meshnet-Start-Layer: 12
The receiving node passes start_layer to backend.forward_bytes(start_layer=12).
The model shard skips transformer blocks below index 12.
Consequences
- Overlapping shard registrations are valid and useful for redundancy
- Route selection does not need to enforce disjoint ranges
- The tracker carries the full route context; nodes are stateless w.r.t. routing
start_layermust be preserved through the relay path (included in hop dict)- Backward compatibility: if
start_layeris absent, the node runs from its registeredshard_start