docs: add US-020–029 issue files, ADR 0011–0014, update prd.json to 29/29
Issue files (.scratch/issues/20-29): retrospective specs for all work done in the current sprint — hardening, route-timeout, start-layer protocol, heartbeat stats, availability map, rolling RPM, smart assignment, throughput routing, routing tests, relay outbound client. ADRs (docs/adr/0011-0014): 0011 — Auto-shard from memory budget and tracker network assignment 0012 — X-Meshnet-Start-Layer overlapping shard execution protocol 0013 — Rolling RPM statistics, smart assignment scoring, throughput routing 0014 — Relay outbound client for NAT/internet pipeline hops prd.json: US-020 through US-029 added, all marked done. ralph_progress.py now shows 29/29 complete (100%). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
67
docs/adr/0012-start-layer-overlapping-shards.md
Normal file
67
docs/adr/0012-start-layer-overlapping-shards.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# ADR-0012: X-Meshnet-Start-Layer protocol for overlapping shard execution
|
||||
|
||||
## Status: Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The greedy route-selection algorithm picks a minimal set of nodes whose shard ranges
|
||||
collectively cover all model layers. This is exact when shard ranges are disjoint
|
||||
(node A: 0–11, node B: 12–23). But two nodes with overlapping ranges can also cover
|
||||
the full model (node A: 0–15, node B: 10–23).
|
||||
|
||||
Without coordination, node B would re-run layers 10–15 on top of an activation tensor
|
||||
that already has those layers applied — producing silently wrong output.
|
||||
|
||||
The question is: who resolves the overlap, and how?
|
||||
|
||||
## Options considered
|
||||
|
||||
**A. Tracker injects start_layer per hop (chosen)**
|
||||
The tracker knows the full route when it builds `X-Meshnet-Route`. It computes
|
||||
`covered_up_to` as it walks the route and sets `start_layer = covered_up_to + 1`
|
||||
for each subsequent hop. The head node forwards this per-hop in
|
||||
`X-Meshnet-Start-Layer`. No peer-to-peer negotiation needed.
|
||||
|
||||
**B. Each node negotiates with the next**
|
||||
Node A would tell node B "I ran layers 0–15, you start from 16". This requires
|
||||
node A to know node B's shard range, which means an extra tracker lookup or
|
||||
exposing shard metadata in the activation wire protocol.
|
||||
|
||||
**C. Strict non-overlapping enforcement**
|
||||
Reject any route that contains overlapping nodes. Simpler but limits redundancy:
|
||||
two nodes with the same shard can't form a route even if their combined coverage
|
||||
is complete.
|
||||
|
||||
## Decision
|
||||
|
||||
Option A. The tracker is already the central coordinator; it already knows every
|
||||
node's shard range. Injecting `start_layer` at route-build time costs nothing and
|
||||
keeps the node implementation simple.
|
||||
|
||||
## Wire protocol
|
||||
|
||||
`X-Meshnet-Route` (JSON array, injected by tracker into the first-hop request):
|
||||
|
||||
```json
|
||||
[
|
||||
{"endpoint": "http://node-b:7002", "start_layer": 12, "relay_addr": null},
|
||||
{"endpoint": "http://node-c:7003", "start_layer": 20}
|
||||
]
|
||||
```
|
||||
|
||||
`X-Meshnet-Start-Layer` (integer header, forwarded by head node to each downstream hop):
|
||||
|
||||
```
|
||||
X-Meshnet-Start-Layer: 12
|
||||
```
|
||||
|
||||
The receiving node passes `start_layer` to `backend.forward_bytes(start_layer=12)`.
|
||||
The model shard skips transformer blocks below index 12.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Overlapping shard registrations are valid and useful for redundancy
|
||||
- Route selection does not need to enforce disjoint ranges
|
||||
- The tracker carries the full route context; nodes are stateless w.r.t. routing
|
||||
- `start_layer` must be preserved through the relay path (included in hop dict)
|
||||
- Backward compatibility: if `start_layer` is absent, the node runs from its registered `shard_start`
|
||||
Reference in New Issue
Block a user