b02e07d308
ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression, 128-token chunked prefill; replaces base64 JSON (~33% overhead removed). ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node — any node serving layers[0..k] becomes the inference entry point for that model; bin-packing fills all coverage gaps before adding redundancy; tracker issues LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization. US-011: binary wire format migration US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2) US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment US-014: tracker-as-node (tracker node serves first layers + handles client requests) CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
49 lines
3.3 KiB
Markdown
49 lines
3.3 KiB
Markdown
# Binary activation wire format with zstd compression and chunked prefill
|
||
|
||
Activation tensors between shard nodes are sent as raw binary (little-endian bfloat16 bytes) over HTTP with metadata in request headers and optional zstd body compression. The previous base64-encoded JSON format is retired.
|
||
|
||
## Problem
|
||
|
||
Base64 adds 33% size overhead plus CPU cost on both ends. For a 700B model at bfloat16 (hidden_dim=16384), a 2048-token prefill generates 64MB of activation data per shard boundary. At 10 boundaries that is 640MB of transfer just for prefill. Encoding that as base64 JSON makes it ~860MB plus repeated parse overhead.
|
||
|
||
## Decision
|
||
|
||
**Control plane** (registration, heartbeat, route selection, health) stays JSON over HTTP — these are small and infrequent.
|
||
|
||
**Activation plane** (node-to-node tensor transfer) uses a binary HTTP body:
|
||
|
||
```
|
||
POST /forward HTTP/1.1
|
||
Content-Type: application/octet-stream
|
||
X-Meshnet-Shape: 1,128,16384
|
||
X-Meshnet-Dtype: bfloat16
|
||
X-Meshnet-Session: <uuid>
|
||
X-Meshnet-Chunk-Index: 0
|
||
X-Meshnet-Chunk-Total: 16
|
||
X-Meshnet-Encoding: zstd
|
||
Content-Length: <compressed_byte_count>
|
||
|
||
<raw zstd-compressed little-endian bfloat16 tensor bytes>
|
||
```
|
||
|
||
**Chunked prefill**: long prompts are split into 128-token chunks before the first forward pass. Each chunk is sent and forwarded independently through the pipeline. This keeps peak transfer per boundary at ~4MB (128 × 16384 × 2 bytes) rather than 64MB+ for a full 2048-token prefill. Generation (one token at a time) is already naturally chunked.
|
||
|
||
**Compression**: zstd level 1 (near-lz4 speed, better ratio). bfloat16 hidden states post-LayerNorm have high compressibility (many near-zero values). Expected 2–4× reduction on typical activations. `X-Meshnet-Encoding` is optional — omit for uncompressed binary; receiving nodes must handle both.
|
||
|
||
**Activation dtype**: always bfloat16 at every shard boundary, regardless of the weight quantization a node uses internally. Weights may be NF4 or INT8; computation upcasts to bfloat16; the result crosses the wire in bfloat16. This avoids dtype negotiation between nodes with different quantization tiers.
|
||
|
||
## Considered Options
|
||
|
||
- **base64 JSON**: existing format, works everywhere, ~33% overhead, no streaming, high CPU cost — retired
|
||
- **msgpack with embedded binary**: compact metadata + binary blob, requires msgpack dep on all nodes — rejected (HTTP headers are sufficient for metadata)
|
||
- **Raw TCP with length-prefixed frames**: maximally efficient, no HTTP overhead — rejected (breaks existing HTTP server infrastructure and complicates firewall traversal)
|
||
- **Binary HTTP body with headers**: chosen — no new protocol, standard HTTP chunked transfer, works with existing server framework, easy to debug with curl
|
||
|
||
## Consequences
|
||
|
||
- Node forward endpoint changes from `POST /forward` with JSON body to `POST /forward` with binary body and header metadata
|
||
- All nodes must be updated together (wire format version mismatch = 400 error with `X-Meshnet-Wire-Version` header for debugging)
|
||
- The stub wire format (`base64 JSON dict`) used in US-001 through US-010 is replaced; stub nodes can emit zeroed binary tensors trivially
|
||
- Chunk boundaries in prefill must align to token positions (no mid-token splits)
|
||
- KV cache invalidation across chunks is the calling node's responsibility; the receiving node treats each chunk as an independent forward pass segment
|