b02e07d308
ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression, 128-token chunked prefill; replaces base64 JSON (~33% overhead removed). ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node — any node serving layers[0..k] becomes the inference entry point for that model; bin-packing fills all coverage gaps before adding redundancy; tracker issues LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization. US-011: binary wire format migration US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2) US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment US-014: tracker-as-node (tracker node serves first layers + handles client requests) CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
51 lines
3.0 KiB
Markdown
51 lines
3.0 KiB
Markdown
# US-011 — Binary activation wire format
|
|
|
|
Replace the base64-encoded JSON activation payload from US-002 with a binary HTTP body, zstd compression, and chunked prefill. This is a protocol migration that must be applied to all nodes and the gateway before US-012 (real model backend) can be built on a sane foundation.
|
|
|
|
## Context
|
|
|
|
The current wire format encodes activation tensors as base64 strings inside a JSON dict:
|
|
```json
|
|
{"shape": [1, 1, 64], "dtype": "float32", "data": "<base64>", "context": {"prompt": "..."}}
|
|
```
|
|
This works for stub nodes (tiny tensors) but is unsuitable for real models:
|
|
- base64 adds 33% size overhead
|
|
- A 2048-token prefill at hidden_dim=16384 generates 64MB of activations per boundary — 85MB base64
|
|
- JSON parsing overhead grows with payload size
|
|
|
|
The new format per ADR-0008:
|
|
```
|
|
POST /forward HTTP/1.1
|
|
Content-Type: application/octet-stream
|
|
X-Meshnet-Shape: 1,128,16384
|
|
X-Meshnet-Dtype: bfloat16
|
|
X-Meshnet-Session: <uuid>
|
|
X-Meshnet-Chunk-Index: 0
|
|
X-Meshnet-Chunk-Total: 16
|
|
X-Meshnet-Encoding: zstd
|
|
Content-Length: <n>
|
|
|
|
<raw zstd-compressed little-endian bfloat16 bytes>
|
|
```
|
|
|
|
Stub nodes still emit zeroed tensors — just in binary now. No real model required for this story.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- `packages/node` `/forward` endpoint reads `X-Meshnet-Shape`, `X-Meshnet-Dtype`, `X-Meshnet-Session`, `X-Meshnet-Chunk-Index`, `X-Meshnet-Chunk-Total`, and `X-Meshnet-Encoding` headers; reads raw body bytes; decompresses if `zstd`; reconstructs tensor as numpy array
|
|
- Node forward handler returns a binary response with the same header set reflecting output shape
|
|
- The gateway sends binary chunked activations to the first node; reassembles binary responses from the last node
|
|
- Chunked prefill: the gateway splits input sequences longer than `MESHNET_CHUNK_TOKENS` (default 128) into N chunks and sends them sequentially through the pipeline; stub nodes pass through each chunk unchanged
|
|
- An integration test sends a 512-token stub activation (4 chunks of 128) through a two-node pipeline and asserts all 4 chunk responses are received with correct headers
|
|
- `zstd` Python package added as a dependency to `packages/node` and `packages/gateway`
|
|
- The old `_make_stub_activations` function in `server.py` is replaced with `_make_stub_binary_activation(shape, dtype) -> bytes`
|
|
- `python -m pytest` passes from repo root
|
|
- Commit only this story's changes
|
|
|
|
## Implementation Notes
|
|
|
|
- Use `numpy` to pack/unpack tensor bytes: `np.frombuffer(body, dtype=np.float16).reshape(shape)` (bfloat16 → numpy uses `np.dtype('bfloat16')` in numpy >= 1.20, or load as uint16 and reinterpret)
|
|
- zstd: `import zstandard as zstd; cctx = zstd.ZstdCompressor(level=1); compressed = cctx.compress(raw_bytes)`
|
|
- Chunk index + total allow the receiving node to know whether this is a mid-sequence chunk (no special handling needed for stub; real model in US-012 will need to manage KV cache across chunks)
|
|
- Wire version header `X-Meshnet-Wire: 2` can be added for debugging mismatches between old and new nodes
|