b02e07d308
ADR-0008: binary activation wire format — raw bfloat16 over HTTP, zstd compression, 128-token chunked prefill; replaces base64 JSON (~33% overhead removed). ADR-0009: coverage-first shard assignment and tracker-as-first-layer-node — any node serving layers[0..k] becomes the inference entry point for that model; bin-packing fills all coverage gaps before adding redundancy; tracker issues LOAD_SHARD/DROP_SHARD rebalance directives; nodes declare VRAM + quantization. US-011: binary wire format migration US-012: real PyTorch layer execution (transformers + bitsandbytes, test on GPT-2) US-013: coverage-first tracker bin-packing with VRAM-aware shard assignment US-014: tracker-as-node (tracker node serves first layers + handles client requests) CONTEXT.md: Tracker Node, Coverage Map, Rebalance Directive terms added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3.0 KiB
3.0 KiB
US-011 — Binary activation wire format
Replace the base64-encoded JSON activation payload from US-002 with a binary HTTP body, zstd compression, and chunked prefill. This is a protocol migration that must be applied to all nodes and the gateway before US-012 (real model backend) can be built on a sane foundation.
Context
The current wire format encodes activation tensors as base64 strings inside a JSON dict:
{"shape": [1, 1, 64], "dtype": "float32", "data": "<base64>", "context": {"prompt": "..."}}
This works for stub nodes (tiny tensors) but is unsuitable for real models:
- base64 adds 33% size overhead
- A 2048-token prefill at hidden_dim=16384 generates 64MB of activations per boundary — 85MB base64
- JSON parsing overhead grows with payload size
The new format per ADR-0008:
POST /forward HTTP/1.1
Content-Type: application/octet-stream
X-Meshnet-Shape: 1,128,16384
X-Meshnet-Dtype: bfloat16
X-Meshnet-Session: <uuid>
X-Meshnet-Chunk-Index: 0
X-Meshnet-Chunk-Total: 16
X-Meshnet-Encoding: zstd
Content-Length: <n>
<raw zstd-compressed little-endian bfloat16 bytes>
Stub nodes still emit zeroed tensors — just in binary now. No real model required for this story.
Acceptance Criteria
packages/node/forwardendpoint readsX-Meshnet-Shape,X-Meshnet-Dtype,X-Meshnet-Session,X-Meshnet-Chunk-Index,X-Meshnet-Chunk-Total, andX-Meshnet-Encodingheaders; reads raw body bytes; decompresses ifzstd; reconstructs tensor as numpy array- Node forward handler returns a binary response with the same header set reflecting output shape
- The gateway sends binary chunked activations to the first node; reassembles binary responses from the last node
- Chunked prefill: the gateway splits input sequences longer than
MESHNET_CHUNK_TOKENS(default 128) into N chunks and sends them sequentially through the pipeline; stub nodes pass through each chunk unchanged - An integration test sends a 512-token stub activation (4 chunks of 128) through a two-node pipeline and asserts all 4 chunk responses are received with correct headers
zstdPython package added as a dependency topackages/nodeandpackages/gateway- The old
_make_stub_activationsfunction inserver.pyis replaced with_make_stub_binary_activation(shape, dtype) -> bytes python -m pytestpasses from repo root- Commit only this story's changes
Implementation Notes
- Use
numpyto pack/unpack tensor bytes:np.frombuffer(body, dtype=np.float16).reshape(shape)(bfloat16 → numpy usesnp.dtype('bfloat16')in numpy >= 1.20, or load as uint16 and reinterpret) - zstd:
import zstandard as zstd; cctx = zstd.ZstdCompressor(level=1); compressed = cctx.compress(raw_bytes) - Chunk index + total allow the receiving node to know whether this is a mid-sequence chunk (no special handling needed for stub; real model in US-012 will need to manage KV cache across chunks)
- Wire version header
X-Meshnet-Wire: 2can be added for debugging mismatches between old and new nodes