neuron-tai/docs/issues/11-binary-wire-format.md

US-011 — Binary activation wire format

Replace the base64-encoded JSON activation payload from US-002 with a binary HTTP body, zstd compression, and chunked prefill. This is a protocol migration that must be applied to all nodes and the gateway before US-012 (real model backend) can be built on a sane foundation.

Context

The current wire format encodes activation tensors as base64 strings inside a JSON dict:

{"shape": [1, 1, 64], "dtype": "float32", "data": "<base64>", "context": {"prompt": "..."}}

This works for stub nodes (tiny tensors) but is unsuitable for real models:

• base64 adds 33% size overhead
• A 2048-token prefill at hidden_dim=16384 generates 64MB of activations per boundary — 85MB base64
• JSON parsing overhead grows with payload size

The new format per ADR-0008:

POST /forward HTTP/1.1
Content-Type: application/octet-stream
X-Meshnet-Shape: 1,128,16384
X-Meshnet-Dtype: bfloat16
X-Meshnet-Session: <uuid>
X-Meshnet-Chunk-Index: 0
X-Meshnet-Chunk-Total: 16
X-Meshnet-Encoding: zstd
Content-Length: <n>

<raw zstd-compressed little-endian bfloat16 bytes>

Stub nodes still emit zeroed tensors — just in binary now. No real model required for this story.

Acceptance Criteria

• packages/node /forward endpoint reads X-Meshnet-Shape, X-Meshnet-Dtype, X-Meshnet-Session, X-Meshnet-Chunk-Index, X-Meshnet-Chunk-Total, and X-Meshnet-Encoding headers; reads raw body bytes; decompresses if zstd; reconstructs tensor as numpy array
• Node forward handler returns a binary response with the same header set reflecting output shape
• The gateway sends binary chunked activations to the first node; reassembles binary responses from the last node
• Chunked prefill: the gateway splits input sequences longer than MESHNET_CHUNK_TOKENS (default 128) into N chunks and sends them sequentially through the pipeline; stub nodes pass through each chunk unchanged
• An integration test sends a 512-token stub activation (4 chunks of 128) through a two-node pipeline and asserts all 4 chunk responses are received with correct headers
• zstd Python package added as a dependency to packages/node and packages/gateway
• The old _make_stub_activations function in server.py is replaced with _make_stub_binary_activation(shape, dtype) -> bytes
• python -m pytest passes from repo root
• Commit only this story's changes

Implementation Notes

• Use numpy to pack/unpack tensor bytes: np.frombuffer(body, dtype=np.float16).reshape(shape) (bfloat16 → numpy uses np.dtype('bfloat16') in numpy >= 1.20, or load as uint16 and reinterpret)
• zstd: import zstandard as zstd; cctx = zstd.ZstdCompressor(level=1); compressed = cctx.compress(raw_bytes)
• Chunk index + total allow the receiving node to know whether this is a mid-sequence chunk (no special handling needed for stub; real model in US-012 will need to manage KV cache across chunks)
• Wire version header X-Meshnet-Wire: 2 can be added for debugging mismatches between old and new nodes