AD-001: Deepen Route Session execution behind one node seam

Status: needs-triage
Priority: p0
Dependencies: none
Blocks: AD-002
Evidence: Graphify identifies torch_server.py as the Activation Transport & Binary Frames hub; _TorchHandler._do_chat_completions has cyclomatic complexity 53 and owns request parsing, complete-model generation, distributed prefill/decode, Hot KV State recovery, transport clients, SSE, telemetry, and cleanup.

Objective

Move distributed Route Session execution behind one deep module interface so the HTTP module only translates a client request into a Route Session result/stream.

Constraints

Preserve ADR-0009: the head worker owns tokenization and shard execution.
Preserve the existing OpenAI-compatible HTTP/SSE behavior.
Keep Hot KV State local to each shard and retain cache-miss re-prefill behavior.
Do not introduce native GGUF worker work; DGR-040 and DGR-041 own that scope.

Acceptance criteria

Characterization tests cover prefill, decode, cache-miss re-prefill, cancellation, and cleanup through the new module interface without an HTTP server.
The HTTP module retains only request translation, response translation, and request accounting.
Route Session lifecycle owns downstream direct/relay client cleanup in one place.
Existing two-node, KV-cache, relay, and OpenAI compatibility tests retain behavior.
pytest targeted tests and python -m compileall packages tests pass.

Likely files

Modify: packages/node/meshnet_node/torch_server.py
Create: module adjacent to torch_server.py for Route Session execution
Modify/add: tests/test_two_node_pipeline.py, tests/test_kv_cache_distributed.py, focused new tests

Non-goals

No change to public route selection, model architecture behavior, native worker protocol, or WAN KV migration.

1.8 KiB Raw Blame History