37 lines
1.8 KiB
Markdown
37 lines
1.8 KiB
Markdown
# AD-001: Deepen Route Session execution behind one node seam
|
|
|
|
- **Status:** needs-triage
|
|
- **Priority:** p0
|
|
- **Dependencies:** none
|
|
- **Blocks:** AD-002
|
|
- **Evidence:** Graphify identifies `torch_server.py` as the Activation Transport & Binary Frames hub; `_TorchHandler._do_chat_completions` has cyclomatic complexity 53 and owns request parsing, complete-model generation, distributed prefill/decode, Hot KV State recovery, transport clients, SSE, telemetry, and cleanup.
|
|
|
|
## Objective
|
|
|
|
Move distributed Route Session execution behind one deep module interface so the HTTP module only translates a client request into a Route Session result/stream.
|
|
|
|
## Constraints
|
|
|
|
- Preserve ADR-0009: the head worker owns tokenization and shard execution.
|
|
- Preserve the existing OpenAI-compatible HTTP/SSE behavior.
|
|
- Keep Hot KV State local to each shard and retain cache-miss re-prefill behavior.
|
|
- Do not introduce native GGUF worker work; DGR-040 and DGR-041 own that scope.
|
|
|
|
## Acceptance criteria
|
|
|
|
- [ ] Characterization tests cover prefill, decode, cache-miss re-prefill, cancellation, and cleanup through the new module interface without an HTTP server.
|
|
- [ ] The HTTP module retains only request translation, response translation, and request accounting.
|
|
- [ ] Route Session lifecycle owns downstream direct/relay client cleanup in one place.
|
|
- [ ] Existing two-node, KV-cache, relay, and OpenAI compatibility tests retain behavior.
|
|
- [ ] `pytest` targeted tests and `python -m compileall packages tests` pass.
|
|
|
|
## Likely files
|
|
|
|
- Modify: `packages/node/meshnet_node/torch_server.py`
|
|
- Create: module adjacent to `torch_server.py` for Route Session execution
|
|
- Modify/add: `tests/test_two_node_pipeline.py`, `tests/test_kv_cache_distributed.py`, focused new tests
|
|
|
|
## Non-goals
|
|
|
|
No change to public route selection, model architecture behavior, native worker protocol, or WAN KV migration.
|