neuron-tai/.scratch/alpha-hardening/issues/24-routing-telemetry-resume.md

Status: ready-for-agent

Scoped 2026-07-07 from an interrupted Claude session. This is a resume/cleanup task for routing and live-progress work that is partly committed and partly left dirty in the working tree.

24 - Finish learned-routing telemetry and live-progress cleanup

Current state

The main dynamic routing feature is already committed at 518c259 (routing improvements - dynamic (wip)):

• packages/tracker/meshnet_tracker/routing_stats.py - decayed-EWMA route stats store, epsilon-greedy route selection, diagnostics.
• packages/tracker/meshnet_tracker/server.py - route enumeration per head, bandit selection in the chat proxy, epoch bumps on node join/leave, /v1/routing, route sample recording with 8-token hygiene.
• packages/tracker/meshnet_tracker/cli.py - --route-explore-share, --route-weight-alpha, --route-stats-half-life and env vars.
• packages/tracker/meshnet_tracker/dashboard.html - "Routing (learned)" panel.
• docs/adr/0021-dynamic-statistical-routing.md - design record.
• tests/test_dynamic_routing.py - includes the exact GPU(0-21)+CPU(0-39) topology, hybrid downstream start_layer=22, 0.6/0.4 traffic split for a 1.5 TPS ratio, and scout-rate behavior.

The current working tree still has uncommitted follow-up work:

• packages/node/meshnet_node/torch_server.py - tracks in-flight chat requests, exposes TorchNodeServer.current_requests, prints generation progress with TPS.
• packages/node/meshnet_node/startup.py - sends current_requests in heartbeat payloads and increases heartbeat cadence while busy.
• packages/tracker/meshnet_tracker/server.py - accepts heartbeat current_requests, includes them in /v1/network/map, and logs proxy connecting before upstream connection.
• packages/tracker/meshnet_tracker/dashboard.html - enriches the call wall from heartbeat current_requests so active requests remain visible even before terminal proxy events.
• tests/test_real_model_backend.py and tests/test_tracker_routing.py - targeted coverage for current-request snapshots, heartbeat sanitization/storage, and TPS progress logging.
• QUICKSTART.md - documents optional linear-attention fast-path packages for Qwen3.5/3.6 GPU nodes.

There is also an untracked local file, .claude/settings.local.json, which should not be included unless the owner explicitly wants local Claude settings committed.

Known blocker found during resume

Targeted pytest currently fails during import before reaching the new tests:

TypeError: unsupported operand type(s) for |: 'builtin_function_or_method' and 'NoneType'

Immediate cause: packages/tracker/meshnet_tracker/server.py:1490 annotates ws_lock: threading.Lock | None = None. threading.Lock is a factory function at runtime, not a type, so | None evaluates eagerly and crashes. This exists on HEAD too, not just in the dirty telemetry changes.

Fix options:

• Add from __future__ import annotations at the top of server.py, then run enough tests to catch any annotation side effects.
• Or change that annotation to a safe runtime type such as Any | None / remove the union annotation. Keep the change minimal.

What to do next

1. Fix the import-time threading.Lock | None crash.

2. Re-run the targeted tests:

   .\.venv\Scripts\python.exe -m pytest tests/test_tracker_routing.py::test_tracker_heartbeat_stores_current_requests tests/test_tracker_routing.py::test_normalize_current_requests_sanitizes_payload tests/test_real_model_backend.py::test_current_requests_snapshot_while_generating tests/test_real_model_backend.py::test_distributed_generating_log_includes_tps -q
   

3. Run the relevant routing regression tests:

   .\.venv\Scripts\python.exe -m pytest tests/test_dynamic_routing.py tests/test_tracker_routing.py -q
   

4. If practical, run the non-integration suite:

   .\.venv\Scripts\python.exe -m pytest tests/ -q -m "not integration"
   

5. Confirm or document the pre-existing failure from the interrupted session: test_proxy_chat_splits_payout_by_tracker_assigned_route_span reportedly failed on HEAD too and was unrelated.

6. Commit the intentional work in two commits if it remains naturally split:

   • learned routing is already committed in 518c259; leave it alone unless fixing regressions there.
   • commit the live-progress/current-request telemetry cleanup separately after tests pass.

Acceptance criteria

• Importing meshnet_tracker.server no longer crashes on the lock annotation.
• Current-request heartbeat payloads are sanitized and surfaced in /v1/network/map.
• Node-side in-flight chat snapshots report request id, model, token count, elapsed seconds, tokens/sec, and routing completion.
• Dashboard call wall can show active requests from heartbeat data, not only tracker console terminal events.
• Targeted telemetry tests pass.
• Dynamic routing tests still pass, including GPU(0-21)+CPU(0-39) hybrid-route enumeration and traffic split behavior.
• Full or non-integration suite result is recorded; unrelated pre-existing failures are named explicitly.
• .claude/settings.local.json remains uncommitted unless intentionally approved.

ADR links

• ADR-0020
• ADR-0021

Blocked by

None. The import-time annotation crash is the first fix.

Blocks

Clean handoff/commit of the interrupted live routing progress work.