wip -more responsive UI, better routing
This commit is contained in:
Dobromir Popov
@@ -9,6 +9,8 @@ Pre-release alpha audit + grilling (2026-07-04). Bucket 1 trust-boundary blocker
|
||||
|
||||
Locked scope: one settlement tracker, open node join, devnet mock-USDT, reputation carries forward → fraud must be bounded. See [ADR-0016](../../docs/adr/0016-alpha-scope-and-known-limitations.md).
|
||||
|
||||
**Resume task (2026-07-07):** [24 - Routing telemetry resume](./issues/24-routing-telemetry-resume.md) is `ready-for-agent`. Learned-routing commit `518c259` is already present; dirty tree contains current-request heartbeat/dashboard telemetry and a known import-time annotation crash in `server.py:1490`.
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Path | Status |
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
Status: ready-for-agent
|
||||
|
||||
Scoped 2026-07-07 from an interrupted Claude session. This is a resume/cleanup task for routing and live-progress work that is partly committed and partly left dirty in the working tree.
|
||||
|
||||
# 24 - Finish learned-routing telemetry and live-progress cleanup
|
||||
|
||||
## Current state
|
||||
|
||||
The main dynamic routing feature is already committed at `518c259` (`routing improvements - dynamic (wip)`):
|
||||
|
||||
- `packages/tracker/meshnet_tracker/routing_stats.py` - decayed-EWMA route stats store, epsilon-greedy route selection, diagnostics.
|
||||
- `packages/tracker/meshnet_tracker/server.py` - route enumeration per head, bandit selection in the chat proxy, epoch bumps on node join/leave, `/v1/routing`, route sample recording with 8-token hygiene.
|
||||
- `packages/tracker/meshnet_tracker/cli.py` - `--route-explore-share`, `--route-weight-alpha`, `--route-stats-half-life` and env vars.
|
||||
- `packages/tracker/meshnet_tracker/dashboard.html` - "Routing (learned)" panel.
|
||||
- `docs/adr/0021-dynamic-statistical-routing.md` - design record.
|
||||
- `tests/test_dynamic_routing.py` - includes the exact GPU(0-21)+CPU(0-39) topology, hybrid downstream `start_layer=22`, 0.6/0.4 traffic split for a 1.5 TPS ratio, and scout-rate behavior.
|
||||
|
||||
The current working tree still has uncommitted follow-up work:
|
||||
|
||||
- `packages/node/meshnet_node/torch_server.py` - tracks in-flight chat requests, exposes `TorchNodeServer.current_requests`, prints generation progress with TPS.
|
||||
- `packages/node/meshnet_node/startup.py` - sends `current_requests` in heartbeat payloads and increases heartbeat cadence while busy.
|
||||
- `packages/tracker/meshnet_tracker/server.py` - accepts heartbeat `current_requests`, includes them in `/v1/network/map`, and logs `proxy connecting` before upstream connection.
|
||||
- `packages/tracker/meshnet_tracker/dashboard.html` - enriches the call wall from heartbeat `current_requests` so active requests remain visible even before terminal proxy events.
|
||||
- `tests/test_real_model_backend.py` and `tests/test_tracker_routing.py` - targeted coverage for current-request snapshots, heartbeat sanitization/storage, and TPS progress logging.
|
||||
- `QUICKSTART.md` - documents optional linear-attention fast-path packages for Qwen3.5/3.6 GPU nodes.
|
||||
|
||||
There is also an untracked local file, `.claude/settings.local.json`, which should not be included unless the owner explicitly wants local Claude settings committed.
|
||||
|
||||
## Known blocker found during resume
|
||||
|
||||
Targeted pytest currently fails during import before reaching the new tests:
|
||||
|
||||
```text
|
||||
TypeError: unsupported operand type(s) for |: 'builtin_function_or_method' and 'NoneType'
|
||||
```
|
||||
|
||||
Immediate cause: `packages/tracker/meshnet_tracker/server.py:1490` annotates `ws_lock: threading.Lock | None = None`. `threading.Lock` is a factory function at runtime, not a type, so `| None` evaluates eagerly and crashes. This exists on `HEAD` too, not just in the dirty telemetry changes.
|
||||
|
||||
Fix options:
|
||||
|
||||
- Add `from __future__ import annotations` at the top of `server.py`, then run enough tests to catch any annotation side effects.
|
||||
- Or change that annotation to a safe runtime type such as `Any | None` / remove the union annotation. Keep the change minimal.
|
||||
|
||||
## What to do next
|
||||
|
||||
1. Fix the import-time `threading.Lock | None` crash.
|
||||
2. Re-run the targeted tests:
|
||||
|
||||
```bash
|
||||
.\.venv\Scripts\python.exe -m pytest tests/test_tracker_routing.py::test_tracker_heartbeat_stores_current_requests tests/test_tracker_routing.py::test_normalize_current_requests_sanitizes_payload tests/test_real_model_backend.py::test_current_requests_snapshot_while_generating tests/test_real_model_backend.py::test_distributed_generating_log_includes_tps -q
|
||||
```
|
||||
|
||||
3. Run the relevant routing regression tests:
|
||||
|
||||
```bash
|
||||
.\.venv\Scripts\python.exe -m pytest tests/test_dynamic_routing.py tests/test_tracker_routing.py -q
|
||||
```
|
||||
|
||||
4. If practical, run the non-integration suite:
|
||||
|
||||
```bash
|
||||
.\.venv\Scripts\python.exe -m pytest tests/ -q -m "not integration"
|
||||
```
|
||||
|
||||
5. Confirm or document the pre-existing failure from the interrupted session: `test_proxy_chat_splits_payout_by_tracker_assigned_route_span` reportedly failed on `HEAD` too and was unrelated.
|
||||
6. Commit the intentional work in two commits if it remains naturally split:
|
||||
- learned routing is already committed in `518c259`; leave it alone unless fixing regressions there.
|
||||
- commit the live-progress/current-request telemetry cleanup separately after tests pass.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Importing `meshnet_tracker.server` no longer crashes on the lock annotation.
|
||||
- [ ] Current-request heartbeat payloads are sanitized and surfaced in `/v1/network/map`.
|
||||
- [ ] Node-side in-flight chat snapshots report request id, model, token count, elapsed seconds, tokens/sec, and routing completion.
|
||||
- [ ] Dashboard call wall can show active requests from heartbeat data, not only tracker console terminal events.
|
||||
- [ ] Targeted telemetry tests pass.
|
||||
- [ ] Dynamic routing tests still pass, including GPU(0-21)+CPU(0-39) hybrid-route enumeration and traffic split behavior.
|
||||
- [ ] Full or non-integration suite result is recorded; unrelated pre-existing failures are named explicitly.
|
||||
- [ ] `.claude/settings.local.json` remains uncommitted unless intentionally approved.
|
||||
|
||||
## ADR links
|
||||
|
||||
- [ADR-0020](../../docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md)
|
||||
- [ADR-0021](../../docs/adr/0021-dynamic-statistical-routing.md)
|
||||
|
||||
## Blocked by
|
||||
|
||||
None. The import-time annotation crash is the first fix.
|
||||
|
||||
## Blocks
|
||||
|
||||
Clean handoff/commit of the interrupted live routing progress work.
|
||||
@@ -483,9 +483,29 @@
|
||||
"notes": "Source issue: .scratch/alpha-hardening/issues/23-dynamic-hf-pricing.md. High priority, ship-soon for launch, NOT an alpha-release blocker (unlike AH-021).",
|
||||
"dependsOn": [],
|
||||
"completionNotes": "Completed by agent"
|
||||
},
|
||||
{
|
||||
"id": "AH-024",
|
||||
"title": "24 - Finish learned-routing telemetry and live-progress cleanup",
|
||||
"description": "Status: ready-for-agent\n\nScoped 2026-07-07 from an interrupted Claude session. The learned-routing feature is already committed at 518c259 (`routing improvements - dynamic (wip)`): routing_stats.py, tracker route enumeration and bandit selection, CLI routing flags, `/v1/routing`, dashboard Routing (learned), ADR-0021, and tests/test_dynamic_routing.py including the GPU(0-21)+CPU(0-39) hybrid topology. The dirty working tree contains follow-up live-progress/current-request telemetry in torch_server.py, startup.py, tracker server/dashboard, tests, and QUICKSTART. Known blocker found during resume: importing meshnet_tracker.server currently crashes at `server.py:1490` because `ws_lock: threading.Lock | None = None` evaluates `threading.Lock` as a factory function, not a type. Fix that first, then verify and commit the telemetry cleanup separately from the already-committed dynamic-routing work. Leave `.claude/settings.local.json` uncommitted unless explicitly approved.\n\nSource issue has exact file list, commands, and the reported pre-existing unrelated failure (`test_proxy_chat_splits_payout_by_tracker_assigned_route_span`).",
|
||||
"acceptanceCriteria": [
|
||||
"Importing `meshnet_tracker.server` no longer crashes on the lock annotation",
|
||||
"Current-request heartbeat payloads are sanitized and surfaced in `/v1/network/map`",
|
||||
"Node-side in-flight chat snapshots report request id, model, token count, elapsed seconds, tokens/sec, and routing completion",
|
||||
"Dashboard call wall can show active requests from heartbeat data, not only tracker console terminal events",
|
||||
"Targeted telemetry tests pass",
|
||||
"Dynamic routing tests still pass, including GPU(0-21)+CPU(0-39) hybrid-route enumeration and traffic split behavior",
|
||||
"Full or non-integration suite result is recorded; unrelated pre-existing failures are named explicitly",
|
||||
"`.claude/settings.local.json` remains uncommitted unless intentionally approved"
|
||||
],
|
||||
"priority": 24,
|
||||
"passes": true,
|
||||
"notes": "Source issue: .scratch/alpha-hardening/issues/24-routing-telemetry-resume.md. Resume task for interrupted 2026-07-07 Claude session; first known fix is server.py:1490 annotation crash.",
|
||||
"dependsOn": [],
|
||||
"completionNotes": ""
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"updatedAt": "2026-07-06T06:01:25.474Z"
|
||||
"updatedAt": "2026-07-07T21:30:00.000Z"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user