[verified] fix: preserve tracker precision eligibility

This commit is contained in:
Dobromir Popov
2026-07-13 10:27:45 +03:00
parent 377346c301
commit b5fa7245df
12 changed files with 557 additions and 48 deletions

View File

@@ -0,0 +1,27 @@
# PRD: Streaming proxy cancellation delivery
## Overview
A tracker proxy route can be selected and registered as active, but a direct SSE upstream may not deliver its first frame to the downstream client until the upstream response completes. A dashboard cancellation request made after the upstream emitted and flushed an SSE frame can therefore arrive after the tracker has already finalized and unregistered the proxy, returning 404.
## Goal
Preserve immediate SSE frame delivery and keep the active-proxy record cancelable until the streaming response actually completes or is canceled.
## Quality gates
- Run the focused tracker streaming/cancellation tests.
- Run full `python -m pytest -q` and record unrelated failures exactly.
- Do not weaken billing, disconnect handling, proxy-inflight accounting, or cancel authorization.
## User story
### PSC-001: Cancel an active direct SSE proxy
As a dashboard operator, I need a cancellation request to reach an active direct SSE proxy after its first streamed frame, so I can stop long-running inference without a race-induced 404.
## Non-goals
- Do not change route selection or quantization policy.
- Do not alter relay transport behavior unless needed to share the same cancellation invariant.
- Do not introduce client-visible buffering.

View File

@@ -0,0 +1,34 @@
# PSC-001 — Direct SSE cancellation race
Status: ready-for-agent
Priority: High
## Problem
`tests/test_tracker_routing.py::test_tracker_dashboard_can_cancel_inflight_proxy` registers a streaming upstream node. The upstream writes and flushes the first SSE frame, then waits three seconds. The trackers client does not receive that frame until after the upstream has completed; by then `_unregister_active_proxy()` has run and the dashboard cancellation endpoint returns 404.
Observed trace:
```text
proxy route selected
proxy connected
proxy progress ... elapsed_seconds≈3
proxy complete ... elapsed_seconds≈3
POST /v1/proxy/requests/<id>/cancel → 404
```
This is a production cancellation/delivery race, not a stale test: the endpoint promises to cancel active proxy work, and the upstream had already emitted a first stream frame before cancellation was attempted.
## Acceptance criteria
- [ ] A direct SSE upstream frame is relayed and flushed to the client before the upstream completes.
- [ ] After that first frame, `/v1/proxy/requests/{request_id}/cancel` returns 200 while the stream is active.
- [ ] Cancellation closes/stops the upstream safely, finalizes inflight accounting exactly once, and records the cancellation event.
- [ ] Cancel authorization remains unchanged.
- [ ] Client disconnect and normal SSE completion retain current billing/throughput behavior.
- [ ] Regression test is deterministic and does not rely on timing races longer than necessary.
- [ ] Focused tracker routing tests and full pytest are run with unrelated failures documented.
## Likely seam
Inspect direct streaming behavior around `_handle_proxy_chat` upstream reads (`packages/tracker/meshnet_tracker/server.py`, roughly lines 39534019). The direct response path writes/flushed each line, but current HTTP response buffering/reading prevents the line from being observed until stream end. Fix delivery at the proxy transport boundary; do not paper over it by retaining completed proxy records indefinitely.

View File

@@ -0,0 +1,27 @@
# PRD: Legacy routing compatibility regression
## Overview
The trackers new request-precision gate rejects legacy node registrations that omit `quantization`. This conflicts with the explicit `capability_policy=compat` rollout: an older node without a capability report remains routable, but an older node without a quantization field is silently excluded from proxy, pinned-route, benchmark, billing, and latency paths.
## Goal
Restore backward-compatible routing for legacy registrations while preserving fail-closed behavior for explicitly declared invalid/unsupported quantization values.
## Quality gates
- Run targeted routing, billing, benchmark, pricing, validation, and latency tests.
- Run `python -m pytest -q` before completion and record any unrelated failures exactly.
- Preserve the admission-policy invariant: invalid capability reports remain non-routable; absent reports route only under `compat`.
## User story
### RCR-001: Legacy registration precision fallback
As an operator upgrading a mixed fleet, I need nodes that predate the `quantization` registration field to serve default-precision requests under `compat`, so the tracker rollout does not make otherwise healthy legacy nodes dark.
## Non-goals
- Do not weaken `enforce` capability policy.
- Do not treat explicitly malformed or unsupported quantization values as valid.
- Do not change requested-precision semantics for nodes that declare a supported precision.

View File

@@ -0,0 +1,38 @@
# RCR-001 — Legacy registration precision fallback
Status: ready-for-agent
Priority: Critical
## Problem
The tracker now filters proxy and pinned-route candidates through `_quantization_satisfies(node.quantization, requested_quantization)`. A registration produced before the protocol added singular `quantization` is normalized to `quantizations=["bfloat16"]`, but leaves `node.quantization is None`; routing then fails every request, including the default `bfloat16` request.
The defect is therefore at protocol normalization: registration correctly records an available plural capability but never chooses a compatible active precision for the legacy entry.
This is inconsistent with the explicit capability admission rollout policy: an absent capability report is routable under `compat`. Logs show `capability=absent routable=True`, while `/v1/chat/completions` returns 503 or a pinned route returns 409 because the separate precision gate excludes the same legacy node.
## Evidence
Representative tight repros fail on current `master`:
- `tests/test_billing_ledger.py::test_proxy_chat_bills_credited_client_and_credits_node` → HTTP 503
- `tests/test_tracker_routing.py::test_tracker_proxy_accepts_hf_model_alias_from_quickstart` → HTTP 503
- `tests/test_manual_route_benchmark.py::test_pinned_route_uses_named_node` → HTTP 409
- `tests/test_model_speed_latency.py::test_tracker_records_increasing_hop_latency_for_model_and_hardware` → HTTP 409 for every parameter
Each fixture registers a complete, healthy legacy node without `quantization`; tracker registration logs show it as routable under `compat`.
## Acceptance criteria
- [ ] Legacy nodes omitting the `quantization` field are treated as the historical default precision for compatibility purposes.
- [ ] Explicit `null`, invalid, or unsupported declared precision never receives that fallback and remains excluded from routing, assignment, and coverage calculations, including after managed shard assignment/rebalancing.
- [ ] Raft replication preserves singular `quantization` and plural `quantizations`; follower routing has identical precision eligibility to the leader.
- [ ] `capability_policy=enforce` still excludes absent capability reports regardless of precision fallback.
- [ ] Automatic proxy routing, pinned routes, benchmark routes, and route-stat sampling use the same compatibility rule.
- [ ] Add behavior-level tracker HTTP tests for legacy omission and explicit invalid declaration.
- [ ] Targeted billing, routing, benchmark, model-speed, pricing, forfeiture, and TOPLOC dispatch tests pass.
- [ ] Run the full suite and document any failures not caused by this issue.
## Implementation notes
Keep the compatibility decision at the protocol-normalization boundary, not scattered into individual route paths. A normalized legacy default can preserve existing `_quantization_satisfies` semantics while differentiating an omitted field from an explicitly invalid value.

View File

@@ -0,0 +1,29 @@
{
"name": "Legacy routing compatibility regression",
"description": "Restore compat-policy routing for legacy node registrations that omit quantization without weakening validation for invalid declared values.",
"branchName": "ralph/routing-compatibility-regression",
"userStories": [
{
"id": "RCR-001",
"title": "Legacy registration precision fallback",
"description": "As an operator upgrading a mixed fleet, I need nodes that predate the quantization registration field to serve default-precision requests under compat, so otherwise healthy legacy nodes do not become dark.",
"acceptanceCriteria": [
"A legacy registration without quantization is eligible for a bfloat16 request when capability policy is compat.",
"The same registration is excluded under capability policy enforce because its capability report is absent.",
"A node that explicitly declares null, an invalid, or an unsupported quantization remains excluded from routing, assignment, and coverage calculations, including after managed shard assignment/rebalancing.",
"Raft-applied registrations preserve both singular quantization and plural quantizations so follower routing matches leader routing.",
"Automatic proxy routing, pinned routes, manual benchmarks, billing proxy paths, and route-latency sampling retain their existing behavior for legacy registrations.",
"Tests cover the legacy/no-field case and explicit-invalid-field case at the tracker HTTP seam.",
"Targeted routing, billing, benchmark, pricing, validation, and latency tests pass.",
"python -m pytest -q is run and unrelated failures are recorded exactly."
],
"priority": 1,
"passes": true,
"dependsOn": [],
"completionNotes": "Completed by agent"
}
],
"metadata": {
"updatedAt": "2026-07-13T07:25:08.460Z"
}
}