popov/neuron-tai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stream chat responses in the dashboard with live progress and unified styles

Browse Source 
Chat now sends stream=true and renders SSE tokens incrementally with live
tok/s status, a stop button (AbortController), and a blinking cursor; because
streamed requests emit tracker 'proxy progress' events, the Call wall now
shows in-flight requests with live TPS too. Chat colors route through :root
tokens instead of hardcoded hex values.

ADR-0020 documents the changes and the mixed-topology routing flaw: a partial
GPU head (0-21) + full CPU node (0-39) gets downstream start_layer=0 instead
of 22, corrupting activations into 1-token generations that were billed and
polluted throughput stats. Fix steps recorded, not yet implemented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dobromir Popov
2026-07-07 19:48:43 +02:00
parent 481ce6c6f5
commit e2b20883ca
3 changed files with 420 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files

127
docs/adr/0020-chat-streaming-live-progress-and-mixed-topology-routing.md Normal file
View File

@@ -0,0 +1,127 @@
# ADR-0020: Dashboard chat streaming, live request progress, and the mixed-topology routing flaw
## Status: Accepted (chat/streaming/styles implemented); routing flaw documented, fix pending
## Context
Live alpha testing (2026-07-07) with `Qwen3.6-35B-A3B` split across two LAN nodes surfaced
three UX gaps and one routing correctness flaw:
1. **No visibility while a request is processing.** The Call wall showed
"no in-flight requests" during a 52-second generation. Cause: the dashboard chat sent
`stream: false`, and the tracker only emits `proxy progress` console events (the Call
wall's live-status source, `_tracker_log_proxy_progress`, `server.py` ~2199) for
**streamed** requests. Non-streamed proxying produces only
`route selected → connected → complete`, and short requests complete inside the
dashboard's 4-second poll window.
2. **Chat did not stream.** The nodes support SSE token-by-token generation
(`generate_text_streaming`, hardened earlier for split shards), and the tracker proxy
passes `text/event-stream` through (`server.py` ~3256), but the chat panel blocked on
full JSON and showed nothing until completion.
3. **Chat panel styles drifted.** The "new chat layout" redesign left hardcoded one-off
colors (`#1f4788`, `#2563b8`, `#10151d`, `#1a1012`, `#5c2020`, `#ffb4b4`) mixed with
the CSS custom-property palette.
## Decisions
### 1. Chat streams by default (SSE)
`dashboard.html` `sendChat()` now sends `stream: true` and consumes the SSE body with a
`ReadableStream` reader:
- Assistant tokens render incrementally into the last bubble (direct DOM update, full
re-render only at boundaries), with a blinking `▍` cursor while streaming.
- Chat status shows live progress: `generating… N tokens · X tok/s`.
- The send button becomes a stop button (`■`) during generation, backed by an
`AbortController`; a stopped generation keeps the partial text.
- Non-SSE responses (JSON fallback, errors) are still handled; `data: {"error": ...}`
stream events surface as error bubbles.
- `streaming` flags are stripped when loading persisted sessions so an interrupted
generation never leaves a stuck cursor.
### 2. Live in-flight visibility rides on streaming
No tracker change was needed: because chat now streams, the tracker emits `proxy progress`
events (throttled to stdout, updated in place in the console ring via
`update_console_key`), and the existing Call wall state machine
(`buildCallWallStates`) renders processing rows with live tokens/TPS/queue.
**Known limitation (accepted):** non-streamed API requests still show no progress between
`proxy connected` and `proxy complete` — there is nothing to report until the node
returns. Callers wanting live visibility should use `stream: true`.
### 3. Chat style tokens
All chat colors route through `:root` custom properties (`--hover-bg`, `--chat-user-bg`
`#1f6feb`, `--chat-user-border`, `--chat-error-bg/border/fg`). No hardcoded hex values
remain in chat rules, so future palette changes are single-line edits.
## Documented flaw: mixed-topology routing (partial GPU head + full CPU node)
### Observed (2026-07-07, tracker 192.168.0.179:8080)
Two nodes registered for `qwen3.6-35b-a3b`:
| node | hardware | shard | benchmark |
|---|---|---|---|
| `5gMLrmyB-ec3afe6f1a03` (192.168.0.20) | RTX 4060, CUDA | 0–21 (partial, fast) | 11,164 |
| `7j77FsPY-55249b0583e5` (192.168.0.179) | CPU | 0–39 (full, slow) | 425 |
When the tracker selected the GPU node as head, it injected:
```
downstream=[{"endpoint": "http://192.168.0.179:7000", "start_layer": 0}]
```
`start_layer: 0` — not 22. The downstream full node re-ran **all 40 layers from layer 0
on hidden states that had already passed through the head's layers 0–21**, producing
garbage logits. Evidence from the logs:
- GPU-headed requests: `generation complete tokens=1` and billed `out=0`/`out=1`/`out=3`
— near-instant EOS from corrupt activations.
- The same prompt routed directly to the CPU full node: 209 tokens over 52 s (healthy).
- Observed TPS for GPU-headed requests was meaningless (2.5–19.0 "tok/s" on 0–3 token
outputs), and those samples now pollute the rolling per-`(node, model)` throughput
stats used for routing preference.
- Clients were **billed** for these broken 1-token responses.
### Root cause
The route planner treats the full-coverage node as a standalone complete route
(`route=7j77FsPY…[0-39]`) but still injects it as the head's downstream with the
downstream node's own `shard_start` (0) instead of `head.shard_end + 1` (22). A partial
head + full-model downstream is a topology the planner never had to handle before —
prior split tests used disjoint shards (0–11 + 12–23) where `shard_start` happened to
equal the correct continuation layer.
### Required fix (not yet implemented)
1. **Correct continuation layer:** when hop N ends at layer `e`, hop N+1 must execute
from `start_layer = e + 1` regardless of the downstream node's own `shard_start`
(the `X-Meshnet-Start-Layer` overlapping-shard mechanism from ADR-0012 exists for
exactly this; the planner must set it for full-model downstream nodes too).
2. **Route preference sanity:** with a healthy single-node full route available, prefer
it over a multi-hop route unless the pipeline is estimated faster; a fast head that
forces a slow full-model tail wins nothing (every token still crosses the CPU node).
3. **Stat hygiene:** exclude or flag throughput samples from responses with ≤ a few
output tokens, so broken routes don't skew routing preference.
4. **Billing guard (consider):** suspiciously short completions from multi-hop routes
during this window were billed; a minimum-viability check (or refund path) may be
warranted once audits land.
### Verification for the fix
Reproduce with a partial GPU head (0–21) + full CPU node (0–39): a chat request routed
through the GPU head must produce output equivalent to the direct CPU route, with
`downstream start_layer=22` visible in `proxy route selected`, and multi-token streamed
output on the Call wall.
## Verification of this ADR's implemented changes
- `pytest tests/test_dashboard.py` — 5 passed (stale "Chat / inference" panel assertion
updated to the tabbed layout).
- Embedded dashboard JS parses (`new Function(script)` under Node 22).
- Live check: open `/dashboard` → Chat, send a prompt to `qwen3.6-35b-a3b` — tokens
must appear incrementally with live tok/s in the status line, the Call wall must show
the request as `processing` with live TPS, and the send button must stop generation
mid-stream keeping partial text.

192
packages/tracker/meshnet_tracker/dashboard.html
View File

@@ -7,7 +7,10 @@
<style> <style>
:root { --bg:#0d1117; --panel:#161b22; --border:#30363d; --fg:#e6edf3; :root { --bg:#0d1117; --panel:#161b22; --border:#30363d; --fg:#e6edf3;
--dim:#8b949e; --accent:#58a6ff; --ok:#3fb950; --bad:#f85149; --warn:#d29922; --dim:#8b949e; --accent:#58a6ff; --ok:#3fb950; --bad:#f85149; --warn:#d29922;
--chat-input-bg:#21262d; --chat-user-bg:#1f4788; --chat-user-border:#388bfd; } --hover-bg:#10151d;
--chat-input-bg:#21262d; --chat-user-bg:#1f6feb; --chat-user-border:#388bfd;
--chat-error-bg:rgba(248,81,73,.08); --chat-error-border:rgba(248,81,73,.35);
--chat-error-fg:#ffa198; }
* { box-sizing:border-box; } * { box-sizing:border-box; }
html, body { height:100%; } html, body { height:100%; }
body { margin:0; background:var(--bg); color:var(--fg); body { margin:0; background:var(--bg); color:var(--fg);
@@ -85,7 +88,7 @@
border:1px solid var(--border); border-radius:8px; padding:10px 12px; border:1px solid var(--border); border-radius:8px; padding:10px 12px;
background:transparent; color:var(--fg); background:transparent; color:var(--fg);
} }
.chat-new-btn:hover { background:#10151d; border-color:var(--accent); } .chat-new-btn:hover { background:var(--hover-bg); border-color:var(--accent); }
.chat-session-list { .chat-session-list {
flex:1; overflow:auto; padding:0 8px 12px; display:flex; flex-direction:column; gap:2px; flex:1; overflow:auto; padding:0 8px 12px; display:flex; flex-direction:column; gap:2px;
} }
@@ -98,8 +101,8 @@
padding:10px 32px 10px 12px; border:1px solid transparent; border-radius:8px; padding:10px 32px 10px 12px; border:1px solid transparent; border-radius:8px;
background:transparent; color:var(--fg); cursor:pointer; background:transparent; color:var(--fg); cursor:pointer;
} }
.chat-session-item:hover { background:#10151d; } .chat-session-item:hover { background:var(--hover-bg); }
.chat-session-item.active { background:#10151d; border-color:#30363d; } .chat-session-item.active { background:var(--hover-bg); border-color:var(--border); }
.chat-session-title { .chat-session-title {
font-size:13px; white-space:nowrap; overflow:hidden; text-overflow:ellipsis; font-size:13px; white-space:nowrap; overflow:hidden; text-overflow:ellipsis;
} }
@@ -114,7 +117,7 @@
} }
.chat-session-item:hover .chat-session-delete, .chat-session-item:hover .chat-session-delete,
.chat-session-item.active .chat-session-delete { opacity:1; } .chat-session-item.active .chat-session-delete { opacity:1; }
.chat-session-delete:hover { color:var(--bad); background:#1a1012; } .chat-session-delete:hover { color:var(--bad); background:var(--chat-error-bg); }
.chat-main { display:flex; flex-direction:column; min-height:0; min-width:0; color-scheme:dark; } .chat-main { display:flex; flex-direction:column; min-height:0; min-width:0; color-scheme:dark; }
.chat-toolbar { .chat-toolbar {
display:flex; gap:12px; align-items:center; flex-shrink:0; display:flex; gap:12px; align-items:center; flex-shrink:0;
@@ -156,8 +159,14 @@
border-bottom-left-radius:4px; max-width:100%; color:var(--fg); border-bottom-left-radius:4px; max-width:100%; color:var(--fg);
} }
.chat-bubble.error { .chat-bubble.error {
background:#1a1012; border:1px solid #5c2020; color:#ffb4b4; border-bottom-left-radius:4px; background:var(--chat-error-bg); border:1px solid var(--chat-error-border);
color:var(--chat-error-fg); border-bottom-left-radius:4px;
} }
.chat-bubble.assistant.streaming::after {
content:"▍"; color:var(--accent); margin-left:2px;
animation:chat-blink 1s steps(2) infinite;
}
@keyframes chat-blink { 50% { opacity:0; } }
.chat-compose-wrap { .chat-compose-wrap {
flex-shrink:0; padding:12px 16px 16px; border-top:1px solid var(--border); flex-shrink:0; padding:12px 16px 16px; border-top:1px solid var(--border);
background:var(--panel); background:var(--panel);
@@ -183,7 +192,7 @@
background:var(--chat-user-bg); color:#f0f6fc; background:var(--chat-user-bg); color:#f0f6fc;
} }
.chat-compose button:hover:not(:disabled) { .chat-compose button:hover:not(:disabled) {
border-color:var(--accent); background:#2563b8; border-color:var(--accent); background:var(--chat-user-border);
} }
.chat-compose button:disabled { opacity:.45; cursor:not-allowed; } .chat-compose button:disabled { opacity:.45; cursor:not-allowed; }
.console { .console {
@@ -242,7 +251,7 @@
<div class="chat-compose-wrap"> <div class="chat-compose-wrap">
<div class="chat-compose"> <div class="chat-compose">
<textarea id="chat-prompt" placeholder="Message…" rows="1" aria-label="Message"></textarea> <textarea id="chat-prompt" placeholder="Message…" rows="1" aria-label="Message"></textarea>
<button type="button" onclick="sendChat()" id="chat-send" title="Send (Enter)">↑</button> <button type="button" onclick="onChatSendClick()" id="chat-send" title="Send (Enter)">↑</button>
</div> </div>
</div> </div>
</div> </div>
@@ -732,7 +741,11 @@ function loadChatSessionsStore() {
try { try {
const raw = localStorage.getItem(CHAT_SESSIONS_KEY); const raw = localStorage.getItem(CHAT_SESSIONS_KEY);
const parsed = raw ? JSON.parse(raw) : []; const parsed = raw ? JSON.parse(raw) : [];
return Array.isArray(parsed) ? parsed : []; if (!Array.isArray(parsed)) return [];
for (const session of parsed) {
for (const msg of session.messages || []) delete msg.streaming;
}
return parsed;
} catch { } catch {
return []; return [];
} }
@@ -950,7 +963,8 @@ function renderChatHistory() {
history.className = "chat-messages"; history.className = "chat-messages";
const rows = chatHistory.map(msg => { const rows = chatHistory.map(msg => {
const role = msg.role === "user" ? "user" : msg.role === "assistant" ? "assistant" : "error"; const role = msg.role === "user" ? "user" : msg.role === "assistant" ? "assistant" : "error";
return `<div class="chat-row ${role}"><div class="chat-bubble ${role}">${esc(msg.content)}</div></div>`; const streaming = msg.streaming ? " streaming" : "";
return `<div class="chat-row ${role}"><div class="chat-bubble ${role}${streaming}">${esc(msg.content)}</div></div>`;
}).join(""); }).join("");
history.innerHTML = `<div class="chat-messages-inner">${rows}</div>`; history.innerHTML = `<div class="chat-messages-inner">${rows}</div>`;
history.scrollTop = history.scrollHeight; history.scrollTop = history.scrollHeight;
@@ -1222,6 +1236,47 @@ async function renderAccountPanel() {
if (account.role === "admin") await renderAdminPanel(); if (account.role === "admin") await renderAdminPanel();
} }
let chatAbortController = null;
function onChatSendClick() {
if (chatBusy && chatAbortController) { chatAbortController.abort(); return; }
sendChat();
}
function setChatSendMode(streaming) {
const btn = $("chat-send");
if (!btn) return;
btn.textContent = streaming ? "■" : "↑";
btn.title = streaming ? "Stop generating" : "Send (Enter)";
}
function updateStreamingChatBubble(content) {
const history = $("chat-history");
if (!history) return;
const bubbles = history.querySelectorAll(".chat-bubble.assistant.streaming");
const bubble = bubbles[bubbles.length - 1];
if (!bubble) { renderChatHistory(); return; }
bubble.textContent = content;
const nearBottom = history.scrollHeight - history.scrollTop - history.clientHeight < 80;
if (nearBottom) history.scrollTop = history.scrollHeight;
}
function finalizeAssistantMessage(msg) {
delete msg.streaming;
if (!chatHistory.includes(msg)) chatHistory.push(msg);
if (!msg.content) msg.content = "(empty response)";
renderChatHistory();
persistActiveChatSession();
}
function removeAssistantMessage(msg) {
const index = chatHistory.indexOf(msg);
if (index >= 0) {
chatHistory.splice(index, 1);
renderChatHistory();
}
}
async function sendChat() { async function sendChat() {
const promptEl = $("chat-prompt"); const promptEl = $("chat-prompt");
const prompt = promptEl.value.trim(); const prompt = promptEl.value.trim();
@@ -1239,44 +1294,107 @@ async function sendChat() {
.map(msg => ({ role: msg.role, content: msg.content })), .map(msg => ({ role: msg.role, content: msg.content })),
{ role: "user", content: prompt }, { role: "user", content: prompt },
], ],
stream: false, stream: true,
max_tokens: 15120, max_tokens: 15120,
}; };
chatBusy = true; chatBusy = true;
$("chat-send").disabled = true; setChatSendMode(true);
promptEl.value = ""; promptEl.value = "";
promptEl.style.height = "auto"; promptEl.style.height = "auto";
chatHistory.push({ role: "user", content: prompt, model: selectedChatModel }); chatHistory.push({ role: "user", content: prompt, model: selectedChatModel });
renderChatHistory(); renderChatHistory();
persistActiveChatSession(); persistActiveChatSession();
renderChatStatus("sending request…"); renderChatStatus("sending request…");
const r = await apiCall("/v1/chat/completions", "POST", body, bearerToken); const assistantMsg = { role: "assistant", content: "", model: selectedChatModel, streaming: true };
chatBusy = false; let tokens = 0;
$("chat-send").disabled = false; let usage = null;
if (!r.ok) { const started = Date.now();
const error = r.data && r.data.error chatAbortController = new AbortController();
? (typeof r.data.error === "string" ? r.data.error : r.data.error.message || "request failed") try {
: "request failed"; const headers = { "Content-Type": "application/json" };
chatHistory.push({ role: "error", content: error, model: selectedChatModel }); if (bearerToken) headers["Authorization"] = "Bearer " + bearerToken;
renderChatHistory(); const resp = await fetch("/v1/chat/completions", {
persistActiveChatSession(); method: "POST",
renderChatStatus(error); headers,
body: JSON.stringify(body),
signal: chatAbortController.signal,
});
if (!resp.ok) {
let error = "request failed";
try {
const data = await resp.json();
error = typeof data.error === "string" ? data.error : (data.error && data.error.message) || error;
} catch { /* keep default */ }
throw new Error(error);
}
const contentType = resp.headers.get("Content-Type") || "";
if (!resp.body || !contentType.includes("text/event-stream")) {
const data = await resp.json();
assistantMsg.content = (data.choices && data.choices[0] && data.choices[0].message && data.choices[0].message.content) || "";
usage = data.usage || null;
tokens = (usage && usage.completion_tokens) || 0;
} else {
chatHistory.push(assistantMsg);
renderChatHistory();
const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffered = "";
for (;;) {
const { done, value } = await reader.read();
if (done) break;
buffered += decoder.decode(value, { stream: true });
let newline;
while ((newline = buffered.indexOf("\n")) >= 0) {
const line = buffered.slice(0, newline).replace(/\r$/, "");
buffered = buffered.slice(newline + 1);
if (!line.startsWith("data:")) continue;
const payload = line.slice(5).trim();
if (!payload || payload === "[DONE]") continue;
let chunk;
try { chunk = JSON.parse(payload); } catch { continue; }
if (chunk.error) {
throw new Error(typeof chunk.error === "string" ? chunk.error : chunk.error.message || "stream error");
}
if (chunk.usage) usage = chunk.usage;
const delta = chunk.choices && chunk.choices[0] && chunk.choices[0].delta;
const piece = delta && delta.content;
if (piece) {
assistantMsg.content += piece;
tokens += 1;
updateStreamingChatBubble(assistantMsg.content);
const secs = Math.max((Date.now() - started) / 1000, 0.001);
renderChatStatus(`generating… ${tokens} tokens · ${(tokens / secs).toFixed(1)} tok/s`);
}
}
}
}
finalizeAssistantMessage(assistantMsg);
renderChatStatus(usage
? `done: ${usage.total_tokens ?? "?"} tokens`
: `done: ${tokens} tokens`);
} catch (err) {
if (err && err.name === "AbortError") {
if (assistantMsg.content) {
finalizeAssistantMessage(assistantMsg);
renderChatStatus(`stopped after ${tokens} tokens`);
} else {
removeAssistantMessage(assistantMsg);
renderChatStatus("stopped");
}
} else {
removeAssistantMessage(assistantMsg);
const error = (err && err.message) || "request failed";
chatHistory.push({ role: "error", content: error, model: selectedChatModel });
renderChatHistory();
persistActiveChatSession();
renderChatStatus(error);
}
} finally {
chatBusy = false;
chatAbortController = null;
setChatSendMode(false);
promptEl.focus(); promptEl.focus();
return;
} }
const reply = (r.data && r.data.choices && r.data.choices[0] && r.data.choices[0].message && r.data.choices[0].message.content) || "";
const usage = r.data && r.data.usage;
chatHistory.push({
role: "assistant",
content: reply || "(empty response)",
model: selectedChatModel,
});
renderChatHistory();
persistActiveChatSession();
renderChatStatus(usage
? `done: ${usage.total_tokens ?? "?"} tokens`
: "done");
promptEl.focus();
} }
function bindChatPromptShortcuts() { function bindChatPromptShortcuts() {

276
tests/test_dashboard.py
View File

@@ -1,138 +1,138 @@
"""US-035: tracker web dashboard — served from any tracker, embedded asset.""" """US-035: tracker web dashboard — served from any tracker, embedded asset."""
import json import json
import time import time
import urllib.request import urllib.request
from meshnet_contracts import LocalSolanaContracts from meshnet_contracts import LocalSolanaContracts
from meshnet_tracker.accounts import AccountStore from meshnet_tracker.accounts import AccountStore
from meshnet_tracker.billing import BillingLedger from meshnet_tracker.billing import BillingLedger
from meshnet_tracker.server import TrackerServer from meshnet_tracker.server import TrackerServer
PANELS = [ PANELS = [
"Tracker hive", "Nodes &amp; coverage", "Client balances", "Tracker hive", "Nodes &amp; coverage", "Client balances",
"Node pending payouts", "Settlement history", "Node pending payouts", "Settlement history",
"Strikes / bans / forfeitures", "Model usage", "Call wall", "Strikes / bans / forfeitures", "Model usage", "Call wall",
"Usage summary", "Node throughput", "Request history", "Usage summary", "Node throughput", "Request history",
"Chat / inference", 'id="tab-chat"',
"Console output", "Console output",
] ]
def test_dashboard_served_with_all_panels(): def test_dashboard_served_with_all_panels():
tracker = TrackerServer(billing=BillingLedger()) tracker = TrackerServer(billing=BillingLedger())
port = tracker.start() port = tracker.start()
try: try:
html = urllib.request.urlopen( html = urllib.request.urlopen(
f"http://127.0.0.1:{port}/dashboard" f"http://127.0.0.1:{port}/dashboard"
).read().decode() ).read().decode()
for panel in PANELS: for panel in PANELS:
assert panel in html assert panel in html
assert "<script>" in html # polling client embedded, no build step assert "<script>" in html # polling client embedded, no build step
finally: finally:
tracker.stop() tracker.stop()
def test_dashboard_served_by_follower(): def test_dashboard_served_by_follower():
"""A tracker that is not the leader (unreachable peers → never elected) """A tracker that is not the leader (unreachable peers → never elected)
still serves the dashboard from its own replicated state.""" still serves the dashboard from its own replicated state."""
tracker = TrackerServer( tracker = TrackerServer(
billing=BillingLedger(), billing=BillingLedger(),
cluster_peers=["http://127.0.0.1:1", "http://127.0.0.1:2"], cluster_peers=["http://127.0.0.1:1", "http://127.0.0.1:2"],
) )
port = tracker.start() port = tracker.start()
try: try:
response = urllib.request.urlopen(f"http://127.0.0.1:{port}/dashboard") response = urllib.request.urlopen(f"http://127.0.0.1:{port}/dashboard")
assert response.status == 200 assert response.status == 200
assert "meshnet tracker" in response.read().decode() assert "meshnet tracker" in response.read().decode()
finally: finally:
tracker.stop() tracker.stop()
def test_registry_wallets_endpoint(): def test_registry_wallets_endpoint():
contracts = LocalSolanaContracts() contracts = LocalSolanaContracts()
contracts.registry.submit_stake("wallet-a", 100) contracts.registry.submit_stake("wallet-a", 100)
contracts.registry.record_strike("wallet-a") contracts.registry.record_strike("wallet-a")
accounts = AccountStore() accounts = AccountStore()
admin = accounts.register(email="admin@example.com", password="admin-pass-123") admin = accounts.register(email="admin@example.com", password="admin-pass-123")
session = accounts.create_session(admin["account_id"]) session = accounts.create_session(admin["account_id"])
tracker = TrackerServer(contracts=contracts, accounts=accounts) tracker = TrackerServer(contracts=contracts, accounts=accounts)
port = tracker.start() port = tracker.start()
try: try:
req = urllib.request.Request( req = urllib.request.Request(
f"http://127.0.0.1:{port}/v1/registry/wallets", f"http://127.0.0.1:{port}/v1/registry/wallets",
headers={"Authorization": f"Bearer {session}"}, headers={"Authorization": f"Bearer {session}"},
) )
data = json.loads(urllib.request.urlopen(req).read()) data = json.loads(urllib.request.urlopen(req).read())
assert data["wallets"]["wallet-a"]["strike_count"] == 1 assert data["wallets"]["wallet-a"]["strike_count"] == 1
assert data["wallets"]["wallet-a"]["banned"] is False assert data["wallets"]["wallet-a"]["banned"] is False
finally: finally:
tracker.stop() tracker.stop()
def test_console_endpoint_exposes_tracker_events(): def test_console_endpoint_exposes_tracker_events():
tracker = TrackerServer() tracker = TrackerServer()
port = tracker.start() port = tracker.start()
try: try:
body = json.dumps({ body = json.dumps({
"endpoint": "http://127.0.0.1:9001", "endpoint": "http://127.0.0.1:9001",
"model": "stub-model", "model": "stub-model",
"shard_start": 0, "shard_start": 0,
"shard_end": 3, "shard_end": 3,
"hardware_profile": {}, "hardware_profile": {},
}).encode() }).encode()
req = urllib.request.Request( req = urllib.request.Request(
f"http://127.0.0.1:{port}/v1/nodes/register", f"http://127.0.0.1:{port}/v1/nodes/register",
data=body, data=body,
headers={"Content-Type": "application/json"}, headers={"Content-Type": "application/json"},
method="POST", method="POST",
) )
urllib.request.urlopen(req).read() urllib.request.urlopen(req).read()
data = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read()) data = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read())
finally: finally:
tracker.stop() tracker.stop()
assert any(event["message"] == "node registered" for event in data["events"]) assert any(event["message"] == "node registered" for event in data["events"])
def test_console_node_lifecycle_events_include_model_health(): def test_console_node_lifecycle_events_include_model_health():
tracker = TrackerServer(heartbeat_timeout=0.05) tracker = TrackerServer(heartbeat_timeout=0.05)
port = tracker.start() port = tracker.start()
try: try:
body = json.dumps({ body = json.dumps({
"endpoint": "http://127.0.0.1:9002", "endpoint": "http://127.0.0.1:9002",
"model": "console-health-test", "model": "console-health-test",
"hf_repo": "example/console-health-test", "hf_repo": "example/console-health-test",
"num_layers": 4, "num_layers": 4,
"shard_start": 0, "shard_start": 0,
"shard_end": 1, "shard_end": 1,
"hardware_profile": {}, "hardware_profile": {},
}).encode() }).encode()
req = urllib.request.Request( req = urllib.request.Request(
f"http://127.0.0.1:{port}/v1/nodes/register", f"http://127.0.0.1:{port}/v1/nodes/register",
data=body, data=body,
headers={"Content-Type": "application/json"}, headers={"Content-Type": "application/json"},
method="POST", method="POST",
) )
urllib.request.urlopen(req).read() urllib.request.urlopen(req).read()
registered = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read()) registered = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read())
registered_event = next( registered_event = next(
event for event in registered["events"] event for event in registered["events"]
if event["message"] == "node registered" if event["message"] == "node registered"
) )
assert registered_event["fields"]["model_health"]["served_model_copies"] == 0.5 assert registered_event["fields"]["model_health"]["served_model_copies"] == 0.5
assert registered_event["fields"]["model_health"]["coverage_percentage"] == 50.0 assert registered_event["fields"]["model_health"]["coverage_percentage"] == 50.0
time.sleep(0.06) time.sleep(0.06)
urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/network/map").read() urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/network/map").read()
expired = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read()) expired = json.loads(urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/console").read())
expired_event = next( expired_event = next(
event for event in expired["events"] event for event in expired["events"]
if event["message"] == "node expired" if event["message"] == "node expired"
) )
assert expired_event["fields"]["model_health"]["served_model_copies"] == 0.0 assert expired_event["fields"]["model_health"]["served_model_copies"] == 0.0
assert expired_event["fields"]["model_health"]["coverage_percentage"] == 0.0 assert expired_event["fields"]["model_health"]["coverage_percentage"] == 0.0
finally: finally:
tracker.stop() tracker.stop()