neuron-tai/.claude/memory/project-status.md

name, description, metadata

name	description	metadata
project-status	Current state of neuron-tai development as of 2026-07-02	node_type type memory project

Project Status (2026-07-13)

Distributed GGUF controller checkpoint (2026-07-21)

• All three fixed detached lanes from the 2026-07-18 checkpoint (Fable/DGR-025, Kimi/DGR-028, Terra/DGR-024) were reviewed, committed, and merged into ralph/distributed-gguf-runtime, now at cd6b4d9. The .claude/worktrees/ralph-fable-loop, ralph-kimi-loop, ralph-terra-loop, ralph-cursor-loop, and ralph-next-task worktrees were removed after merge (cursor-loop/next-task were idle with no work in progress). Only .claude/worktrees/distributed-gguf-runtime (the integration checkout) remains, plus the unrelated fix-tracker-incomplete-snapshot worktree (locked, not part of this arc).
• DGR-025 and DGR-028 merged cleanly with no conflicts (105 + 7 tests passing, 112 together). DGR-024 required real completion work first: the worktree's prd.json note was stale (written for the old, policy-rejected in-memory fake-seam story). The actual code already present (shard_runtime_server.py, test_shard_runtime_harness.py) had correctly pivoted to a real subprocess/socket gRPC harness with direct-vs-opaque-relay byte-identity proof, but was missing the required fail-closed negative paths (stale epoch, expired deadline, malformed/checksum-corrupt fragments, exhausted flow-control credit, duplicate idempotency steps, in-band/out-of-band cancel). Implemented those (SessionState per route_session_id, _validate_bundle), added 9 new tests (11 total, all passing), and rewrote evidence/DGR-024/README.md to describe the actual implementation instead of the nonexistent FakeShardSeam.
• prd.json's passes field was left false for all three stories — that flag is only flipped by the project's own independent controller review process, not by whoever lands the merge. DGR-028's evidence README explicitly still awaits that P0/P1 review.
• Post-merge full suite (before the DGR-024 merge): 3 failed / 1116 passed / 20 skipped — down from the prior 9-failure baseline, and the 3 remaining failures (billing default-db, dynamic-routing ADD_SHARD/LOAD_SHARD, preset dedup) are pre-existing and unrelated to native/runtime-identity code. Did not get a chance to rerun the full suite after the DGR-024 merge landed (interrupted); only the focused tests/test_shard_runtime_harness.py (11 passed) was reconfirmed post-merge — a fresh full-suite run is worth doing before treating the whole arc as done.
• The unrelated dirty edits on the integration worktree (model_catalog.py / tests/test_mining_cli.py, adding a Qwen2.5-Coder-1.5B-Instruct-Q2_K-GGUF preset) were stashed during the merge sequence and popped back afterward — still uncommitted, as before.

Why: user directed "distributed-gguf-runtime is where we need to merge all ralph-* branches" and to prune worktrees once their task's work is merged, so completed lanes don't linger and new worktrees signal new tasks unambiguously. How to apply: next Ralph session picking up this project should start from ralph/distributed-gguf-runtime at cd6b4d9, rerun the full suite once to get a clean current baseline, and check whether independent review has flipped DGR-024/025/028's passes flags before selecting new work.

Distributed GGUF controller checkpoint (2026-07-18)

• Integration branch ralph/distributed-gguf-runtime is at 377bc3475c41b762ebbcab038c05adf15d7749d0, matching its remote; preserve the unrelated dirty integration edits in model_catalog.py and test_mining_cli.py.
• Fixed detached lanes hold three claimed, uncommitted tasks: Fable DGR-025/Gitea #9 executing-artifact identity repair; Terra DGR-024/#8 fake gRPC seam repair; Kimi DGR-028/#12 numbered llama.cpp patch stack. No provider worker was running at the latest reconciliation, so continue these dirty trees before selecting new work.
• DGR-028 controller repair added working apply/reverse/verify enforcement, exact assumptions, and a passing exact-pin native fixture; final independent P0/P1 review is still required before commit/integration. DGR-024 and DGR-025 focused gates pass but likewise remain provisional pending the current independent reviews. Controller reruns on 2026-07-18 passed DGR-025 (235 impacted tests), DGR-024 (78 shared/focused tests), and DGR-028 (7 Python tests plus exact-pin native CTest 1/1 and clean apply/reverse). Integrate DGR-028 before DGR-025, then replay DGR-025 on the new integration HEAD and regenerate/retest its runtime fingerprint vectors because DGR-028 changes the pinned patch-stack/tree identity. The integration full-suite baseline is 9 failures/1076 passes/22 skips: missing optional zstd/langchain dependencies plus pre-existing billing and dynamic-routing expectations; lane-only extra failures map to stale DGR-023 projection ancestry and a timing-sensitive cancel test, not the focused story paths.

Selected-node model placement (2026-07-14)

• Admin Model placement now opens a node selector for load and release; the control-plane accepts optional node_id and targets only that registry assignment. Multi-model serving remains supported through ADD_SHARD and max_loaded_shards.
• Total node pool resource values are rendered from /v1/network/map's node.capacity contract. Route selection remains assignment/capability/throughput/queue based; capacity is used for placement and falls back to tracker defaults only if a node truly omits it.

Distributed inference performance (2026-07-14)

DIP-001 is done in .scratch/distributed-inference-performance/: the deterministic two-node Route Session stub benchmark covers direct/relay plus cached/stateless prefill and decode. Its JSON and concise summary explicitly attribute model execution, activation encode/decode, compression, connection setup, relay queueing, local HTTP forwarding, and end-to-end seam latency. PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py passed (7); the fixture assertion checks output-token identity and connection attempts.

Doc reconciliation 2026-07-13: docs/prd.json tracks US-001…US-050 (048 memory budget, 049 mainnet pilot, 050 Qwen demand placement). ADRs 0025–0026 added (TAI phase B/C, assignment ownership).

All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:

• BillingLedger (packages/tracker/meshnet_tracker/billing.py): event-sourced USDT ledger, gossip-replicated across the hive (id-deduped events), SQLite-persisted. 90/10 split by work units, per-model per-1K-token pricing, 402 before routing.
• Solana custodial adapter (packages/contracts/meshnet_contracts/solana_adapter.py): urllib JSON-RPC + solders signing. NOTE: installed solana-py 0.40 has NO sync client — don't import solana.rpc.api / spl.token.client.
• scripts/devnet_setup.py: creates mock-USDT mint + treasury, writes .env.devnet; --mint-to funds test clients.
• TrackerServer threads: deposit watcher (exactly-once via deposit- event ids) + leader-only settlement loop (threshold OR max-period, dust floor, resend-by-settlement-id → no double-pay).
• Forfeiture penalty: validator forfeits pending balance + strike; 3 strikes ban; probation redirects shares to protocol cut. Math in packages/validator/README.md.
• Web dashboard: GET /dashboard on any tracker, embedded dashboard.html, 4s polling.
• Observed routing throughput: tracker records rolling observed tokens/sec per (node_id, model) from completed proxied inference requests, exposes it via /v1/stats and /v1/network/map, shows it on the dashboard, and prefers observed per-model TPS over startup benchmark for routing when samples exist.

Suite: 222 passed, 3 skipped (openai/langchain packages missing in .venv — pre-existing).

Why: design locked in ADR-0015 (USDT custodial settlement; TAI deferred, protocol cut = future TAI liquidity). How to apply: next steps are live devnet verification (run devnet_setup.py, start tracker with --solana-rpc-url/--usdt-mint/--treasury-keypair --billing-db), then the TAI mint when volume justifies it. Work not yet committed to git as of session end — check git status.

Alpha hardening (2026-07-05)

Implementation complete for alpha-scoped blockers in .scratch/alpha-hardening/: 16/22 issues are done, including auth/gossip/service-token, persisted strike/ban/reputation, zero starting credit + spend cap, tracker-authoritative accounting, wallet binding proof, TOPLOC audit primitive, hop bisection, reputation scoring/routing, adaptive audits, and penalty wiring. Remaining 6/22 are ready-for-human / post-alpha or ops-gated: 12–15 multi-tracker money/Raft ordering hardening, 17 duplicate US-020 renumbering approval, and 21 honest-noise calibration corpus before production audit thresholds. Current verification: uv run pytest -k 'not test_legacy_start_without_port_uses_next_available_port' passes 316, skips 3; full uv run pytest has one environmental failure while local meshnet-node PID 1263451 occupies port 7000.

Historical handoff note: /mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff-2026-07-04.md is useful for navigation and original audit context, but it predates the completed .scratch/alpha-hardening/ planning artifacts. Treat its "missing ADR/issues/README" statements as stale; prefer .scratch/alpha-hardening/README.md and .scratch/alpha-hardening/handoff.md for current task order.

Node capability admission P0 (2026-07-09)

Planning is ready at .scratch/node-capability-admission/ with five sequential Ralph stories and ADR-0023. The design is model-agnostic: a Node must validate its selected Model Artifact/shard with a bounded real forward before Tracker routing; Qwen3.6 is only an optional development fixture. P0 adds a versioned local recipe-manifest/report contract, meshnet-node doctor, fail-closed startup admission, and tracker route gating. It intentionally excludes dynamic recipe/dependency installation and the future signed Node updater.

Windows CUDA node (working as of 2026-07-01)

• miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
• RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200
• Run: meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
• Known: tracker registration fails with http:// — must use https://
• pynvml deprecation warning is harmless (use nvidia-ml-py to silence it)

Model cache layout (2026-07-07)

• Node downloads now cache files directly under <download_dir>/<model>/, not <model>/layers_<start>-<end>/, so a wider cached layer assignment can satisfy a later narrower assignment without duplicate shard folders.
• Downloader checks tracker-advertised files + file_sizes before peer/HF download; complete local files return immediately and preserve any extra files already in the model folder.
• Verification: downloader/startup targeted subset passes (pytest tests/test_node_startup.py -k "download_shard or same_shard"). Full tests/test_node_startup.py has 46 passed and 4 unrelated Windows chmod/path separator failures.
• Live Windows confirmation: meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B reuses F:\_STORAGE\models\qwen3.6-35b-a3b, prints Cached at, registers, and reaches ready as node 5gMLrmyB-26b1f8a4204a.
• Follow-up fix: preset-model startup now starts the heartbeat thread after registration; without this, the node appeared briefly on the dashboard and was purged on first inference/route after heartbeat expiry. Tracker dashboard now has a "Console output" panel backed by /v1/console for node register/expiry, routing failures, and proxy events.
• Qwen3.6-35B-A3B CPU runtime cap (2026-07-08): the old reserve-based split could assign an 79 GB CPU node layers 0-36, but real partial loading can exceed that budget and die without a Python traceback. Node startup now clips oversized CPU auto-assignments before loading, and tracker CPU assignment uses a stricter runtime headroom factor; do not revert this to the old 20% reserve-only policy.
• Route hardening: tracker chat proxy and /v1/route diagnostics now use alias-aware preset node matching for split Qwen3.6 routes; dashboard derives grouped inference history from proxy route/complete console events and shows observed TPS after completion.
• Live proxy hardening: model lookup trims outer whitespace before alias matching (qwen3.6-35b-a3b resolves), and tracker route logs/dashboard queue depth combine heartbeat queue with tracker-local proxy in-flight counts so Postman-style bursts no longer show every selected route as queue 0.
• Split-shard streaming hardening: Qwen3.6-style distributed generation now emits SSE chunks token-by-token from the head node instead of buffering all generated text until completion. Tracker direct/relay stream proxy logs proxy progress with live tokens/TPS, dashboard Inference history shows currently processing requests with live TPS/tokens/queue, and relay stream completion no longer references an undefined session_id.
• Native Windows Qwen3.6-MoE import fix: flash-linear-attention imports triton; without triton-windows, startup fails with misleading Could not import module 'Qwen3_5MoeForCausalLM'. Installed triton-windows in C:\Users\popov\miniforge3 and added it as a Windows-only node dependency.