neuron-tai/.claude/memory/project-status.md

---
name: project-status
description: Current state of neuron-tai development as of 2026-07-02
metadata:
  node_type: memory
  type: project
---

# Project Status (2026-07-13)

## Distributed GGUF controller checkpoint (2026-07-21)

- All three fixed detached lanes from the 2026-07-18 checkpoint (Fable/DGR-025, Kimi/DGR-028, Terra/DGR-024) were reviewed, committed, and merged into `ralph/distributed-gguf-runtime`, now at `cd6b4d9`. The `.claude/worktrees/ralph-fable-loop`, `ralph-kimi-loop`, `ralph-terra-loop`, `ralph-cursor-loop`, and `ralph-next-task` worktrees were removed after merge (cursor-loop/next-task were idle with no work in progress). Only `.claude/worktrees/distributed-gguf-runtime` (the integration checkout) remains, plus the unrelated `fix-tracker-incomplete-snapshot` worktree (locked, not part of this arc).
- DGR-025 and DGR-028 merged cleanly with no conflicts (105 + 7 tests passing, 112 together). DGR-024 required real completion work first: the worktree's `prd.json` note was stale (written for the old, policy-rejected in-memory fake-seam story). The actual code already present (`shard_runtime_server.py`, `test_shard_runtime_harness.py`) had correctly pivoted to a real subprocess/socket gRPC harness with direct-vs-opaque-relay byte-identity proof, but was missing the required fail-closed negative paths (stale epoch, expired deadline, malformed/checksum-corrupt fragments, exhausted flow-control credit, duplicate idempotency steps, in-band/out-of-band cancel). Implemented those (`SessionState` per `route_session_id`, `_validate_bundle`), added 9 new tests (11 total, all passing), and rewrote `evidence/DGR-024/README.md` to describe the actual implementation instead of the nonexistent `FakeShardSeam`.
- `prd.json`'s `passes` field was left `false` for all three stories — that flag is only flipped by the project's own independent controller review process, not by whoever lands the merge. DGR-028's evidence README explicitly still awaits that P0/P1 review.
- Post-merge full suite (before the DGR-024 merge): 3 failed / 1116 passed / 20 skipped — down from the prior 9-failure baseline, and the 3 remaining failures (billing default-db, dynamic-routing ADD_SHARD/LOAD_SHARD, preset dedup) are pre-existing and unrelated to native/runtime-identity code. Did not get a chance to rerun the full suite after the DGR-024 merge landed (interrupted); only the focused `tests/test_shard_runtime_harness.py` (11 passed) was reconfirmed post-merge — a fresh full-suite run is worth doing before treating the whole arc as done.
- The unrelated dirty edits on the integration worktree (`model_catalog.py` / `tests/test_mining_cli.py`, adding a `Qwen2.5-Coder-1.5B-Instruct-Q2_K-GGUF` preset) were stashed during the merge sequence and popped back afterward — still uncommitted, as before.

**Why:** user directed "distributed-gguf-runtime is where we need to merge all ralph-* branches" and to prune worktrees once their task's work is merged, so completed lanes don't linger and new worktrees signal new tasks unambiguously.
**How to apply:** next Ralph session picking up this project should start from `ralph/distributed-gguf-runtime` at `cd6b4d9`, rerun the full suite once to get a clean current baseline, and check whether independent review has flipped DGR-024/025/028's `passes` flags before selecting new work.

## Distributed GGUF controller checkpoint (2026-07-18)

- Integration branch `ralph/distributed-gguf-runtime` is at `377bc3475c41b762ebbcab038c05adf15d7749d0`, matching its remote; preserve the unrelated dirty integration edits in `model_catalog.py` and `test_mining_cli.py`.
- Fixed detached lanes hold three claimed, uncommitted tasks: Fable DGR-025/Gitea #9 executing-artifact identity repair; Terra DGR-024/#8 fake gRPC seam repair; Kimi DGR-028/#12 numbered llama.cpp patch stack. No provider worker was running at the latest reconciliation, so continue these dirty trees before selecting new work.
- DGR-028 controller repair added working apply/reverse/verify enforcement, exact assumptions, and a passing exact-pin native fixture; final independent P0/P1 review is still required before commit/integration. DGR-024 and DGR-025 focused gates pass but likewise remain provisional pending the current independent reviews. Controller reruns on 2026-07-18 passed DGR-025 (235 impacted tests), DGR-024 (78 shared/focused tests), and DGR-028 (7 Python tests plus exact-pin native CTest 1/1 and clean apply/reverse). Integrate DGR-028 before DGR-025, then replay DGR-025 on the new integration HEAD and regenerate/retest its runtime fingerprint vectors because DGR-028 changes the pinned patch-stack/tree identity. The integration full-suite baseline is 9 failures/1076 passes/22 skips: missing optional zstd/langchain dependencies plus pre-existing billing and dynamic-routing expectations; lane-only extra failures map to stale DGR-023 projection ancestry and a timing-sensitive cancel test, not the focused story paths.

## Selected-node model placement (2026-07-14)

- Admin Model placement now opens a node selector for load and release; the control-plane accepts optional `node_id` and targets only that registry assignment. Multi-model serving remains supported through `ADD_SHARD` and `max_loaded_shards`.
- Total node pool resource values are rendered from `/v1/network/map`'s `node.capacity` contract. Route selection remains assignment/capability/throughput/queue based; capacity is used for placement and falls back to tracker defaults only if a node truly omits it.

## Distributed inference performance (2026-07-14)

`DIP-001` is done in `.scratch/distributed-inference-performance/`: the deterministic two-node Route Session stub benchmark covers direct/relay plus cached/stateless prefill and decode. Its JSON and concise summary explicitly attribute model execution, activation encode/decode, compression, connection setup, relay queueing, local HTTP forwarding, and end-to-end seam latency. `PYTHONPATH=packages/node pytest -q tests/test_route_session_benchmark.py` passed (7); the fixture assertion checks output-token identity and connection attempts.

> Doc reconciliation 2026-07-13: `docs/prd.json` tracks US-001…US-050 (048 memory budget, 049 mainnet pilot, 050 Qwen demand placement). ADRs 0025–0026 added (TAI phase B/C, assignment ownership).

All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:

- **BillingLedger** (packages/tracker/meshnet_tracker/billing.py): event-sourced USDT ledger, gossip-replicated across the hive (id-deduped events), SQLite-persisted. 90/10 split by work units, per-model per-1K-token pricing, 402 before routing.
- **Solana custodial adapter** (packages/contracts/meshnet_contracts/solana_adapter.py): urllib JSON-RPC + solders signing. NOTE: installed solana-py 0.40 has NO sync client — don't import solana.rpc.api / spl.token.client.
- **scripts/devnet_setup.py**: creates mock-USDT mint + treasury, writes .env.devnet; --mint-to funds test clients.
- **TrackerServer threads**: deposit watcher (exactly-once via deposit-<sig> event ids) + leader-only settlement loop (threshold OR max-period, dust floor, resend-by-settlement-id → no double-pay).
- **Forfeiture penalty**: validator forfeits pending balance + strike; 3 strikes ban; probation redirects shares to protocol cut. Math in packages/validator/README.md.
- **Web dashboard**: GET /dashboard on any tracker, embedded dashboard.html, 4s polling.
- **Observed routing throughput**: tracker records rolling observed tokens/sec per `(node_id, model)` from completed proxied inference requests, exposes it via `/v1/stats` and `/v1/network/map`, shows it on the dashboard, and prefers observed per-model TPS over startup benchmark for routing when samples exist.

Suite: 222 passed, 3 skipped (openai/langchain packages missing in .venv — pre-existing).

**Why:** design locked in ADR-0015 (USDT custodial settlement; TAI deferred, protocol cut = future TAI liquidity).
**How to apply:** next steps are live devnet verification (run devnet_setup.py, start tracker with --solana-rpc-url/--usdt-mint/--treasury-keypair --billing-db), then the TAI mint when volume justifies it. Work not yet committed to git as of session end — check git status.

## Alpha hardening (2026-07-05)

Implementation complete for alpha-scoped blockers in `.scratch/alpha-hardening/`: 16/22 issues are `done`, including auth/gossip/service-token, persisted strike/ban/reputation, zero starting credit + spend cap, tracker-authoritative accounting, wallet binding proof, TOPLOC audit primitive, hop bisection, reputation scoring/routing, adaptive audits, and penalty wiring. Remaining 6/22 are `ready-for-human` / post-alpha or ops-gated: 12–15 multi-tracker money/Raft ordering hardening, 17 duplicate US-020 renumbering approval, and 21 honest-noise calibration corpus before production audit thresholds. Current verification: `uv run pytest -k 'not test_legacy_start_without_port_uses_next_available_port'` passes 316, skips 3; full `uv run pytest` has one environmental failure while local `meshnet-node` PID 1263451 occupies port 7000.

Historical handoff note: `/mnt/c/Users/popov/Downloads/neuron-tai-alpha-handoff-2026-07-04.md` is useful for navigation and original audit context, but it predates the completed `.scratch/alpha-hardening/` planning artifacts. Treat its "missing ADR/issues/README" statements as stale; prefer `.scratch/alpha-hardening/README.md` and `.scratch/alpha-hardening/handoff.md` for current task order.

## Node capability admission P0 (2026-07-09)

Planning is ready at `.scratch/node-capability-admission/` with five sequential Ralph stories and ADR-0023. The design is model-agnostic: a Node must validate its selected Model Artifact/shard with a bounded real forward before Tracker routing; Qwen3.6 is only an optional development fixture. P0 adds a versioned local recipe-manifest/report contract, `meshnet-node doctor`, fail-closed startup admission, and tracker route gating. It intentionally excludes dynamic recipe/dependency installation and the future signed Node updater.

## Windows CUDA node (working as of 2026-07-01)
- miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
- RTX 4060 Laptop GPU, 8 GB VRAM, benchmark index ~11,200
- Run: `meshnet-node start --tracker https://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct`
- Known: tracker registration fails with `http://` — must use `https://`
- pynvml deprecation warning is harmless (use nvidia-ml-py to silence it)

## Model cache layout (2026-07-07)
- Node downloads now cache files directly under `<download_dir>/<model>/`, not `<model>/layers_<start>-<end>/`, so a wider cached layer assignment can satisfy a later narrower assignment without duplicate shard folders.
- Downloader checks tracker-advertised `files` + `file_sizes` before peer/HF download; complete local files return immediately and preserve any extra files already in the model folder.
- Verification: downloader/startup targeted subset passes (`pytest tests/test_node_startup.py -k "download_shard or same_shard"`). Full `tests/test_node_startup.py` has 46 passed and 4 unrelated Windows chmod/path separator failures.
- Live Windows confirmation: `meshnet-node start --tracker http://192.168.0.179:8080 --model Qwen3.6-35B-A3B` reuses `F:\_STORAGE\models\qwen3.6-35b-a3b`, prints `Cached at`, registers, and reaches ready as node `5gMLrmyB-26b1f8a4204a`.
- Follow-up fix: preset-model startup now starts the heartbeat thread after registration; without this, the node appeared briefly on the dashboard and was purged on first inference/route after heartbeat expiry. Tracker dashboard now has a "Console output" panel backed by `/v1/console` for node register/expiry, routing failures, and proxy events.
- Qwen3.6-35B-A3B CPU runtime cap (2026-07-08): the old reserve-based split could assign an 79 GB CPU node layers 0-36, but real partial loading can exceed that budget and die without a Python traceback. Node startup now clips oversized CPU auto-assignments before loading, and tracker CPU assignment uses a stricter runtime headroom factor; do not revert this to the old 20% reserve-only policy.
- Route hardening: tracker chat proxy and `/v1/route` diagnostics now use alias-aware preset node matching for split Qwen3.6 routes; dashboard derives grouped inference history from proxy route/complete console events and shows observed TPS after completion.
- Live proxy hardening: model lookup trims outer whitespace before alias matching (`qwen3.6-35b-a3b ` resolves), and tracker route logs/dashboard queue depth combine heartbeat queue with tracker-local proxy in-flight counts so Postman-style bursts no longer show every selected route as queue `0`.
- Split-shard streaming hardening: Qwen3.6-style distributed generation now emits SSE chunks token-by-token from the head node instead of buffering all generated text until completion. Tracker direct/relay stream proxy logs `proxy progress` with live tokens/TPS, dashboard Inference history shows currently processing requests with live TPS/tokens/queue, and relay stream completion no longer references an undefined `session_id`.
- Native Windows Qwen3.6-MoE import fix: `flash-linear-attention` imports `triton`; without `triton-windows`, startup fails with misleading `Could not import module 'Qwen3_5MoeForCausalLM'`. Installed `triton-windows` in `C:\Users\popov\miniforge3` and added it as a Windows-only node dependency.