Compare commits
2 Commits
4d4ab607ca
...
a938c19a82
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a938c19a82 | ||
|
|
1e0aa6ea8f |
13
.claude/memory/autonomous-work-style.md
Normal file
13
.claude/memory/autonomous-work-style.md
Normal file
@@ -0,0 +1,13 @@
|
||||
---
|
||||
name: autonomous-work-style
|
||||
description: Dobromir wants autonomous batch execution — ask only for architecture decisions, never permissions
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: feedback
|
||||
---
|
||||
|
||||
When given a backlog, work through all open tasks autonomously and report back when done. Ask questions only for implementation/architecture decisions that genuinely need his input (grilling-style, one at a time with a recommendation) — never for permission to proceed, and don't checkpoint between tasks. Running tests is ALWAYS allowed (allowlisted in settings).
|
||||
|
||||
**Why:** he said "work on all the tasks and come back when done. ask only for implementation or architecture decisions and not for permissions" and "I ALWAYS ALLOW running tests! stop asking" (2026-07-02, reward-system session).
|
||||
|
||||
**How to apply:** default to acting; batch the full backlog from docs/prd.json ([[project-status]]); surface completed-work summaries at the end, not between stories.
|
||||
@@ -1,27 +1,26 @@
|
||||
---
|
||||
name: project-status
|
||||
description: Current state of neuron-tai development as of 2026-07-01
|
||||
metadata:
|
||||
description: Current state of neuron-tai development as of 2026-07-02
|
||||
metadata:
|
||||
node_type: memory
|
||||
type: project
|
||||
originSessionId: 8fb120ee-7b8e-45be-98c0-b5ae9c64d1ec
|
||||
---
|
||||
|
||||
# Project Status (2026-07-01)
|
||||
# Project Status (2026-07-02)
|
||||
|
||||
29/30 user stories done. US-030 is the only open story, ready for ralph.
|
||||
All 35 user stories in docs/prd.json are done (35/35), including the reward-system arc US-030…US-035 completed 2026-07-02:
|
||||
|
||||
## US-030 — Manual route selection + hop-penalty benchmarking
|
||||
- Status: open / ready
|
||||
- Optional `"route": [node_id, ...]` in POST /v1/chat/completions body
|
||||
- `POST /v1/benchmark/hop-penalty` — privileged (non-empty Authorization header), fans out to 1/2/3-node routes, records per-hop latency
|
||||
- Results appended to `benchmark_results.json` in tracker working dir
|
||||
- `GET /v1/benchmark/results` — also auth-gated
|
||||
- Routing algorithm unchanged — data collection only
|
||||
- Source: `.scratch/distributed-inference-network/issues/30-manual-route-and-hop-benchmark.md`
|
||||
- **BillingLedger** (packages/tracker/meshnet_tracker/billing.py): event-sourced USDT ledger, gossip-replicated across the hive (id-deduped events), SQLite-persisted. 90/10 split by work units, per-model per-1K-token pricing, 402 before routing.
|
||||
- **Solana custodial adapter** (packages/contracts/meshnet_contracts/solana_adapter.py): urllib JSON-RPC + solders signing. NOTE: installed solana-py 0.40 has NO sync client — don't import solana.rpc.api / spl.token.client.
|
||||
- **scripts/devnet_setup.py**: creates mock-USDT mint + treasury, writes .env.devnet; --mint-to funds test clients.
|
||||
- **TrackerServer threads**: deposit watcher (exactly-once via deposit-<sig> event ids) + leader-only settlement loop (threshold OR max-period, dust floor, resend-by-settlement-id → no double-pay).
|
||||
- **Forfeiture penalty**: validator forfeits pending balance + strike; 3 strikes ban; probation redirects shares to protocol cut. Math in packages/validator/README.md.
|
||||
- **Web dashboard**: GET /dashboard on any tracker, embedded dashboard.html, 4s polling.
|
||||
|
||||
**Why:** Need real hop-latency data to eventually optimize route selection beyond synthetic benchmarks.
|
||||
**How to apply:** When asked about next steps, US-030 is the one ready story.
|
||||
Suite: 222 passed, 3 skipped (openai/langchain packages missing in .venv — pre-existing).
|
||||
|
||||
**Why:** design locked in ADR-0015 (USDT custodial settlement; TAI deferred, protocol cut = future TAI liquidity).
|
||||
**How to apply:** next steps are live devnet verification (run devnet_setup.py, start tracker with --solana-rpc-url/--usdt-mint/--treasury-keypair --billing-db), then the TAI mint when volume justifies it. Work not yet committed to git as of session end — check git status.
|
||||
|
||||
## Windows CUDA node (working as of 2026-07-01)
|
||||
- miniforge3 base env, torch 2.7.1+cu118, torchvision 0.22.x+cu118
|
||||
|
||||
12
.dockerignore
Normal file
12
.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
||||
.git
|
||||
.venv
|
||||
__pycache__
|
||||
*.py[cod]
|
||||
.pytest_cache
|
||||
*.egg-info
|
||||
build
|
||||
dist
|
||||
.ralph-tui
|
||||
.scratch
|
||||
.claude
|
||||
.env*
|
||||
282
CONTEXT.md
282
CONTEXT.md
@@ -1,141 +1,141 @@
|
||||
# Distributed Inference Network
|
||||
|
||||
A volunteer GPU network where nodes independently load model shards, a tracker routes inference through optimal node chains, and contributors earn tokens for serving compute.
|
||||
|
||||
## Language
|
||||
|
||||
### Nodes & compute
|
||||
|
||||
**Node**:
|
||||
A volunteer machine that runs the node client, holds one or more shards on disk, and serves inference requests for those shards.
|
||||
_Avoid_: worker, peer, miner, server
|
||||
|
||||
**Shard**:
|
||||
A contiguous range of transformer layers from a model that a node loads and serves. Shards are the unit of storage, assignment, and reward.
|
||||
_Avoid_: partition, slice, chunk, segment
|
||||
|
||||
**Shard Swarm**:
|
||||
The P2P group of nodes that collectively seed and download a specific shard. One swarm exists per shard.
|
||||
_Avoid_: torrent, cluster, pool
|
||||
|
||||
**Inference Route**:
|
||||
An ordered sequence of nodes whose shards together cover all layers of a model. The tracker selects the optimal route per request.
|
||||
_Avoid_: pipeline, chain, path
|
||||
|
||||
**Gateway**:
|
||||
The network entry point that accepts client requests (OpenAI-compatible HTTP), selects an inference route from the tracker, and streams results back.
|
||||
_Avoid_: proxy, relay, orchestrator, primary
|
||||
|
||||
### Tracker
|
||||
|
||||
**Tracker**:
|
||||
The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
|
||||
_Avoid_: coordinator, scheduler, director
|
||||
|
||||
**Tracker Node**:
|
||||
A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results back. Any node advertising a new model to the network becomes its tracker node.
|
||||
_Avoid_: primary node, master node, gateway node
|
||||
|
||||
**Coverage Map**:
|
||||
The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.
|
||||
_Avoid_: shard map, assignment table, coverage report
|
||||
|
||||
**Rebalance Directive**:
|
||||
A `LOAD_SHARD` or `DROP_SHARD` instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response.
|
||||
_Avoid_: rebalance command, shard instruction, migration order
|
||||
|
||||
**Node Score**:
|
||||
A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry.
|
||||
_Avoid_: reputation, rating, rank
|
||||
|
||||
### Payments & fraud
|
||||
|
||||
**Stake**:
|
||||
Collateral a node stands to lose for fraud. In the current design the node's Pending Balance serves as stake — no upfront deposit is required. An optional USDT/TAI deposit may return later for routing priority.
|
||||
_Avoid_: deposit, bond, escrow
|
||||
|
||||
**Treasury**:
|
||||
The single project-owned Solana wallet that custodially holds client deposits, pays node payouts, and accumulates the Protocol Cut. Its keypair is loaded only on settlement-capable trackers.
|
||||
_Avoid_: escrow, vault, hot wallet
|
||||
|
||||
**Pending Balance**:
|
||||
A node's accrued, not-yet-paid USDT earnings on the tracker ledger. Doubles as the node's fraud collateral: it is forfeited in full when a validator catches a divergent output.
|
||||
_Avoid_: unpaid rewards, accrual, balance due
|
||||
|
||||
**Settlement Period**:
|
||||
The dynamic interval driving on-chain payouts: a node is paid when its Pending Balance exceeds the Payout Threshold or the period elapses, whichever comes first. Short in development (seconds), long in production (daily), configurable to grow with volume.
|
||||
_Avoid_: epoch, payout cycle, billing cycle
|
||||
|
||||
**Payout Threshold**:
|
||||
The minimum Pending Balance that triggers an immediate payout before the Settlement Period elapses. Includes a dust floor so payouts are never smaller than they are worth.
|
||||
_Avoid_: minimum payout, dust limit
|
||||
|
||||
**Protocol Cut**:
|
||||
The 10% of inference fees retained by the project for infrastructure; the remaining 90% goes to the nodes that served the request. Accumulates in the Treasury as the future TAI liquidity reserve.
|
||||
_Avoid_: spread, commission, house fee
|
||||
|
||||
**Deposit Watcher**:
|
||||
The tracker component that observes the Treasury's on-chain USDT deposits and credits the sending client's API-key ledger balance.
|
||||
_Avoid_: payment listener, chain scanner
|
||||
|
||||
**Mock USDT**:
|
||||
The self-created 6-decimal SPL mint that stands in for USDT on devnet, where real USDT does not exist. The mint address is configuration, so mainnet cutover is a config change.
|
||||
_Avoid_: test token, fake USDT, devnet dollar
|
||||
|
||||
**Tax**:
|
||||
The share of caller payments distributed to compute nodes as rewards. Taxes are weighted by completed work and historical node speed so faster, larger nodes earn proportionally more.
|
||||
_Avoid_: fee, toll, commission
|
||||
|
||||
**Caller Credit**:
|
||||
Free starting balance granted to a new caller/API key so they can try the network before topping up.
|
||||
_Avoid_: signup bonus, faucet, airdrop
|
||||
|
||||
**Free Compute Job**:
|
||||
Work a compute node performs without earning immediate rewards, usually during probation or bootstrap phases.
|
||||
_Avoid_: unpaid labor, warmup request
|
||||
|
||||
**Slash**:
|
||||
The penalty for a proven fraud incident: the node's entire Pending Balance is forfeited to the Treasury and a Strike is recorded.
|
||||
_Avoid_: penalize, burn, fine, forfeit
|
||||
|
||||
**Strike**:
|
||||
A fraud incident recorded on-chain against a node. Enough strikes result in a ban.
|
||||
_Avoid_: infraction, violation, flag
|
||||
|
||||
**Ban**:
|
||||
Permanent exclusion of a wallet from the network after exceeding the strike threshold. Recorded on-chain.
|
||||
_Avoid_: blacklist, block, suspension
|
||||
|
||||
**Probationary Period**:
|
||||
The first N jobs a new wallet must complete without earning, to raise the cost of re-entering after a ban.
|
||||
_Avoid_: trial period, warmup, grace period
|
||||
|
||||
**Token**:
|
||||
TAI, our native Solana SPL token. Deferred (ADR-0015): nodes are currently paid directly in USDT; TAI returns as the reward/upside layer once volume exists, funded by the accumulated Protocol Cut. Clients never need to hold it.
|
||||
_Avoid_: coin, reward token, native token
|
||||
|
||||
**Contract Boundary**:
|
||||
The Python interface in `packages/contracts` that represents registry, payment, and settlement behavior. During the prototype it is implemented by deterministic local wrappers; later the same boundary is backed by real Solana programs.
|
||||
_Avoid_: mock contract, fake chain, temporary hack
|
||||
|
||||
**Validator**:
|
||||
A trusted node (or the tracker itself) that re-runs a sample of inference requests to detect fraud.
|
||||
_Avoid_: auditor, checker, referee
|
||||
|
||||
**Validation Event**:
|
||||
A completed inference record that contains enough information for a validator to decide whether to sample and re-run the request: session id, model preset, messages, inference route, node wallets, and observed output.
|
||||
_Avoid_: audit log, trace, receipt
|
||||
|
||||
**Slash Proof**:
|
||||
The record submitted by a validator when a sampled re-run diverges from the observed output beyond tolerance. In the prototype this is deterministic local contract state; later it maps to an on-chain proof transaction.
|
||||
_Avoid_: accusation, report, claim
|
||||
|
||||
### Client-facing
|
||||
|
||||
**Client**:
|
||||
Any application or user that sends inference requests to the gateway. Prepays USDT into the Treasury; each request is metered against the resulting ledger balance at a per-1K-tokens price set per model.
|
||||
_Avoid_: user, caller, consumer
|
||||
|
||||
**Model Preset**:
|
||||
A named, versioned model available on the network (e.g. `llama-3-70b`). The tracker knows which nodes hold which shards for each preset.
|
||||
_Avoid_: model, checkpoint, version
|
||||
# Distributed Inference Network
|
||||
|
||||
A volunteer GPU network where nodes independently load model shards, a tracker routes inference through optimal node chains, and contributors earn tokens for serving compute.
|
||||
|
||||
## Language
|
||||
|
||||
### Nodes & compute
|
||||
|
||||
**Node**:
|
||||
A volunteer machine that runs the node client, holds one or more shards on disk, and serves inference requests for those shards.
|
||||
_Avoid_: worker, peer, miner, server
|
||||
|
||||
**Shard**:
|
||||
A contiguous range of transformer layers from a model that a node loads and serves. Shards are the unit of storage, assignment, and reward.
|
||||
_Avoid_: partition, slice, chunk, segment
|
||||
|
||||
**Shard Swarm**:
|
||||
The P2P group of nodes that collectively seed and download a specific shard. One swarm exists per shard.
|
||||
_Avoid_: torrent, cluster, pool
|
||||
|
||||
**Inference Route**:
|
||||
An ordered sequence of nodes whose shards together cover all layers of a model. The tracker selects the optimal route per request.
|
||||
_Avoid_: pipeline, chain, path
|
||||
|
||||
**Gateway**:
|
||||
The network entry point that accepts client requests (OpenAI-compatible HTTP), selects an inference route from the tracker, and streams results back.
|
||||
_Avoid_: proxy, relay, orchestrator, primary
|
||||
|
||||
### Tracker
|
||||
|
||||
**Tracker**:
|
||||
The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback.
|
||||
_Avoid_: coordinator, scheduler, director
|
||||
|
||||
**Tracker Node**:
|
||||
A node that serves at least the first-layer shard (`layers[0..k]`) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and `embed_tokens`, receive client requests directly, select the onward route from the coverage map, and stream results back. Any node advertising a new model to the network becomes its tracker node.
|
||||
_Avoid_: primary node, master node, gateway node
|
||||
|
||||
**Coverage Map**:
|
||||
The tracker's per-model mapping of layer ranges to node counts: `[(start_layer, end_layer, node_count), ...]`. A layer range with `node_count=0` is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy.
|
||||
_Avoid_: shard map, assignment table, coverage report
|
||||
|
||||
**Rebalance Directive**:
|
||||
A `LOAD_SHARD` or `DROP_SHARD` instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response.
|
||||
_Avoid_: rebalance command, shard instruction, migration order
|
||||
|
||||
**Node Score**:
|
||||
A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry.
|
||||
_Avoid_: reputation, rating, rank
|
||||
|
||||
### Payments & fraud
|
||||
|
||||
**Stake**:
|
||||
Collateral a node stands to lose for fraud. In the current design the node's Pending Balance serves as stake — no upfront deposit is required. An optional USDT/TAI deposit may return later for routing priority.
|
||||
_Avoid_: deposit, bond, escrow
|
||||
|
||||
**Treasury**:
|
||||
The single project-owned Solana wallet that custodially holds client deposits, pays node payouts, and accumulates the Protocol Cut. Its keypair is loaded only on settlement-capable trackers.
|
||||
_Avoid_: escrow, vault, hot wallet
|
||||
|
||||
**Pending Balance**:
|
||||
A node's accrued, not-yet-paid USDT earnings on the tracker ledger. Doubles as the node's fraud collateral: it is forfeited in full when a validator catches a divergent output.
|
||||
_Avoid_: unpaid rewards, accrual, balance due
|
||||
|
||||
**Settlement Period**:
|
||||
The dynamic interval driving on-chain payouts: a node is paid when its Pending Balance exceeds the Payout Threshold or the period elapses, whichever comes first. Short in development (seconds), long in production (daily), configurable to grow with volume.
|
||||
_Avoid_: epoch, payout cycle, billing cycle
|
||||
|
||||
**Payout Threshold**:
|
||||
The minimum Pending Balance that triggers an immediate payout before the Settlement Period elapses. Includes a dust floor so payouts are never smaller than they are worth.
|
||||
_Avoid_: minimum payout, dust limit
|
||||
|
||||
**Protocol Cut**:
|
||||
The 10% of inference fees retained by the project for infrastructure; the remaining 90% goes to the nodes that served the request. Accumulates in the Treasury as the future TAI liquidity reserve.
|
||||
_Avoid_: spread, commission, house fee
|
||||
|
||||
**Deposit Watcher**:
|
||||
The tracker component that observes the Treasury's on-chain USDT deposits and credits the sending client's API-key ledger balance.
|
||||
_Avoid_: payment listener, chain scanner
|
||||
|
||||
**Mock USDT**:
|
||||
The self-created 6-decimal SPL mint that stands in for USDT on devnet, where real USDT does not exist. The mint address is configuration, so mainnet cutover is a config change.
|
||||
_Avoid_: test token, fake USDT, devnet dollar
|
||||
|
||||
**Tax**:
|
||||
The share of caller payments distributed to compute nodes as rewards. Taxes are weighted by completed work and historical node speed so faster, larger nodes earn proportionally more.
|
||||
_Avoid_: fee, toll, commission
|
||||
|
||||
**Caller Credit**:
|
||||
Free starting balance granted to a new caller/API key so they can try the network before topping up.
|
||||
_Avoid_: signup bonus, faucet, airdrop
|
||||
|
||||
**Free Compute Job**:
|
||||
Work a compute node performs without earning immediate rewards, usually during probation or bootstrap phases.
|
||||
_Avoid_: unpaid labor, warmup request
|
||||
|
||||
**Slash**:
|
||||
The penalty for a proven fraud incident: the node's entire Pending Balance is forfeited to the Treasury and a Strike is recorded.
|
||||
_Avoid_: penalize, burn, fine, forfeit
|
||||
|
||||
**Strike**:
|
||||
A fraud incident recorded on-chain against a node. Enough strikes result in a ban.
|
||||
_Avoid_: infraction, violation, flag
|
||||
|
||||
**Ban**:
|
||||
Permanent exclusion of a wallet from the network after exceeding the strike threshold. Recorded on-chain.
|
||||
_Avoid_: blacklist, block, suspension
|
||||
|
||||
**Probationary Period**:
|
||||
The first N jobs a new wallet must complete without earning, to raise the cost of re-entering after a ban.
|
||||
_Avoid_: trial period, warmup, grace period
|
||||
|
||||
**Token**:
|
||||
TAI, our native Solana SPL token. Deferred (ADR-0015): nodes are currently paid directly in USDT; TAI returns as the reward/upside layer once volume exists, funded by the accumulated Protocol Cut. Clients never need to hold it.
|
||||
_Avoid_: coin, reward token, native token
|
||||
|
||||
**Contract Boundary**:
|
||||
The Python interface in `packages/contracts` that represents registry, payment, and settlement behavior. During the prototype it is implemented by deterministic local wrappers; later the same boundary is backed by real Solana programs.
|
||||
_Avoid_: mock contract, fake chain, temporary hack
|
||||
|
||||
**Validator**:
|
||||
A trusted node (or the tracker itself) that re-runs a sample of inference requests to detect fraud.
|
||||
_Avoid_: auditor, checker, referee
|
||||
|
||||
**Validation Event**:
|
||||
A completed inference record that contains enough information for a validator to decide whether to sample and re-run the request: session id, model preset, messages, inference route, node wallets, and observed output.
|
||||
_Avoid_: audit log, trace, receipt
|
||||
|
||||
**Slash Proof**:
|
||||
The record submitted by a validator when a sampled re-run diverges from the observed output beyond tolerance. In the prototype this is deterministic local contract state; later it maps to an on-chain proof transaction.
|
||||
_Avoid_: accusation, report, claim
|
||||
|
||||
### Client-facing
|
||||
|
||||
**Client**:
|
||||
Any application or user that sends inference requests to the gateway. Prepays USDT into the Treasury; each request is metered against the resulting ledger balance at a per-1K-tokens price set per model.
|
||||
_Avoid_: user, caller, consumer
|
||||
|
||||
**Model Preset**:
|
||||
A named, versioned model available on the network (e.g. `llama-3-70b`). The tracker knows which nodes hold which shards for each preset.
|
||||
_Avoid_: model, checkpoint, version
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
# tracker
|
||||
.venv/bin/meshnet-tracker start --host 0.0.0.0 --port 8080
|
||||
.\.venv\Scripts\python.exe -m meshnet_tracker.cli start --host 127.0.0.1 --port 8080 --billing-db .\billing.sqlite
|
||||
|
||||
|
||||
# node
|
||||
.\.venv\Scripts\python.exe -m meshnet_node.cli start --tracker http://localhost:8080 --model Qwen/Qwen2.5-0.5B-Instruct --port 7000 --debug
|
||||
|
||||
@win
|
||||
.venv/bin/meshnet-node start --tracker http://192.168.0.179:8081 --model Qwen/Qwen2.5-0.5B-Instruct --shard-start 20 --advertise-host 192.168.0.20
|
||||
|
||||
-.\.venv\Scripts\meshnet-node.exe start http://192.168.0.179:8081 --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
||||
-.\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
||||
.\.venv\Scripts\meshnet-node.exe start http://192.168.0.179:8081 --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
||||
.\.venv\Scripts\meshnet-node.exe start --tracker http://ai.neuron.d-popov.com --model-id Qwen/Qwen2.5-0.5B-Instruct --advertise-host 192.168.0.20
|
||||
|
||||
we .\.venv\Scripts\meshnet-node.exe start `
|
||||
--tracker http://192.168.0.179:8081 `
|
||||
@@ -13,3 +21,18 @@
|
||||
HF_HOME=/run/media/popov/d/DEV/models .venv/bin/meshnet-node start --model-id Qwen/Qwen2.5-0.5B-Instruct --shard-start 0 --shard-end 21 --quantization bfloat16 --tracker http://localhost:8081
|
||||
# win
|
||||
meshnet-node start --tracker http://ai.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
|
||||
|
||||
|
||||
|
||||
|
||||
Then test tracker API:
|
||||
|
||||
curl http://localhost:8080/v1/health
|
||||
curl http://localhost:8080/v1/models
|
||||
|
||||
Because billing is enabled, chat calls need a Bearer key:
|
||||
|
||||
curl http://localhost:8080/v1/chat/completions `
|
||||
-H "Content-Type: application/json" `
|
||||
-H "Authorization: Bearer test-key" `
|
||||
-d '{"model":"stub-model","messages":[{"role":"user","content":"hi"}]}'
|
||||
BIN
billing.sqlite
Normal file
BIN
billing.sqlite
Normal file
Binary file not shown.
27
deploy/docker/Dockerfile
Normal file
27
deploy/docker/Dockerfile
Normal file
@@ -0,0 +1,27 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN useradd --create-home --uid 10001 meshnet \
|
||||
&& mkdir -p /var/lib/meshnet \
|
||||
&& chown meshnet:meshnet /var/lib/meshnet
|
||||
|
||||
COPY packages/contracts /app/packages/contracts
|
||||
COPY packages/tracker /app/packages/tracker
|
||||
COPY packages/relay /app/packages/relay
|
||||
|
||||
RUN python -m pip install --upgrade pip setuptools wheel \
|
||||
&& python -m pip install \
|
||||
-e /app/packages/contracts \
|
||||
-e /app/packages/tracker \
|
||||
-e /app/packages/relay
|
||||
|
||||
USER meshnet
|
||||
|
||||
EXPOSE 8081 8765
|
||||
|
||||
CMD ["meshnet-tracker", "start", "--host", "0.0.0.0", "--port", "8081"]
|
||||
153
deploy/portainer/meshnet-tracker-nobuild-stack.yml
Normal file
153
deploy/portainer/meshnet-tracker-nobuild-stack.yml
Normal file
@@ -0,0 +1,153 @@
|
||||
# Meshnet public tracker + relay stack for Portainer without a custom image.
|
||||
#
|
||||
# This stack does NOT use deploy/docker/Dockerfile and does NOT require pushing an
|
||||
# image to a registry. Each service starts from the public python:3.12-slim image,
|
||||
# downloads a source tarball, installs the tracker/relay packages into a named
|
||||
# venv volume, then starts the service.
|
||||
#
|
||||
# Required Portainer variables:
|
||||
# SOURCE_TARBALL_URL URL to a .tar.gz archive of this repo
|
||||
# PUBLIC_TRACKER_URL e.g. https://cloud.neuron.d-popov.com
|
||||
# PUBLIC_PROXY_NETWORK Docker network shared with nginx/NPM, e.g. npm_proxy
|
||||
#
|
||||
# Optional:
|
||||
# CLUSTER_PEERS e.g. https://ai.neuron.d-popov.com
|
||||
# PUBLIC_RELAY_URL defaults to PUBLIC_TRACKER_URL with wss/ws + /ws
|
||||
# SOURCE_STRIP_COMPONENTS defaults to 1 for GitHub/Gitea archive tarballs
|
||||
|
||||
services:
|
||||
meshnet-tracker:
|
||||
image: python:3.12-slim
|
||||
container_name: meshnet-tracker
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SOURCE_TARBALL_URL: ${SOURCE_TARBALL_URL:?set SOURCE_TARBALL_URL}
|
||||
SOURCE_STRIP_COMPONENTS: ${SOURCE_STRIP_COMPONENTS:-1}
|
||||
PUBLIC_TRACKER_URL: ${PUBLIC_TRACKER_URL:?set PUBLIC_TRACKER_URL, e.g. https://cloud.neuron.d-popov.com}
|
||||
CLUSTER_PEERS: ${CLUSTER_PEERS:-}
|
||||
PUBLIC_RELAY_URL: ${PUBLIC_RELAY_URL:-}
|
||||
HEARTBEAT_TIMEOUT: ${HEARTBEAT_TIMEOUT:-30}
|
||||
ENABLE_BILLING_DB: ${ENABLE_BILLING_DB:-1}
|
||||
SOLANA_RPC_URL: ${SOLANA_RPC_URL:-}
|
||||
USDT_MINT: ${USDT_MINT:-}
|
||||
MESHNET_TREASURY_KEYPAIR_B64: ${MESHNET_TREASURY_KEYPAIR_B64:-}
|
||||
SETTLE_PERIOD: ${SETTLE_PERIOD:-86400}
|
||||
PAYOUT_THRESHOLD: ${PAYOUT_THRESHOLD:-5}
|
||||
PAYOUT_DUST_FLOOR: ${PAYOUT_DUST_FLOOR:-0.01}
|
||||
command:
|
||||
- /bin/sh
|
||||
- -lc
|
||||
- |
|
||||
set -eu
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends ca-certificates curl tar
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
rm -rf /opt/meshnet-src
|
||||
mkdir -p /opt/meshnet-src
|
||||
curl -fsSL "$${SOURCE_TARBALL_URL}" -o /tmp/meshnet-src.tar.gz
|
||||
tar -xzf /tmp/meshnet-src.tar.gz -C /opt/meshnet-src --strip-components "$${SOURCE_STRIP_COMPONENTS:-1}"
|
||||
|
||||
python -m venv /opt/meshnet-venv
|
||||
/opt/meshnet-venv/bin/python -m pip install --upgrade pip setuptools wheel
|
||||
/opt/meshnet-venv/bin/pip install \
|
||||
-e /opt/meshnet-src/packages/contracts \
|
||||
-e /opt/meshnet-src/packages/tracker \
|
||||
-e /opt/meshnet-src/packages/relay
|
||||
|
||||
PEER_ARGS=""
|
||||
if [ -n "$${CLUSTER_PEERS:-}" ]; then
|
||||
PEER_ARGS="--cluster-peers $${CLUSTER_PEERS}"
|
||||
fi
|
||||
RELAY_URL="$${PUBLIC_RELAY_URL:-}"
|
||||
if [ -z "$${RELAY_URL}" ]; then
|
||||
RELAY_URL="$$(printf '%s' "$${PUBLIC_TRACKER_URL}" | sed 's#^https://#wss://#; s#^http://#ws://#')/ws"
|
||||
fi
|
||||
BILLING_ARGS=""
|
||||
if [ "$${ENABLE_BILLING_DB:-1}" = "1" ]; then
|
||||
BILLING_ARGS="--billing-db /var/lib/meshnet/billing.sqlite"
|
||||
fi
|
||||
TREASURY_ARGS=""
|
||||
if [ -n "$${SOLANA_RPC_URL:-}" ] || [ -n "$${USDT_MINT:-}" ] || [ -n "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
|
||||
if [ -z "$${SOLANA_RPC_URL:-}" ] || [ -z "$${USDT_MINT:-}" ] || [ -z "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
|
||||
echo "SOLANA_RPC_URL, USDT_MINT, and MESHNET_TREASURY_KEYPAIR_B64 must all be set together" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s' "$${MESHNET_TREASURY_KEYPAIR_B64}" | base64 -d > /var/lib/meshnet/treasury-keypair.json
|
||||
chmod 600 /var/lib/meshnet/treasury-keypair.json
|
||||
TREASURY_ARGS="--solana-rpc-url $${SOLANA_RPC_URL} --usdt-mint $${USDT_MINT} --treasury-keypair /var/lib/meshnet/treasury-keypair.json --settle-period $${SETTLE_PERIOD} --payout-threshold $${PAYOUT_THRESHOLD} --payout-dust-floor $${PAYOUT_DUST_FLOOR}"
|
||||
fi
|
||||
exec /opt/meshnet-venv/bin/meshnet-tracker start \
|
||||
--host 0.0.0.0 \
|
||||
--port 8081 \
|
||||
--heartbeat-timeout "$${HEARTBEAT_TIMEOUT}" \
|
||||
--self-url "$${PUBLIC_TRACKER_URL}" \
|
||||
--relay-url "$${RELAY_URL}" \
|
||||
--stats-db /var/lib/meshnet/tracker-stats.sqlite \
|
||||
$${BILLING_ARGS} \
|
||||
$${TREASURY_ARGS} \
|
||||
$${PEER_ARGS}
|
||||
volumes:
|
||||
- meshnet-tracker-data:/var/lib/meshnet
|
||||
- meshnet-tracker-venv:/opt/meshnet-venv
|
||||
expose:
|
||||
- "8081"
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
networks:
|
||||
- public-proxy
|
||||
|
||||
meshnet-relay:
|
||||
image: python:3.12-slim
|
||||
container_name: meshnet-relay
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
SOURCE_TARBALL_URL: ${SOURCE_TARBALL_URL:?set SOURCE_TARBALL_URL}
|
||||
SOURCE_STRIP_COMPONENTS: ${SOURCE_STRIP_COMPONENTS:-1}
|
||||
command:
|
||||
- /bin/sh
|
||||
- -lc
|
||||
- |
|
||||
set -eu
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends ca-certificates curl tar
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
rm -rf /opt/meshnet-src
|
||||
mkdir -p /opt/meshnet-src
|
||||
curl -fsSL "$${SOURCE_TARBALL_URL}" -o /tmp/meshnet-src.tar.gz
|
||||
tar -xzf /tmp/meshnet-src.tar.gz -C /opt/meshnet-src --strip-components "$${SOURCE_STRIP_COMPONENTS:-1}"
|
||||
|
||||
python -m venv /opt/meshnet-venv
|
||||
/opt/meshnet-venv/bin/python -m pip install --upgrade pip setuptools wheel
|
||||
/opt/meshnet-venv/bin/pip install \
|
||||
-e /opt/meshnet-src/packages/tracker \
|
||||
-e /opt/meshnet-src/packages/relay
|
||||
|
||||
exec /opt/meshnet-venv/bin/meshnet-relay --host 0.0.0.0 --port 8765 --log-level INFO
|
||||
volumes:
|
||||
- meshnet-relay-venv:/opt/meshnet-venv
|
||||
expose:
|
||||
- "8765"
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1', 8765), 3); s.close()"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
networks:
|
||||
- public-proxy
|
||||
|
||||
volumes:
|
||||
meshnet-tracker-data:
|
||||
meshnet-tracker-venv:
|
||||
meshnet-relay-venv:
|
||||
|
||||
networks:
|
||||
public-proxy:
|
||||
external: true
|
||||
name: ${PUBLIC_PROXY_NETWORK:-npm_proxy}
|
||||
107
deploy/portainer/meshnet-tracker-stack.yml
Normal file
107
deploy/portainer/meshnet-tracker-stack.yml
Normal file
@@ -0,0 +1,107 @@
|
||||
# Meshnet public tracker + relay stack for Portainer.
|
||||
#
|
||||
# Intended topology when Nginx Proxy Manager (or another nginx reverse proxy)
|
||||
# runs on the same Docker host:
|
||||
# https://YOUR_DOMAIN/v1/* -> meshnet-tracker:8081
|
||||
# https://YOUR_DOMAIN/ws -> meshnet-relay:8765 (WebSocket)
|
||||
# https://YOUR_DOMAIN/rpc/* -> meshnet-relay:8765 (WebSocket)
|
||||
#
|
||||
# Before deploying, create or identify the Docker network shared with nginx/NPM,
|
||||
# then set PUBLIC_PROXY_NETWORK to its name in Portainer environment variables.
|
||||
|
||||
services:
|
||||
meshnet-tracker:
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: deploy/docker/Dockerfile
|
||||
image: meshnet-tracker-relay:local
|
||||
container_name: meshnet-tracker
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
PUBLIC_TRACKER_URL: ${PUBLIC_TRACKER_URL:?set PUBLIC_TRACKER_URL, e.g. https://cloud.neuron.d-popov.com}
|
||||
CLUSTER_PEERS: ${CLUSTER_PEERS:-}
|
||||
PUBLIC_RELAY_URL: ${PUBLIC_RELAY_URL:-}
|
||||
HEARTBEAT_TIMEOUT: ${HEARTBEAT_TIMEOUT:-30}
|
||||
ENABLE_BILLING_DB: ${ENABLE_BILLING_DB:-1}
|
||||
SOLANA_RPC_URL: ${SOLANA_RPC_URL:-}
|
||||
USDT_MINT: ${USDT_MINT:-}
|
||||
MESHNET_TREASURY_KEYPAIR_B64: ${MESHNET_TREASURY_KEYPAIR_B64:-}
|
||||
SETTLE_PERIOD: ${SETTLE_PERIOD:-86400}
|
||||
PAYOUT_THRESHOLD: ${PAYOUT_THRESHOLD:-5}
|
||||
PAYOUT_DUST_FLOOR: ${PAYOUT_DUST_FLOOR:-0.01}
|
||||
command:
|
||||
- /bin/sh
|
||||
- -lc
|
||||
- |
|
||||
set -eu
|
||||
PEER_ARGS=""
|
||||
if [ -n "$${CLUSTER_PEERS:-}" ]; then
|
||||
PEER_ARGS="--cluster-peers $${CLUSTER_PEERS}"
|
||||
fi
|
||||
RELAY_URL="$${PUBLIC_RELAY_URL:-}"
|
||||
if [ -z "$${RELAY_URL}" ]; then
|
||||
RELAY_URL="$$(printf '%s' "$${PUBLIC_TRACKER_URL}" | sed 's#^https://#wss://#; s#^http://#ws://#')/ws"
|
||||
fi
|
||||
BILLING_ARGS=""
|
||||
if [ "$${ENABLE_BILLING_DB:-1}" = "1" ]; then
|
||||
BILLING_ARGS="--billing-db /var/lib/meshnet/billing.sqlite"
|
||||
fi
|
||||
TREASURY_ARGS=""
|
||||
if [ -n "$${SOLANA_RPC_URL:-}" ] || [ -n "$${USDT_MINT:-}" ] || [ -n "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
|
||||
if [ -z "$${SOLANA_RPC_URL:-}" ] || [ -z "$${USDT_MINT:-}" ] || [ -z "$${MESHNET_TREASURY_KEYPAIR_B64:-}" ]; then
|
||||
echo "SOLANA_RPC_URL, USDT_MINT, and MESHNET_TREASURY_KEYPAIR_B64 must all be set together" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s' "$${MESHNET_TREASURY_KEYPAIR_B64}" | base64 -d > /var/lib/meshnet/treasury-keypair.json
|
||||
chmod 600 /var/lib/meshnet/treasury-keypair.json
|
||||
TREASURY_ARGS="--solana-rpc-url $${SOLANA_RPC_URL} --usdt-mint $${USDT_MINT} --treasury-keypair /var/lib/meshnet/treasury-keypair.json --settle-period $${SETTLE_PERIOD} --payout-threshold $${PAYOUT_THRESHOLD} --payout-dust-floor $${PAYOUT_DUST_FLOOR}"
|
||||
fi
|
||||
exec meshnet-tracker start \
|
||||
--host 0.0.0.0 \
|
||||
--port 8081 \
|
||||
--heartbeat-timeout "$${HEARTBEAT_TIMEOUT}" \
|
||||
--self-url "$${PUBLIC_TRACKER_URL}" \
|
||||
--relay-url "$${RELAY_URL}" \
|
||||
--stats-db /var/lib/meshnet/tracker-stats.sqlite \
|
||||
$${BILLING_ARGS} \
|
||||
$${TREASURY_ARGS} \
|
||||
$${PEER_ARGS}
|
||||
volumes:
|
||||
- meshnet-tracker-data:/var/lib/meshnet
|
||||
expose:
|
||||
- "8081"
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8081/v1/health', timeout=3).read()"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
networks:
|
||||
- public-proxy
|
||||
|
||||
meshnet-relay:
|
||||
image: meshnet-tracker-relay:local
|
||||
container_name: meshnet-relay
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
meshnet-tracker:
|
||||
condition: service_started
|
||||
command: ["meshnet-relay", "--host", "0.0.0.0", "--port", "8765", "--log-level", "INFO"]
|
||||
expose:
|
||||
- "8765"
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1', 8765), 3); s.close()"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
networks:
|
||||
- public-proxy
|
||||
|
||||
volumes:
|
||||
meshnet-tracker-data:
|
||||
|
||||
networks:
|
||||
public-proxy:
|
||||
external: true
|
||||
name: ${PUBLIC_PROXY_NETWORK:-npm_proxy}
|
||||
171
docs/DEPLOY_PORTAINER.md
Normal file
171
docs/DEPLOY_PORTAINER.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# One-shot Portainer deploy: public tracker + relay
|
||||
|
||||
No Dockerfile, no custom image, no registry. Portainer runs `python:3.12-slim`, downloads this repo tarball, installs tracker/relay, and starts both services.
|
||||
|
||||
## 1. One-time host setup
|
||||
|
||||
DNS points to the Docker host, e.g. `cloud.neuron.d-popov.com`.
|
||||
Firewall exposes only `80`, `443`, and admin SSH.
|
||||
Nginx Proxy Manager and this stack share a Docker network, e.g. `npm_proxy`.
|
||||
|
||||
If needed:
|
||||
|
||||
```bash
|
||||
docker network create npm_proxy
|
||||
docker network connect npm_proxy <nginx-proxy-manager-container>
|
||||
```
|
||||
|
||||
## 2. Portainer stack
|
||||
|
||||
Portainer → Stacks → Add stack → Repository.
|
||||
|
||||
Stack file:
|
||||
|
||||
```text
|
||||
deploy/portainer/meshnet-tracker-nobuild-stack.yml
|
||||
```
|
||||
|
||||
Deploy only after the current branch has been pushed. The stack passes the
|
||||
current tracker billing flags (`--billing-db`, and optional treasury flags), so
|
||||
an older remote `master.tar.gz` will crash on startup with unrecognized
|
||||
arguments.
|
||||
|
||||
Variables:
|
||||
|
||||
```text
|
||||
SOURCE_TARBALL_URL=https://git.d-popov.com/<owner>/<repo>/archive/master.tar.gz
|
||||
PUBLIC_TRACKER_URL=https://cloud.neuron.d-popov.com
|
||||
PUBLIC_PROXY_NETWORK=npm_proxy
|
||||
CLUSTER_PEERS=https://ai.neuron.d-popov.com
|
||||
PUBLIC_RELAY_URL=wss://cloud.neuron.d-popov.com/ws
|
||||
HEARTBEAT_TIMEOUT=30
|
||||
ENABLE_BILLING_DB=1
|
||||
```
|
||||
|
||||
For first cloud-only test, use `CLUSTER_PEERS=`. Click **Deploy the stack**.
|
||||
|
||||
`ENABLE_BILLING_DB=1` makes billing public behavior active: `/v1/chat/completions`
|
||||
requires `Authorization: Bearer <api-key>`. Any key is accepted and starts with
|
||||
the configured starter credit; calls return `402` after that balance is
|
||||
exhausted. Set `ENABLE_BILLING_DB=0` if existing unauthenticated clients must
|
||||
keep working during the first redeploy.
|
||||
|
||||
Optional Solana treasury settlement variables:
|
||||
|
||||
```text
|
||||
SOLANA_RPC_URL=https://api.devnet.solana.com
|
||||
USDT_MINT=<mock-usdt-mint-from-scripts/devnet_setup.py>
|
||||
MESHNET_TREASURY_KEYPAIR_B64=<base64 of solana keypair JSON>
|
||||
SETTLE_PERIOD=86400
|
||||
PAYOUT_THRESHOLD=5
|
||||
PAYOUT_DUST_FLOOR=0.01
|
||||
```
|
||||
|
||||
Set all three treasury identity values together (`SOLANA_RPC_URL`, `USDT_MINT`,
|
||||
`MESHNET_TREASURY_KEYPAIR_B64`) or leave all three empty. The stack writes the
|
||||
decoded keypair into the tracker data volume at startup and passes it to
|
||||
`meshnet-tracker`; do not put this key on relay-only hosts or non-settlement
|
||||
trackers.
|
||||
|
||||
Expected containers:
|
||||
|
||||
```text
|
||||
meshnet-tracker internal 8081
|
||||
meshnet-relay internal 8765
|
||||
```
|
||||
|
||||
## 3. Nginx Proxy Manager
|
||||
|
||||
One Proxy Host for `cloud.neuron.d-popov.com`:
|
||||
|
||||
```text
|
||||
Forward Hostname/IP: meshnet-tracker
|
||||
Forward Port: 8081
|
||||
Websockets Support: ON
|
||||
SSL: Let's Encrypt + Force SSL
|
||||
```
|
||||
|
||||
Custom locations on the same proxy host:
|
||||
|
||||
```text
|
||||
/ws -> http://meshnet-relay:8765
|
||||
/rpc -> http://meshnet-relay:8765
|
||||
```
|
||||
|
||||
Leave sub-folder forwarding empty.
|
||||
|
||||
If WebSockets fail, Advanced:
|
||||
|
||||
```nginx
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $http_connection;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
```
|
||||
|
||||
## 4. Smoke test
|
||||
|
||||
```bash
|
||||
curl -s https://cloud.neuron.d-popov.com/v1/health
|
||||
curl -s https://cloud.neuron.d-popov.com/v1/network/map | python3 -m json.tool
|
||||
curl -s https://cloud.neuron.d-popov.com/v1/raft/status | python3 -m json.tool
|
||||
```
|
||||
|
||||
Plain curl to `/ws` or `/rpc/test-peer` may show `426 Upgrade Required`; OK. It must not show nginx `502`.
|
||||
|
||||
Dashboard:
|
||||
|
||||
```bash
|
||||
curl -s https://cloud.neuron.d-popov.com/dashboard | head
|
||||
```
|
||||
|
||||
Then open:
|
||||
|
||||
```text
|
||||
https://cloud.neuron.d-popov.com/dashboard
|
||||
```
|
||||
|
||||
The dashboard is served by the tracker through the same proxy host; no extra NPM
|
||||
location is required.
|
||||
|
||||
If you previously deployed the build-image variant before `/var/lib/meshnet`
|
||||
was created as the `meshnet` user, the named volume may already be root-owned.
|
||||
Recreate that volume or chown it once before retrying the fixed image.
|
||||
|
||||
## 5. Start a node
|
||||
|
||||
`meshnet-node` is the worker/miner process. It will ask for a model and load a
|
||||
model shard. Do not use it to test the tracker dashboard.
|
||||
|
||||
```bash
|
||||
meshnet-node start --tracker https://cloud.neuron.d-popov.com --model Qwen/Qwen2.5-0.5B-Instruct
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```text
|
||||
Relay advertised by tracker — using outbound tunnel wss://cloud.neuron.d-popov.com/ws
|
||||
Relay connected — wss://cloud.neuron.d-popov.com/rpc/<peer_id>
|
||||
```
|
||||
|
||||
## 6. Two tracker sync
|
||||
|
||||
Cloud stack:
|
||||
|
||||
```text
|
||||
PUBLIC_TRACKER_URL=https://cloud.neuron.d-popov.com
|
||||
CLUSTER_PEERS=https://ai.neuron.d-popov.com
|
||||
PUBLIC_RELAY_URL=wss://cloud.neuron.d-popov.com/ws
|
||||
```
|
||||
|
||||
Local tracker:
|
||||
|
||||
```bash
|
||||
meshnet-tracker start --host 0.0.0.0 --port 8081 \
|
||||
--self-url https://ai.neuron.d-popov.com \
|
||||
--cluster-peers https://cloud.neuron.d-popov.com \
|
||||
--relay-url wss://ai.neuron.d-popov.com/ws
|
||||
```
|
||||
|
||||
Two Raft peers are enough for sync testing; real HA needs three trackers.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Coverage-first shard assignment and tracker-as-first-layer-node
|
||||
# Coverage-first shard assignment and tracker-routed inference
|
||||
|
||||
The tracker assigns shard ranges to nodes using a coverage-first, speed-weighted bin-packing algorithm. Tracker nodes must host at least the first layer shard of every model they coordinate, making them the natural inference entry point. Any node serving layers[0..k] can become a tracker node for that model.
|
||||
The tracker assigns shard ranges to worker nodes using a coverage-first, speed-weighted bin-packing algorithm. The tracker is a control-plane service and public inference API endpoint: it stores registry state, selects routes, enforces billing, and proxies OpenAI-compatible requests to the selected head worker. It does not download or load model weights.
|
||||
|
||||
## Problem
|
||||
|
||||
@@ -23,24 +23,29 @@ Example: 700B NF4 model (~350GB weights). Node A has 128GB, Node B and C each ha
|
||||
- Node C gets layers[k_b..N] (the remainder)
|
||||
- If Node B benchmarks 2× faster than Node C, the tracker shifts the B/C boundary so B carries more layers
|
||||
|
||||
### Tracker-as-first-layer-node
|
||||
### Tracker-routed head worker
|
||||
|
||||
Any node that advertises a new model to the network becomes a **tracker node** for that model. Tracker nodes have one hard requirement: they must hold and serve `layers[0..k]` (the first-layer shard) for every model they coordinate.
|
||||
A worker that serves `layers[0..k]` is the **head worker** for that model. The tracker forwards `/v1/chat/completions` to a live head worker and injects the remaining downstream route. The worker, not the tracker process:
|
||||
|
||||
The reason is functional: a tracker node is also the inference entry point. When a client request arrives, the tracker node:
|
||||
When a client request arrives, the tracker:
|
||||
1. Authenticates/bills the request
|
||||
2. Selects a live head worker and full downstream route from the coverage map
|
||||
3. Proxies the request to that head worker
|
||||
4. Records usage and credits node shares after completion
|
||||
|
||||
The head worker:
|
||||
1. Tokenizes the input (owns the tokenizer)
|
||||
2. Runs `model.embed_tokens` + `model.layers[0..k]`
|
||||
3. Selects the optimal onward route from the coverage map
|
||||
4. Forwards activations to the next node in the route
|
||||
5. Receives the final hidden state back and streams tokens to the client
|
||||
3. Forwards activations to the next node in the route
|
||||
4. Receives the final hidden state back and streams tokens to the client
|
||||
|
||||
This collapses the separate "gateway" role: the tracker node that starts an inference request IS the gateway for that request. A standalone HTTP proxy/load-balancer may sit in front to pick which tracker node handles the request, but it carries no model weights.
|
||||
This keeps the public tracker lightweight: a standalone HTTP proxy/load-balancer may sit in front to pick which tracker handles the request, but neither proxy nor tracker carries model weights.
|
||||
|
||||
Multiple tracker nodes for the same model = multiple entry points = horizontal scale for both routing decisions and the first-layer compute.
|
||||
Multiple head workers for the same model = multiple inference entry points = horizontal scale for first-layer compute. Multiple trackers scale routing and billing, not model execution.
|
||||
|
||||
### Last-layer node (tail)
|
||||
|
||||
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the tracker node, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
|
||||
The node assigned `layers[N-k..N]` also runs `model.norm` and `model.lm_head`. It returns decoded token IDs (not hidden states) to the head worker, which assembles the response. The tail shard assignment is marked `is_tail: true` in the shard registry.
|
||||
|
||||
### Adaptive quantization
|
||||
|
||||
@@ -78,8 +83,8 @@ Nodes obey directives asynchronously; the tracker waits up to a configurable tim
|
||||
|
||||
## Consequences
|
||||
|
||||
- The standalone `meshnet-gateway` service from US-005 becomes a thin load-balancer that routes to tracker nodes; tracker nodes do the actual inference orchestration
|
||||
- Tracker nodes must download more model data (tokenizer + first-layer shard) — this is the price of being an entry point
|
||||
- The standalone `meshnet-gateway` service from US-005 becomes a thin compatibility proxy/load-balancer; the public tracker can also serve the OpenAI-compatible endpoint directly
|
||||
- Tracker processes do not download or load model data. Only worker nodes load model shards.
|
||||
- Benchmark data is self-reported by nodes at registration; the validator can detect fraudulent benchmarks (a node claiming 100 tokens/sec but delivering 2 gets slashed for under-performance)
|
||||
- VRAM reservation for KV cache means nodes can host fewer layers than their raw VRAM suggests — this is intentional; running out of KV cache during inference causes OOM crashes
|
||||
- New CONTEXT.md terms: **Tracker Node** (node serving first-layer shard + inference routing for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)
|
||||
- New CONTEXT.md terms: **Head Worker** (worker node serving first-layer shard for a model), **Coverage Map** (tracker's per-model layer-range → node-count mapping), **Rebalance Directive** (tracker instruction to a node to load or drop a shard)
|
||||
|
||||
@@ -1,48 +1,48 @@
|
||||
# US-020 — Manual route selection + hop-penalty benchmarking
|
||||
|
||||
## Context
|
||||
|
||||
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
|
||||
the real cost of adding hops (latency per node boundary), we need:
|
||||
|
||||
1. A way to pin a request to a specific route so we control the variable.
|
||||
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
|
||||
routes and records per-hop latency.
|
||||
|
||||
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
|
||||
data collection only. The data will inform a future routing optimisation story.
|
||||
|
||||
## Design decisions (grilled 2026-07-01)
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
|
||||
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
|
||||
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
|
||||
| Routing integration | Store data only; routing algorithm unchanged |
|
||||
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
|
||||
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
|
||||
If absent, existing routing is unchanged (no breaking change for unaware clients).
|
||||
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
|
||||
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
|
||||
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
|
||||
"max_new_tokens": 64}`.
|
||||
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
|
||||
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
|
||||
currently registered. Routes with insufficient coverage are skipped, not errored.
|
||||
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
|
||||
`tokens_generated`, `route: [node_id, ...]`.
|
||||
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
|
||||
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
|
||||
per-route breakdown.
|
||||
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
|
||||
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
|
||||
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
|
||||
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
|
||||
in `benchmark_results.json`.
|
||||
- `python -m pytest` passes from repo root.
|
||||
- Commit only this story's changes.
|
||||
# US-020 — Manual route selection + hop-penalty benchmarking
|
||||
|
||||
## Context
|
||||
|
||||
The tracker auto-selects inference routes based on synthetic benchmark scores. To measure
|
||||
the real cost of adding hops (latency per node boundary), we need:
|
||||
|
||||
1. A way to pin a request to a specific route so we control the variable.
|
||||
2. A benchmark endpoint that runs the same prompt through 1-node, 2-node, and 3-node
|
||||
routes and records per-hop latency.
|
||||
|
||||
Results are stored to disk. Routing algorithm is **not** changed in this story — this is
|
||||
data collection only. The data will inform a future routing optimisation story.
|
||||
|
||||
## Design decisions (grilled 2026-07-01)
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Route spec | Optional `route` field in JSON request body (list of node IDs) |
|
||||
| Trigger | Explicit only — `POST /v1/benchmark/hop-penalty` endpoint |
|
||||
| Auth | Header-presence stub (`Authorization` must be non-empty); real auth in future story |
|
||||
| Routing integration | Store data only; routing algorithm unchanged |
|
||||
| Persistence | Append to `benchmark_results.json` in tracker working dir; in-memory queryable |
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `POST /v1/chat/completions` accepts optional `"route": ["<node_id>", ...]` in the
|
||||
request body. If present, the tracker uses those nodes in order instead of auto-selecting.
|
||||
If absent, existing routing is unchanged (no breaking change for unaware clients).
|
||||
- Missing or invalid node IDs in `route` return HTTP 400 with a descriptive error.
|
||||
- `POST /v1/benchmark/hop-penalty` is auth-gated: requests without a non-empty
|
||||
`Authorization` header return HTTP 401. Body: `{"model": "...", "prompt": "...",
|
||||
"max_new_tokens": 64}`.
|
||||
- Benchmark fans out to up to three routes: 1-node (single node covering all layers),
|
||||
2-node (two consecutive shard nodes), 3-node (three nodes) — using whatever is
|
||||
currently registered. Routes with insufficient coverage are skipped, not errored.
|
||||
- Response includes per-route breakdown: `total_ms`, `per_hop_ms: [...]`,
|
||||
`tokens_generated`, `route: [node_id, ...]`.
|
||||
- Results are appended to `<tracker_working_dir>/benchmark_results.json` (created if
|
||||
absent) as a JSON array. Each entry includes timestamp, model, prompt hash, and the
|
||||
per-route breakdown.
|
||||
- `GET /v1/benchmark/results` returns the stored results array. Also auth-gated.
|
||||
- Clients that never send `route` or call `/v1/benchmark/*` are completely unaffected.
|
||||
- Integration test: send the same prompt via a pinned 1-node route and a pinned 2-node
|
||||
route; assert 2-node result has 2 entries in `per_hop_ms`; assert both records appear
|
||||
in `benchmark_results.json`.
|
||||
- `python -m pytest` passes from repo root.
|
||||
- Commit only this story's changes.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
Status: ready-for-agent
|
||||
Status: done
|
||||
|
||||
# 32 — Devnet custodial treasury: mock-USDT mint + deposit watcher
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
Status: ready-for-agent
|
||||
Status: done
|
||||
|
||||
# 33 — Settlement loop: leader-only batched USDT payouts
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
Status: ready-for-agent
|
||||
Status: done
|
||||
|
||||
# 34 — Hardened proof-of-work: pending-balance forfeiture penalty
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
Status: ready-for-agent
|
||||
Status: done
|
||||
|
||||
# 35 — Tracker web dashboard
|
||||
|
||||
|
||||
@@ -704,12 +704,13 @@
|
||||
"Commit only this story's changes"
|
||||
],
|
||||
"priority": 30,
|
||||
"status": "open",
|
||||
"status": "done",
|
||||
"notes": "Source issue: .scratch/distributed-inference-network/issues/30-manual-route-and-hop-benchmark.md. Design decisions grilled 2026-07-01: route via body field, explicit-only benchmark trigger, auth stub, routing algorithm unchanged, persist to JSON file.",
|
||||
"dependsOn": [
|
||||
"US-014",
|
||||
"US-019"
|
||||
]
|
||||
],
|
||||
"completionNotes": "Pinned routes: optional \"route\": [node_id,...] in POST /v1/chat/completions (400 on unknown/invalid ids; absent field unchanged). POST /v1/benchmark/hop-penalty + GET /v1/benchmark/results, both auth-gated (non-empty Authorization). Fans out to 1/2/3-node routes via _find_pinned_route; per-hop latency derived incrementally (k-node total minus (k-1)-node total); appends to benchmark_results.json (path configurable). 6 tests in tests/test_manual_route_benchmark.py."
|
||||
},
|
||||
{
|
||||
"id": "US-031",
|
||||
@@ -745,11 +746,12 @@
|
||||
"Deterministic local adapter still works for CI without any validator"
|
||||
],
|
||||
"priority": 32,
|
||||
"status": "open",
|
||||
"status": "done",
|
||||
"notes": "Supersedes 'testnet never devnet' note in issue 06. solana-py + spl-token client libs.",
|
||||
"dependsOn": [
|
||||
"US-031"
|
||||
]
|
||||
],
|
||||
"completionNotes": "SolanaCustodialTreasury in packages/contracts/meshnet_contracts/solana_adapter.py: urllib JSON-RPC client + solders signing (installed solana-py 0.40 has no sync client), deposit parsing from jsonParsed token balances, batched payouts, devnet helpers (create_mock_usdt_mint, mint_mock_usdt). scripts/devnet_setup.py writes .env.devnet. POST /v1/wallet/register binds wallet→api_key (replicated ledger 'bind' events). Deposit watcher thread credits exactly-once via deposit-<signature> event ids. Tests: fake-treasury watcher tests + skipif-gated solana-test-validator e2e."
|
||||
},
|
||||
{
|
||||
"id": "US-033",
|
||||
@@ -764,12 +766,13 @@
|
||||
"End-to-end devnet test: fund client → run inference → observe node wallet USDT balance increase"
|
||||
],
|
||||
"priority": 33,
|
||||
"status": "open",
|
||||
"status": "done",
|
||||
"notes": "Mining-pool standard: threshold + max-interval, dust floor. Treasury key only on settlement-capable trackers.",
|
||||
"dependsOn": [
|
||||
"US-019",
|
||||
"US-032"
|
||||
]
|
||||
],
|
||||
"completionNotes": "Settlement loop in TrackerServer: leader-only (raft.is_leader; standalone settles), trigger pending≥threshold OR pending age≥settle_period, dust floor. Pending debited via payout events referencing settlement id BEFORE send; unconfirmed batches resent with same id → no double-pay. confirm via 'settlement' event with tx signature. GET /v1/billing/settlements. CLI: --settle-period/--payout-threshold/--payout-dust-floor (prod 24h/5/0.01; dev 60/0). Banned wallets skipped. 6 tests in tests/test_settlement_loop.py incl. non-leader-never-signs."
|
||||
},
|
||||
{
|
||||
"id": "US-034",
|
||||
@@ -784,11 +787,12 @@
|
||||
"Slash/forfeiture events visible in tracker logs and settlement history"
|
||||
],
|
||||
"priority": 34,
|
||||
"status": "open",
|
||||
"status": "done",
|
||||
"notes": "No stake deposit — pending balance IS the collateral. Amends ADR-0003 penalty; sampling mechanics unchanged.",
|
||||
"dependsOn": [
|
||||
"US-031"
|
||||
]
|
||||
],
|
||||
"completionNotes": "ValidatorProcess(billing=ledger) forfeits entire pending balance on divergence (same cycle as strike). Tracker POST /v1/billing/forfeit (auth stub) for remote validators: forfeit + strike + ban state. Probation wired into _bill_completed: wallets with probationary_jobs_remaining>0 have their share redirected to protocol cut; record_completed_job per request. Penalty math in packages/validator/README.md. 5 tests in tests/test_forfeiture_penalty.py; banned-never-paid asserted in settlement tests."
|
||||
},
|
||||
{
|
||||
"id": "US-035",
|
||||
@@ -802,12 +806,13 @@
|
||||
"Works against a 3-tracker hive in the two-machine LAN test setup"
|
||||
],
|
||||
"priority": 35,
|
||||
"status": "open",
|
||||
"status": "done",
|
||||
"notes": "CLI dashboard already exists from US-016; this is the web counterpart. Write ops (price config, manual settle) deferred.",
|
||||
"dependsOn": [
|
||||
"US-031",
|
||||
"US-033"
|
||||
]
|
||||
],
|
||||
"completionNotes": "GET /dashboard served from embedded dashboard.html (package-data, no build step) by any tracker. Panels: hive/leader (raft status), nodes+coverage grouped by model, client balances, node pending + protocol cut, settlement history with devnet explorer links, strikes/bans/forfeitures (GET /v1/registry/wallets + snapshot forfeits), RPM stats. 4s auto-refresh via fetch polling. 3 tests in tests/test_dashboard.py."
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
297
packages/contracts/meshnet_contracts/solana_adapter.py
Normal file
297
packages/contracts/meshnet_contracts/solana_adapter.py
Normal file
@@ -0,0 +1,297 @@
|
||||
"""Custodial Solana treasury adapter (ADR-0015, US-032/US-033).
|
||||
|
||||
The entire on-chain surface is plain SPL token operations against a single
|
||||
project-owned treasury wallet: read confirmed incoming USDT transfers, send
|
||||
batched payout transfers, plus devnet helpers to create the mock-USDT mint.
|
||||
No Anchor programs.
|
||||
|
||||
RPC transport is a minimal urllib JSON-RPC client (the installed solana-py
|
||||
0.40 removed its sync client); signing and instruction building use solders /
|
||||
spl.token. Import this module only where a cluster is actually configured —
|
||||
the deterministic local boundary in ``meshnet_contracts`` stays
|
||||
dependency-free for CI.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
|
||||
from solders.hash import Hash
|
||||
from solders.keypair import Keypair
|
||||
from solders.pubkey import Pubkey
|
||||
from solders.system_program import CreateAccountParams, create_account
|
||||
from solders.transaction import Transaction
|
||||
from spl.token.constants import TOKEN_PROGRAM_ID
|
||||
from spl.token.instructions import (
|
||||
create_associated_token_account,
|
||||
get_associated_token_address,
|
||||
initialize_mint,
|
||||
mint_to,
|
||||
transfer_checked,
|
||||
)
|
||||
from spl.token.models import (
|
||||
InitializeMintParams,
|
||||
MintToParams,
|
||||
TransferCheckedParams,
|
||||
)
|
||||
|
||||
USDT_DECIMALS = 6 # matches real USDT on Solana mainnet
|
||||
_MINT_ACCOUNT_SIZE = 82
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Deposit:
|
||||
"""A confirmed incoming USDT transfer into the treasury token account."""
|
||||
|
||||
signature: str
|
||||
sender_wallet: str
|
||||
amount_usdt: float
|
||||
|
||||
|
||||
def load_keypair(path: str) -> Keypair:
|
||||
"""Load a keypair from a solana-cli style JSON byte-array file."""
|
||||
with open(os.path.expanduser(path), encoding="utf-8") as fh:
|
||||
secret = json.load(fh)
|
||||
return Keypair.from_bytes(bytes(secret))
|
||||
|
||||
|
||||
class RpcClient:
|
||||
"""Minimal synchronous Solana JSON-RPC client."""
|
||||
|
||||
def __init__(self, url: str, timeout: float = 30.0) -> None:
|
||||
self.url = url
|
||||
self._timeout = timeout
|
||||
self._request_id = 0
|
||||
|
||||
def call(self, method: str, params: list | None = None):
|
||||
self._request_id += 1
|
||||
body = json.dumps({
|
||||
"jsonrpc": "2.0",
|
||||
"id": self._request_id,
|
||||
"method": method,
|
||||
"params": params or [],
|
||||
}).encode()
|
||||
request = urllib.request.Request(
|
||||
self.url,
|
||||
data=body,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(request, timeout=self._timeout) as response:
|
||||
payload = json.loads(response.read())
|
||||
if "error" in payload:
|
||||
raise RuntimeError(f"RPC {method} failed: {payload['error']}")
|
||||
return payload.get("result")
|
||||
|
||||
|
||||
class SolanaCustodialTreasury:
|
||||
"""Read deposits into / send payouts from the custodial treasury wallet."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
rpc_url: str,
|
||||
usdt_mint: str,
|
||||
treasury_keypair: Keypair | str,
|
||||
*,
|
||||
decimals: int = USDT_DECIMALS,
|
||||
) -> None:
|
||||
self.rpc = RpcClient(rpc_url)
|
||||
self._mint = Pubkey.from_string(usdt_mint)
|
||||
self._treasury = (
|
||||
load_keypair(treasury_keypair)
|
||||
if isinstance(treasury_keypair, str)
|
||||
else treasury_keypair
|
||||
)
|
||||
self._decimals = decimals
|
||||
self._treasury_ata = get_associated_token_address(
|
||||
self._treasury.pubkey(), self._mint
|
||||
)
|
||||
|
||||
@property
|
||||
def treasury_wallet(self) -> str:
|
||||
return str(self._treasury.pubkey())
|
||||
|
||||
@property
|
||||
def treasury_token_account(self) -> str:
|
||||
return str(self._treasury_ata)
|
||||
|
||||
# ---- deposits (US-032) ----
|
||||
|
||||
def list_new_deposits(self, is_seen, *, limit: int = 200) -> list[Deposit]:
|
||||
"""Confirmed incoming USDT transfers whose signature is not yet seen.
|
||||
|
||||
``is_seen(signature) -> bool`` lets the caller (the deposit watcher)
|
||||
dedupe against its ledger, so replayed and re-observed transactions
|
||||
are credited exactly once.
|
||||
"""
|
||||
infos = self.rpc.call(
|
||||
"getSignaturesForAddress",
|
||||
[str(self._treasury_ata), {"limit": limit}],
|
||||
) or []
|
||||
deposits: list[Deposit] = []
|
||||
for info in infos:
|
||||
signature = info.get("signature", "")
|
||||
if not signature or info.get("err") is not None or is_seen(signature):
|
||||
continue
|
||||
deposit = self._parse_deposit(signature)
|
||||
if deposit is not None:
|
||||
deposits.append(deposit)
|
||||
return deposits
|
||||
|
||||
def _parse_deposit(self, signature: str) -> Deposit | None:
|
||||
result = self.rpc.call("getTransaction", [
|
||||
signature,
|
||||
{"encoding": "jsonParsed", "maxSupportedTransactionVersion": 0},
|
||||
])
|
||||
meta = (result or {}).get("meta")
|
||||
if not meta:
|
||||
return None
|
||||
pre = {b["accountIndex"]: b for b in meta.get("preTokenBalances") or []}
|
||||
post = {b["accountIndex"]: b for b in meta.get("postTokenBalances") or []}
|
||||
|
||||
treasury_delta = 0
|
||||
sender_wallet: str | None = None
|
||||
for index, balance in post.items():
|
||||
if balance.get("mint") != str(self._mint):
|
||||
continue
|
||||
before = pre.get(index)
|
||||
before_amount = int(before["uiTokenAmount"]["amount"]) if before else 0
|
||||
delta = int(balance["uiTokenAmount"]["amount"]) - before_amount
|
||||
owner = balance.get("owner")
|
||||
if owner == self.treasury_wallet and delta > 0:
|
||||
treasury_delta = delta
|
||||
elif delta < 0 and owner:
|
||||
sender_wallet = owner
|
||||
if sender_wallet is None:
|
||||
# a token account emptied and closed may appear only in pre balances
|
||||
for index, balance in pre.items():
|
||||
if balance.get("mint") != str(self._mint):
|
||||
continue
|
||||
if index not in post and balance.get("owner"):
|
||||
sender_wallet = balance["owner"]
|
||||
if treasury_delta <= 0 or sender_wallet is None:
|
||||
return None
|
||||
return Deposit(
|
||||
signature=signature,
|
||||
sender_wallet=sender_wallet,
|
||||
amount_usdt=treasury_delta / (10 ** self._decimals),
|
||||
)
|
||||
|
||||
# ---- payouts (US-033) ----
|
||||
|
||||
def send_payouts(self, payouts: list[tuple[str, float]]) -> str:
|
||||
"""Send one batched transaction of USDT transfers treasury → wallets.
|
||||
|
||||
Creates the recipient's associated token account when missing (fee
|
||||
paid by the treasury). Returns the transaction signature.
|
||||
"""
|
||||
if not payouts:
|
||||
raise ValueError("payouts must be non-empty")
|
||||
instructions = []
|
||||
for wallet, amount in payouts:
|
||||
recipient = Pubkey.from_string(wallet)
|
||||
recipient_ata = get_associated_token_address(recipient, self._mint)
|
||||
if not self._account_exists(recipient_ata):
|
||||
instructions.append(create_associated_token_account(
|
||||
payer=self._treasury.pubkey(),
|
||||
owner=recipient,
|
||||
mint=self._mint,
|
||||
))
|
||||
instructions.append(transfer_checked(TransferCheckedParams(
|
||||
program_id=TOKEN_PROGRAM_ID,
|
||||
source=self._treasury_ata,
|
||||
mint=self._mint,
|
||||
dest=recipient_ata,
|
||||
owner=self._treasury.pubkey(),
|
||||
amount=int(round(amount * (10 ** self._decimals))),
|
||||
decimals=self._decimals,
|
||||
)))
|
||||
return self.send_instructions(instructions)
|
||||
|
||||
# ---- shared plumbing + devnet helpers (scripts/devnet_setup.py) ----
|
||||
|
||||
def _account_exists(self, pubkey: Pubkey) -> bool:
|
||||
result = self.rpc.call("getAccountInfo", [str(pubkey), {"encoding": "base64"}])
|
||||
return bool(result and result.get("value"))
|
||||
|
||||
def send_instructions(self, instructions: list, extra_signers: list | None = None) -> str:
|
||||
blockhash_result = self.rpc.call("getLatestBlockhash")
|
||||
blockhash = Hash.from_string(blockhash_result["value"]["blockhash"])
|
||||
transaction = Transaction.new_signed_with_payer(
|
||||
instructions,
|
||||
self._treasury.pubkey(),
|
||||
[self._treasury, *(extra_signers or [])],
|
||||
blockhash,
|
||||
)
|
||||
encoded = base64.b64encode(bytes(transaction)).decode()
|
||||
return self.rpc.call("sendTransaction", [
|
||||
encoded,
|
||||
{"encoding": "base64", "preflightCommitment": "confirmed"},
|
||||
])
|
||||
|
||||
def get_sol_balance(self, wallet: str | None = None) -> float:
|
||||
result = self.rpc.call("getBalance", [wallet or self.treasury_wallet])
|
||||
return result["value"] / 1e9
|
||||
|
||||
def request_airdrop(self, sol: float = 2.0) -> str:
|
||||
return self.rpc.call(
|
||||
"requestAirdrop", [self.treasury_wallet, int(sol * 1e9)]
|
||||
)
|
||||
|
||||
def create_mock_usdt_mint(self) -> tuple["SolanaCustodialTreasury", str]:
|
||||
"""Create a fresh 6-decimal mock-USDT mint (treasury = mint authority).
|
||||
|
||||
Returns a new adapter bound to the created mint plus its address.
|
||||
"""
|
||||
mint_keypair = Keypair()
|
||||
rent = self.rpc.call(
|
||||
"getMinimumBalanceForRentExemption", [_MINT_ACCOUNT_SIZE]
|
||||
)
|
||||
instructions = [
|
||||
create_account(CreateAccountParams(
|
||||
from_pubkey=self._treasury.pubkey(),
|
||||
to_pubkey=mint_keypair.pubkey(),
|
||||
lamports=rent,
|
||||
space=_MINT_ACCOUNT_SIZE,
|
||||
owner=TOKEN_PROGRAM_ID,
|
||||
)),
|
||||
initialize_mint(InitializeMintParams(
|
||||
program_id=TOKEN_PROGRAM_ID,
|
||||
mint=mint_keypair.pubkey(),
|
||||
decimals=self._decimals,
|
||||
mint_authority=self._treasury.pubkey(),
|
||||
freeze_authority=None,
|
||||
)),
|
||||
]
|
||||
self.send_instructions(instructions, extra_signers=[mint_keypair])
|
||||
fresh = SolanaCustodialTreasury(
|
||||
self.rpc.url,
|
||||
str(mint_keypair.pubkey()),
|
||||
self._treasury,
|
||||
decimals=self._decimals,
|
||||
)
|
||||
return fresh, str(mint_keypair.pubkey())
|
||||
|
||||
def ensure_token_account(self, wallet: str) -> str:
|
||||
owner = Pubkey.from_string(wallet)
|
||||
ata = get_associated_token_address(owner, self._mint)
|
||||
if not self._account_exists(ata):
|
||||
self.send_instructions([create_associated_token_account(
|
||||
payer=self._treasury.pubkey(), owner=owner, mint=self._mint,
|
||||
)])
|
||||
return str(ata)
|
||||
|
||||
def mint_mock_usdt(self, wallet: str, amount: float) -> str:
|
||||
"""Mint mock USDT to a wallet (devnet only — treasury is mint authority)."""
|
||||
ata = self.ensure_token_account(wallet)
|
||||
return self.send_instructions([mint_to(MintToParams(
|
||||
program_id=TOKEN_PROGRAM_ID,
|
||||
mint=self._mint,
|
||||
dest=Pubkey.from_string(ata),
|
||||
mint_authority=self._treasury.pubkey(),
|
||||
amount=int(round(amount * (10 ** self._decimals))),
|
||||
))])
|
||||
@@ -7,6 +7,10 @@ name = "meshnet-contracts"
|
||||
version = "0.1.0"
|
||||
description = "Distributed Inference Network Solana smart contract wrappers"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
"solana>=0.36",
|
||||
"solders>=0.23",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["."]
|
||||
|
||||
@@ -159,7 +159,9 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
|
||||
"shard_coverage_percentage": float(model.get("shard_coverage_percentage", 0.0)),
|
||||
}
|
||||
for model in models_resp.get("data", [])
|
||||
if isinstance(model, dict) and isinstance(model.get("id"), str)
|
||||
if isinstance(model, dict)
|
||||
and isinstance(model.get("id"), str)
|
||||
and float(model.get("shard_coverage_percentage", 0.0)) > 0.0
|
||||
]
|
||||
self._send_json(200, {"data": data})
|
||||
return
|
||||
@@ -244,7 +246,7 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
|
||||
|
||||
def _handle_chat_completions(self):
|
||||
server: _GatewayHTTPServer = self.server # type: ignore[assignment]
|
||||
# Read raw bytes first so we can proxy them if tracker-nodes are available
|
||||
# Read raw bytes first so we can proxy them if head workers are available.
|
||||
length = int(self.headers.get("Content-Length", 0))
|
||||
raw_body = self.rfile.read(length)
|
||||
try:
|
||||
@@ -257,12 +259,11 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
|
||||
return
|
||||
|
||||
model = str(body.get("model", "stub-model"))
|
||||
tracker_nodes = _get_tracker_nodes(server, model)
|
||||
if tracker_nodes:
|
||||
# Proxy to a tracker-node (round-robin by request count)
|
||||
target = tracker_nodes[server.request_count % len(tracker_nodes)]
|
||||
head_workers = _get_head_workers(server, model)
|
||||
if head_workers:
|
||||
target = head_workers[server.request_count % len(head_workers)]
|
||||
server.request_count += 1
|
||||
return self._proxy_to_tracker_node(target, raw_body)
|
||||
return self._proxy_to_head_worker(target, raw_body)
|
||||
|
||||
# Fallback: use existing direct pipeline (backward compat)
|
||||
streaming = bool(body.get("stream", False))
|
||||
@@ -284,8 +285,8 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
|
||||
else:
|
||||
self._send_json(200, completion)
|
||||
|
||||
def _proxy_to_tracker_node(self, url: str, body_bytes: bytes) -> None:
|
||||
"""Forward a raw request body to a tracker-node and stream the response back."""
|
||||
def _proxy_to_head_worker(self, url: str, body_bytes: bytes) -> None:
|
||||
"""Forward a raw request body to a head worker and stream the response back."""
|
||||
target_url = f"{url}/v1/chat/completions"
|
||||
req = urllib.request.Request(
|
||||
target_url,
|
||||
@@ -307,7 +308,7 @@ class _GatewayHandler(http.server.BaseHTTPRequestHandler):
|
||||
self.wfile.write(resp_body)
|
||||
return
|
||||
except Exception as exc:
|
||||
self._send_json_error(503, f"tracker-node unreachable: {exc}")
|
||||
self._send_json_error(503, f"head worker unreachable: {exc}")
|
||||
return
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", content_type)
|
||||
@@ -816,13 +817,13 @@ def _get_json(url: str, timeout: float = 5.0) -> dict:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
def _get_tracker_nodes(server: _GatewayHTTPServer, model: str) -> list[str]:
|
||||
"""Return tracker-node endpoint URLs for this model, or empty list on any failure."""
|
||||
def _get_head_workers(server: _GatewayHTTPServer, model: str) -> list[str]:
|
||||
"""Return head-worker endpoint URLs for this model, or empty list on failure."""
|
||||
if server.tracker_url is None:
|
||||
return []
|
||||
try:
|
||||
resp = _get_json(f"{server.tracker_url}/v1/tracker-nodes/{urllib.parse.quote(model)}")
|
||||
return [n["endpoint"] for n in resp.get("tracker_nodes", [])]
|
||||
resp = _get_json(f"{server.tracker_url}/v1/head-workers/{urllib.parse.quote(model)}")
|
||||
return [n["endpoint"] for n in resp.get("head_workers", [])]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
@@ -45,7 +45,12 @@ class BillingLedger:
|
||||
self._lock = threading.Lock()
|
||||
self._client_balances: dict[str, float] = {}
|
||||
self._node_pending: dict[str, float] = {}
|
||||
self._wallet_bindings: dict[str, str] = {} # client wallet pubkey -> api_key
|
||||
self._protocol_cut: float = 0.0
|
||||
self._pending_since: dict[str, float] = {} # wallet -> ts pending became > 0
|
||||
self._last_payout_ts: dict[str, float] = {}
|
||||
# settlement id -> {"payouts": [(wallet, amount)], "signature": str|None, "ts": float}
|
||||
self._settlements: dict[str, dict] = {}
|
||||
self._seen_event_ids: set[str] = set()
|
||||
self._event_log: list[dict] = [] # full log, order of local application
|
||||
self._dirty = False
|
||||
@@ -144,6 +149,117 @@ class BillingLedger:
|
||||
self._apply_locked(event)
|
||||
return event
|
||||
|
||||
def bind_wallet(self, api_key: str, wallet: str) -> dict:
|
||||
"""Bind a client wallet pubkey to an API key (US-032 deposits)."""
|
||||
with self._lock:
|
||||
event = {
|
||||
"id": f"bind-{uuid.uuid4().hex}",
|
||||
"type": "bind",
|
||||
"api_key": api_key,
|
||||
"wallet": wallet,
|
||||
"ts": time.time(),
|
||||
}
|
||||
self._apply_locked(event)
|
||||
return event
|
||||
|
||||
def api_key_for_wallet(self, wallet: str) -> str | None:
|
||||
with self._lock:
|
||||
return self._wallet_bindings.get(wallet)
|
||||
|
||||
def credit_deposit(self, api_key: str, amount: float, signature: str) -> dict | None:
|
||||
"""Credit an on-chain deposit exactly once.
|
||||
|
||||
The event id embeds the transaction signature, so replayed or
|
||||
re-observed transfers — locally or via hive gossip — are no-ops.
|
||||
Returns None when the signature was already credited.
|
||||
"""
|
||||
if amount <= 0:
|
||||
raise ValueError("deposit amount must be positive")
|
||||
event_id = f"deposit-{signature}"
|
||||
with self._lock:
|
||||
if event_id in self._seen_event_ids:
|
||||
return None
|
||||
event = {
|
||||
"id": event_id,
|
||||
"type": "credit",
|
||||
"api_key": api_key,
|
||||
"amount": amount,
|
||||
"signature": signature,
|
||||
"ts": time.time(),
|
||||
"note": "onchain-deposit",
|
||||
}
|
||||
self._apply_locked(event)
|
||||
return event
|
||||
|
||||
def has_event(self, event_id: str) -> bool:
|
||||
with self._lock:
|
||||
return event_id in self._seen_event_ids
|
||||
|
||||
# ---- settlement (US-033) ----
|
||||
|
||||
def payables(
|
||||
self,
|
||||
*,
|
||||
threshold: float,
|
||||
max_period: float,
|
||||
dust_floor: float,
|
||||
now: float | None = None,
|
||||
exclude: set[str] | None = None,
|
||||
) -> list[tuple[str, float]]:
|
||||
"""Wallets due a payout: pending ≥ threshold OR pending age ≥ max_period,
|
||||
never below the dust floor (mining-pool standard, ADR-0015)."""
|
||||
now = now if now is not None else time.time()
|
||||
exclude = exclude or set()
|
||||
due: list[tuple[str, float]] = []
|
||||
with self._lock:
|
||||
for wallet, pending in self._node_pending.items():
|
||||
if wallet in exclude or pending < max(dust_floor, 1e-9):
|
||||
continue
|
||||
if pending >= threshold:
|
||||
due.append((wallet, pending))
|
||||
continue
|
||||
since = self._pending_since.get(wallet)
|
||||
if since is not None and now - since >= max_period:
|
||||
due.append((wallet, pending))
|
||||
return due
|
||||
|
||||
def confirm_settlement(self, settlement_id: str, signature: str) -> dict:
|
||||
"""Record the on-chain transaction signature for a settlement batch."""
|
||||
with self._lock:
|
||||
event = {
|
||||
"id": f"settlement-{settlement_id}",
|
||||
"type": "settlement",
|
||||
"settlement_id": settlement_id,
|
||||
"signature": signature,
|
||||
"ts": time.time(),
|
||||
}
|
||||
self._apply_locked(event)
|
||||
return event
|
||||
|
||||
def unconfirmed_settlements(self) -> dict[str, list[tuple[str, float]]]:
|
||||
"""Settlement batches whose pending was debited but whose transaction
|
||||
has not been confirmed — resend these (idempotent by settlement id)."""
|
||||
with self._lock:
|
||||
return {
|
||||
sid: list(s["payouts"])
|
||||
for sid, s in self._settlements.items()
|
||||
if s["signature"] is None and s["payouts"]
|
||||
}
|
||||
|
||||
def settlement_history(self) -> list[dict]:
|
||||
with self._lock:
|
||||
return [
|
||||
{
|
||||
"settlement_id": sid,
|
||||
"signature": s["signature"],
|
||||
"payouts": [
|
||||
{"wallet": w, "amount": a} for w, a in s["payouts"]
|
||||
],
|
||||
"ts": s["ts"],
|
||||
}
|
||||
for sid, s in sorted(self._settlements.items(), key=lambda kv: kv[1]["ts"])
|
||||
]
|
||||
|
||||
def settle_node_payout(self, wallet: str, amount: float, *, reference: str = "") -> dict:
|
||||
"""Deduct a paid-out amount from a node's pending balance (US-033 hook)."""
|
||||
if amount <= 0:
|
||||
@@ -203,15 +319,35 @@ class BillingLedger:
|
||||
self._client_balances[key] = self._client_balances.get(key, 0.0) - float(event["cost"])
|
||||
for wallet, amount in (event.get("shares") or {}).items():
|
||||
self._node_pending[wallet] = self._node_pending.get(wallet, 0.0) + float(amount)
|
||||
if amount > 0:
|
||||
self._pending_since.setdefault(wallet, float(event.get("ts", time.time())))
|
||||
self._protocol_cut += float(event.get("protocol_amount", 0.0))
|
||||
elif etype == "payout":
|
||||
wallet = event["wallet"]
|
||||
self._node_pending[wallet] = self._node_pending.get(wallet, 0.0) - float(event["amount"])
|
||||
amount = float(event["amount"])
|
||||
ts = float(event.get("ts", time.time()))
|
||||
self._node_pending[wallet] = self._node_pending.get(wallet, 0.0) - amount
|
||||
self._pending_since.pop(wallet, None)
|
||||
self._last_payout_ts[wallet] = ts
|
||||
reference = event.get("reference") or ""
|
||||
if reference:
|
||||
settlement = self._settlements.setdefault(
|
||||
reference, {"payouts": [], "signature": None, "ts": ts}
|
||||
)
|
||||
settlement["payouts"].append((wallet, amount))
|
||||
elif etype == "settlement":
|
||||
settlement = self._settlements.setdefault(
|
||||
event["settlement_id"],
|
||||
{"payouts": [], "signature": None, "ts": float(event.get("ts", 0.0))},
|
||||
)
|
||||
settlement["signature"] = event.get("signature")
|
||||
elif etype == "forfeit":
|
||||
wallet = event["wallet"]
|
||||
amount = float(event.get("amount", 0.0))
|
||||
self._node_pending[wallet] = self._node_pending.get(wallet, 0.0) - amount
|
||||
self._protocol_cut += amount
|
||||
elif etype == "bind":
|
||||
self._wallet_bindings[event["wallet"]] = event["api_key"]
|
||||
else:
|
||||
return
|
||||
self._seen_event_ids.add(event["id"])
|
||||
@@ -233,11 +369,21 @@ class BillingLedger:
|
||||
return {
|
||||
"clients": dict(self._client_balances),
|
||||
"node_pending": dict(self._node_pending),
|
||||
"wallet_bindings": dict(self._wallet_bindings),
|
||||
"protocol_cut": self._protocol_cut,
|
||||
"prices": dict(self._prices),
|
||||
"default_price_per_1k_tokens": self._default_price_per_1k,
|
||||
"node_share": self._node_share,
|
||||
"event_count": len(self._event_log),
|
||||
"forfeits": [
|
||||
{
|
||||
"wallet": e["wallet"],
|
||||
"amount": e.get("amount", 0.0),
|
||||
"reason": e.get("reason", ""),
|
||||
"ts": e.get("ts", 0.0),
|
||||
}
|
||||
for e in self._event_log if e.get("type") == "forfeit"
|
||||
][-20:],
|
||||
}
|
||||
|
||||
# ---- persistence ----
|
||||
|
||||
@@ -1,85 +1,130 @@
|
||||
"""meshnet-tracker CLI entry point."""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
|
||||
from .server import TrackerServer, derive_relay_url_from_public_tracker_url
|
||||
|
||||
|
||||
def main() -> None:
|
||||
common = argparse.ArgumentParser(add_help=False)
|
||||
common.add_argument("--host", default="0.0.0.0", help="Host interface to listen on")
|
||||
common.add_argument("--port", type=int, default=8080, help="Port to listen on")
|
||||
common.add_argument(
|
||||
"--heartbeat-timeout",
|
||||
type=float,
|
||||
default=30.0,
|
||||
help="Seconds before a node is removed from the registry after missed heartbeat",
|
||||
)
|
||||
common.add_argument(
|
||||
"--cluster-peers",
|
||||
default="",
|
||||
help="Comma-separated URLs of peer tracker nodes (enables Raft cluster mode)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--self-url",
|
||||
default=None,
|
||||
help="This tracker's own URL as seen by peers (auto-derived from --host/--port if omitted)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--stats-db",
|
||||
default=None,
|
||||
metavar="PATH",
|
||||
help="SQLite database path for persistent model usage statistics",
|
||||
)
|
||||
common.add_argument(
|
||||
"--relay-url",
|
||||
default=None,
|
||||
help="Public ws(s):// relay URL advertised to nodes, for example wss://ai.neuron.d-popov.com/ws",
|
||||
)
|
||||
common.add_argument(
|
||||
"--billing-db",
|
||||
default=None,
|
||||
metavar="PATH",
|
||||
help="SQLite database path for the USDT billing ledger (enables billing; ADR-0015)",
|
||||
)
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="meshnet-tracker",
|
||||
description="Distributed Inference Network node registry and route selection",
|
||||
parents=[common],
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command")
|
||||
|
||||
subparsers.add_parser("start", help="Start the tracker server", parents=[common])
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command in {None, "start"}:
|
||||
cluster_peers = [u.strip() for u in args.cluster_peers.split(",") if u.strip()]
|
||||
relay_url = args.relay_url or derive_relay_url_from_public_tracker_url(args.self_url)
|
||||
server = TrackerServer(
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
heartbeat_timeout=args.heartbeat_timeout,
|
||||
cluster_peers=cluster_peers or None,
|
||||
cluster_self_url=args.self_url,
|
||||
stats_db=getattr(args, "stats_db", None),
|
||||
relay_url=relay_url,
|
||||
billing_db=getattr(args, "billing_db", None),
|
||||
)
|
||||
port = server.start()
|
||||
print(f"meshnet-tracker listening on http://{args.host}:{port}", flush=True)
|
||||
try:
|
||||
while True:
|
||||
time.sleep(1)
|
||||
except KeyboardInterrupt:
|
||||
server.stop()
|
||||
sys.exit(0)
|
||||
else:
|
||||
parser.print_help()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
"""meshnet-tracker CLI entry point."""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
|
||||
from .server import TrackerServer, derive_relay_url_from_public_tracker_url
|
||||
|
||||
|
||||
def main() -> None:
|
||||
common = argparse.ArgumentParser(add_help=False)
|
||||
common.add_argument("--host", default="0.0.0.0", help="Host interface to listen on")
|
||||
common.add_argument("--port", type=int, default=8080, help="Port to listen on")
|
||||
common.add_argument(
|
||||
"--heartbeat-timeout",
|
||||
type=float,
|
||||
default=30.0,
|
||||
help="Seconds before a node is removed from the registry after missed heartbeat",
|
||||
)
|
||||
common.add_argument(
|
||||
"--cluster-peers",
|
||||
default="",
|
||||
help="Comma-separated URLs of peer tracker nodes (enables Raft cluster mode)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--self-url",
|
||||
default=None,
|
||||
help="This tracker's own URL as seen by peers (auto-derived from --host/--port if omitted)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--stats-db",
|
||||
default=None,
|
||||
metavar="PATH",
|
||||
help="SQLite database path for persistent model usage statistics",
|
||||
)
|
||||
common.add_argument(
|
||||
"--relay-url",
|
||||
default=None,
|
||||
help="Public ws(s):// relay URL advertised to nodes, for example wss://ai.neuron.d-popov.com/ws",
|
||||
)
|
||||
common.add_argument(
|
||||
"--billing-db",
|
||||
default=None,
|
||||
metavar="PATH",
|
||||
help="SQLite database path for the USDT billing ledger (enables billing; ADR-0015)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--solana-rpc-url",
|
||||
default=None,
|
||||
help="Solana RPC URL (e.g. https://api.devnet.solana.com); enables the on-chain treasury",
|
||||
)
|
||||
common.add_argument(
|
||||
"--usdt-mint",
|
||||
default=None,
|
||||
help="SPL mint address of (mock) USDT — see scripts/devnet_setup.py",
|
||||
)
|
||||
common.add_argument(
|
||||
"--treasury-keypair",
|
||||
default=None,
|
||||
metavar="PATH",
|
||||
help="Treasury keypair JSON path (only on settlement-capable trackers)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--settle-period",
|
||||
type=float,
|
||||
default=86400.0,
|
||||
help="Max seconds between payouts to a node (dev: 60, prod: 86400)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--payout-threshold",
|
||||
type=float,
|
||||
default=5.0,
|
||||
help="Pending USDT that triggers an immediate payout (dev: 0)",
|
||||
)
|
||||
common.add_argument(
|
||||
"--payout-dust-floor",
|
||||
type=float,
|
||||
default=0.01,
|
||||
help="Never pay out less than this many USDT",
|
||||
)
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="meshnet-tracker",
|
||||
description="Distributed Inference Network node registry and route selection",
|
||||
parents=[common],
|
||||
)
|
||||
subparsers = parser.add_subparsers(dest="command")
|
||||
|
||||
subparsers.add_parser("start", help="Start the tracker server", parents=[common])
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command in {None, "start"}:
|
||||
cluster_peers = [u.strip() for u in args.cluster_peers.split(",") if u.strip()]
|
||||
relay_url = args.relay_url or derive_relay_url_from_public_tracker_url(args.self_url)
|
||||
treasury = None
|
||||
if args.solana_rpc_url and args.usdt_mint and args.treasury_keypair:
|
||||
from meshnet_contracts.solana_adapter import SolanaCustodialTreasury
|
||||
|
||||
treasury = SolanaCustodialTreasury(
|
||||
args.solana_rpc_url, args.usdt_mint, args.treasury_keypair,
|
||||
)
|
||||
server = TrackerServer(
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
heartbeat_timeout=args.heartbeat_timeout,
|
||||
cluster_peers=cluster_peers or None,
|
||||
cluster_self_url=args.self_url,
|
||||
stats_db=getattr(args, "stats_db", None),
|
||||
relay_url=relay_url,
|
||||
billing_db=getattr(args, "billing_db", None),
|
||||
treasury=treasury,
|
||||
settle_period=args.settle_period,
|
||||
payout_threshold=args.payout_threshold,
|
||||
payout_dust_floor=args.payout_dust_floor,
|
||||
)
|
||||
port = server.start()
|
||||
print(f"meshnet-tracker listening on http://{args.host}:{port}", flush=True)
|
||||
try:
|
||||
while True:
|
||||
time.sleep(1)
|
||||
except KeyboardInterrupt:
|
||||
server.stop()
|
||||
sys.exit(0)
|
||||
else:
|
||||
parser.print_help()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
197
packages/tracker/meshnet_tracker/dashboard.html
Normal file
197
packages/tracker/meshnet_tracker/dashboard.html
Normal file
@@ -0,0 +1,197 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>meshnet tracker</title>
|
||||
<style>
|
||||
:root { --bg:#0d1117; --panel:#161b22; --border:#30363d; --fg:#c9d1d9;
|
||||
--dim:#8b949e; --accent:#58a6ff; --ok:#3fb950; --bad:#f85149; --warn:#d29922; }
|
||||
* { box-sizing:border-box; }
|
||||
body { margin:0; background:var(--bg); color:var(--fg);
|
||||
font:13px/1.5 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace; }
|
||||
header { display:flex; align-items:baseline; gap:14px; padding:14px 20px;
|
||||
border-bottom:1px solid var(--border); }
|
||||
header h1 { font-size:16px; margin:0; color:var(--accent); }
|
||||
header .meta { color:var(--dim); font-size:12px; }
|
||||
main { display:grid; grid-template-columns:repeat(auto-fit,minmax(340px,1fr));
|
||||
gap:14px; padding:14px 20px; }
|
||||
section { background:var(--panel); border:1px solid var(--border);
|
||||
border-radius:8px; padding:12px 14px; min-height:80px; }
|
||||
section h2 { margin:0 0 8px; font-size:12px; text-transform:uppercase;
|
||||
letter-spacing:.08em; color:var(--dim); }
|
||||
table { width:100%; border-collapse:collapse; font-size:12px; }
|
||||
th { text-align:left; color:var(--dim); font-weight:normal;
|
||||
border-bottom:1px solid var(--border); padding:2px 6px 4px 0; }
|
||||
td { padding:3px 6px 3px 0; border-bottom:1px solid #21262d; }
|
||||
.ok { color:var(--ok); } .bad { color:var(--bad); } .warn { color:var(--warn); }
|
||||
.dim { color:var(--dim); } .num { text-align:right; }
|
||||
a { color:var(--accent); text-decoration:none; }
|
||||
.empty { color:var(--dim); font-style:italic; }
|
||||
.pill { display:inline-block; padding:0 7px; border-radius:9px;
|
||||
border:1px solid var(--border); font-size:11px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>meshnet tracker</h1>
|
||||
<span class="meta" id="self-url"></span>
|
||||
<span class="meta" id="refreshed"></span>
|
||||
</header>
|
||||
<main>
|
||||
<section><h2>Tracker hive</h2><div id="hive" class="empty">loading…</div></section>
|
||||
<section><h2>Nodes & coverage</h2><div id="nodes" class="empty">loading…</div></section>
|
||||
<section><h2>Client balances</h2><div id="clients" class="empty">loading…</div></section>
|
||||
<section><h2>Node pending payouts</h2><div id="pending" class="empty">loading…</div></section>
|
||||
<section><h2>Settlement history</h2><div id="settlements" class="empty">loading…</div></section>
|
||||
<section><h2>Strikes / bans / forfeitures</h2><div id="fraud" class="empty">loading…</div></section>
|
||||
<section><h2>Model usage (RPM)</h2><div id="stats" class="empty">loading…</div></section>
|
||||
</main>
|
||||
<script>
|
||||
"use strict";
|
||||
const $ = id => document.getElementById(id);
|
||||
const esc = s => String(s).replace(/[&<>"]/g,
|
||||
c => ({"&":"&","<":"<",">":">",'"':"""}[c]));
|
||||
const usdt = v => (Math.round(v * 1e6) / 1e6).toFixed(6);
|
||||
const short = (s, n=14) => { s = String(s); return s.length > n ? s.slice(0, 6) + "…" + s.slice(-5) : s; };
|
||||
|
||||
async function fetchJson(path) {
|
||||
try {
|
||||
const r = await fetch(path);
|
||||
if (!r.ok) return null;
|
||||
return await r.json();
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function table(headers, rows) {
|
||||
if (!rows.length) return '<div class="empty">nothing yet</div>';
|
||||
return '<table><tr>' + headers.map(h => `<th>${esc(h)}</th>`).join("") + '</tr>'
|
||||
+ rows.map(r => '<tr>' + r.map(c => `<td>${c}</td>`).join("") + '</tr>').join("")
|
||||
+ '</table>';
|
||||
}
|
||||
|
||||
function renderHive(raft) {
|
||||
const role = raft && (raft.state || raft.role);
|
||||
if (!raft || role === "standalone") {
|
||||
$("hive").innerHTML = '<span class="pill">standalone</span> single tracker, no cluster';
|
||||
return;
|
||||
}
|
||||
const cls = role === "leader" ? "ok" : "warn";
|
||||
$("hive").innerHTML =
|
||||
`role <span class="${cls}">${esc(role || "?")}</span> · term ${esc(raft.term ?? "?")}` +
|
||||
`<br>leader: ${raft.leader ? esc(raft.leader) : '<span class="dim">unknown</span>'}` +
|
||||
(raft.peers ? `<br>peers: ${esc(Array.isArray(raft.peers) ? raft.peers.join(", ") : raft.peers)}` : "");
|
||||
}
|
||||
|
||||
function renderNodes(map) {
|
||||
const nodes = (map && map.nodes) || [];
|
||||
if (!nodes.length) {
|
||||
$("nodes").innerHTML = '<div class="empty">no nodes registered</div>'; return;
|
||||
}
|
||||
const byModel = {};
|
||||
for (const n of nodes) {
|
||||
const key = n.model || n.hf_repo || "?";
|
||||
(byModel[key] = byModel[key] || []).push(n);
|
||||
}
|
||||
let html = "";
|
||||
for (const [model, group] of Object.entries(byModel)) {
|
||||
html += `<div><b>${esc(model)}</b> <span class="dim">(${group.length} node${group.length===1?"":"s"})</span></div>`;
|
||||
html += table(["node", "shard", "mode", "health"], group.map(n => [
|
||||
esc(short(n.node_id || "?")),
|
||||
esc(`${n.shard_start ?? "?"}-${n.shard_end ?? "?"}`),
|
||||
n.tracker_mode ? '<span class="pill">tracker</span>' : '<span class="dim">worker</span>',
|
||||
(n.stats && (n.stats.alive === false || n.stats.healthy === false))
|
||||
? '<span class="bad">down</span>' : '<span class="ok">up</span>',
|
||||
]));
|
||||
}
|
||||
$("nodes").innerHTML = html;
|
||||
}
|
||||
|
||||
function renderBilling(summary) {
|
||||
if (!summary) {
|
||||
$("clients").innerHTML = '<div class="empty">billing disabled</div>';
|
||||
$("pending").innerHTML = '<div class="empty">billing disabled</div>';
|
||||
return summary;
|
||||
}
|
||||
const clients = Object.entries(summary.clients || {});
|
||||
$("clients").innerHTML = table(["api key", "balance (USDT)"],
|
||||
clients.map(([k, v]) => [esc(short(k)),
|
||||
`<span class="num ${v > 0 ? "ok" : "bad"}">${usdt(v)}</span>`]));
|
||||
const pending = Object.entries(summary.node_pending || {});
|
||||
const cut = summary.protocol_cut || 0;
|
||||
$("pending").innerHTML =
|
||||
table(["node wallet", "pending (USDT)"],
|
||||
pending.map(([w, v]) => [esc(short(w)), `<span class="num">${usdt(v)}</span>`]))
|
||||
+ `<div style="margin-top:6px" class="dim">protocol cut: <b>${usdt(cut)}</b> USDT</div>`;
|
||||
return summary;
|
||||
}
|
||||
|
||||
function renderSettlements(data) {
|
||||
if (!data || !data.settlements) {
|
||||
$("settlements").innerHTML = '<div class="empty">billing disabled</div>'; return;
|
||||
}
|
||||
const rows = data.settlements.slice(-15).reverse().map(s => {
|
||||
const total = (s.payouts || []).reduce((a, p) => a + p.amount, 0);
|
||||
const sig = s.signature;
|
||||
const link = sig && !sig.startsWith("fake-") && !sig.startsWith("local-")
|
||||
? `<a target="_blank" href="https://explorer.solana.com/tx/${encodeURIComponent(sig)}?cluster=devnet">${esc(short(sig))}</a>`
|
||||
: (sig ? esc(short(sig)) : '<span class="warn">unconfirmed</span>');
|
||||
return [new Date(s.ts * 1000).toLocaleTimeString(), String((s.payouts || []).length),
|
||||
`<span class="num">${usdt(total)}</span>`, link];
|
||||
});
|
||||
$("settlements").innerHTML = table(["time", "payouts", "total (USDT)", "tx"], rows);
|
||||
}
|
||||
|
||||
function renderFraud(wallets, summary) {
|
||||
const rows = Object.entries((wallets && wallets.wallets) || {})
|
||||
.filter(([, w]) => w.strike_count > 0 || w.banned)
|
||||
.map(([addr, w]) => [esc(short(addr)), String(w.strike_count),
|
||||
w.banned ? '<span class="bad">banned</span>' : '<span class="ok">active</span>']);
|
||||
let html = rows.length ? table(["wallet", "strikes", "status"], rows)
|
||||
: '<div class="empty">no strikes recorded</div>';
|
||||
const forfeits = (summary && summary.forfeits) || [];
|
||||
if (forfeits.length) {
|
||||
html += '<div style="margin-top:6px"><b class="dim">recent forfeitures</b></div>'
|
||||
+ table(["wallet", "amount", "reason"], forfeits.slice(-8).reverse().map(f =>
|
||||
[esc(short(f.wallet)), `<span class="num bad">-${usdt(f.amount)}</span>`, esc(f.reason)]));
|
||||
}
|
||||
$("fraud").innerHTML = html;
|
||||
}
|
||||
|
||||
function renderStats(stats) {
|
||||
const models = (stats && (stats.models || stats.stats)) || stats;
|
||||
if (!models || typeof models !== "object" || !Object.keys(models).length) {
|
||||
$("stats").innerHTML = '<div class="empty">no requests yet</div>'; return;
|
||||
}
|
||||
const rows = Object.entries(models).map(([m, s]) => [
|
||||
esc(m),
|
||||
esc(String(s.rpm_last_hour ?? "?")),
|
||||
esc(String(s.rpm_last_day ?? "?")),
|
||||
esc(String(s.rpm_last_month ?? "?")),
|
||||
]);
|
||||
$("stats").innerHTML = table(["model", "rpm (1h)", "rpm (24h)", "rpm (30d)"], rows);
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
$("self-url").textContent = location.host;
|
||||
const [raft, map, summary, settlements, wallets, stats] = await Promise.all([
|
||||
fetchJson("/v1/raft/status"),
|
||||
fetchJson("/v1/network/map"),
|
||||
fetchJson("/v1/billing/summary"),
|
||||
fetchJson("/v1/billing/settlements"),
|
||||
fetchJson("/v1/registry/wallets"),
|
||||
fetchJson("/v1/stats"),
|
||||
]);
|
||||
renderHive(raft);
|
||||
renderNodes(map);
|
||||
renderBilling(summary);
|
||||
renderSettlements(settlements);
|
||||
renderFraud(wallets, summary);
|
||||
renderStats(stats);
|
||||
$("refreshed").textContent = "refreshed " + new Date().toLocaleTimeString();
|
||||
}
|
||||
refresh();
|
||||
setInterval(refresh, 4000);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -23,7 +23,9 @@ HTTP API contract:
|
||||
|
||||
import http.server
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
import socketserver
|
||||
import sqlite3
|
||||
import threading
|
||||
@@ -718,6 +720,30 @@ def _relay_http_request(
|
||||
return None
|
||||
|
||||
|
||||
def _find_pinned_route(
|
||||
nodes: list[_NodeEntry],
|
||||
required_start: int,
|
||||
required_end: int,
|
||||
hop_count: int,
|
||||
) -> list[_NodeEntry] | None:
|
||||
"""First combination of exactly ``hop_count`` distinct nodes covering the
|
||||
layer range, where every node extends coverage (US-030 benchmark routes)."""
|
||||
for combo in itertools.permutations(nodes, hop_count):
|
||||
covered = required_start - 1
|
||||
valid = True
|
||||
for candidate in combo:
|
||||
if candidate.shard_start is None or candidate.shard_end is None:
|
||||
valid = False
|
||||
break
|
||||
if candidate.shard_start > covered + 1 or candidate.shard_end <= covered:
|
||||
valid = False
|
||||
break
|
||||
covered = candidate.shard_end
|
||||
if valid and covered >= required_end:
|
||||
return list(combo)
|
||||
return None
|
||||
|
||||
|
||||
def _nodes_and_bounds_for_model(
|
||||
server: "_TrackerHTTPServer",
|
||||
model: str,
|
||||
@@ -1046,6 +1072,7 @@ class _TrackerHTTPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
|
||||
gossip: "NodeGossip | None" = None,
|
||||
stats: "_StatsCollector | None" = None,
|
||||
billing: "BillingLedger | None" = None,
|
||||
benchmark_results_path: str | None = None,
|
||||
) -> None:
|
||||
super().__init__(addr, handler)
|
||||
self.registry = registry
|
||||
@@ -1059,6 +1086,10 @@ class _TrackerHTTPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
|
||||
self.gossip = gossip
|
||||
self.stats: _StatsCollector | None = stats
|
||||
self.billing: BillingLedger | None = billing
|
||||
self.benchmark_results_path = benchmark_results_path or os.path.join(
|
||||
os.getcwd(), "benchmark_results.json"
|
||||
)
|
||||
self.benchmark_lock = threading.Lock()
|
||||
|
||||
|
||||
class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
@@ -1114,6 +1145,15 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
if self.path == "/v1/billing/gossip":
|
||||
self._handle_billing_gossip()
|
||||
return
|
||||
if self.path == "/v1/billing/forfeit":
|
||||
self._handle_billing_forfeit()
|
||||
return
|
||||
if self.path == "/v1/benchmark/hop-penalty":
|
||||
self._handle_benchmark_hop_penalty()
|
||||
return
|
||||
if self.path == "/v1/wallet/register":
|
||||
self._handle_wallet_register()
|
||||
return
|
||||
parts = self.path.split("/")
|
||||
# /v1/nodes/<node_id>/heartbeat -> ['', 'v1', 'nodes', '<id>', 'heartbeat']
|
||||
if len(parts) == 5 and parts[1] == "v1" and parts[2] == "nodes" and parts[4] == "heartbeat":
|
||||
@@ -1142,12 +1182,23 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
elif parsed.path.startswith("/v1/tracker-nodes/"):
|
||||
model = urllib.parse.unquote(parsed.path.removeprefix("/v1/tracker-nodes/"))
|
||||
self._handle_tracker_nodes(model)
|
||||
elif parsed.path.startswith("/v1/head-workers/"):
|
||||
model = urllib.parse.unquote(parsed.path.removeprefix("/v1/head-workers/"))
|
||||
self._handle_tracker_nodes(model)
|
||||
elif parsed.path == "/v1/raft/status":
|
||||
self._handle_raft_status()
|
||||
elif parsed.path == "/v1/stats":
|
||||
self._handle_stats()
|
||||
elif parsed.path == "/v1/billing/summary":
|
||||
self._handle_billing_summary()
|
||||
elif parsed.path == "/v1/billing/settlements":
|
||||
self._handle_billing_settlements()
|
||||
elif parsed.path == "/v1/benchmark/results":
|
||||
self._handle_benchmark_results()
|
||||
elif parsed.path == "/v1/registry/wallets":
|
||||
self._handle_registry_wallets()
|
||||
elif parsed.path in ("/dashboard", "/dashboard/"):
|
||||
self._handle_dashboard()
|
||||
elif parsed.path == "/v1/health":
|
||||
self._send_json(200, {"status": "ok"})
|
||||
else:
|
||||
@@ -1263,7 +1314,11 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
self._send_json(200, {"model": model, "coverage": coverage})
|
||||
|
||||
def _handle_tracker_nodes(self, model: str):
|
||||
"""Return nodes registered with tracker_mode=True whose shard starts at layer 0."""
|
||||
"""Return head workers: worker nodes that can start inference for a model.
|
||||
|
||||
The historical endpoint name is /v1/tracker-nodes, but these are not
|
||||
tracker processes and they are the only machines that load model shards.
|
||||
"""
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
resolved_name, preset = _resolve_model_preset(server.model_presets, model)
|
||||
if preset is None:
|
||||
@@ -1289,6 +1344,15 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
]
|
||||
self._send_json(200, {
|
||||
"model": resolved_name,
|
||||
"head_workers": [
|
||||
{
|
||||
"node_id": node.node_id,
|
||||
"endpoint": node.endpoint,
|
||||
"relay_addr": node.relay_addr,
|
||||
"peer_id": node.peer_id,
|
||||
}
|
||||
for node in tracker_nodes
|
||||
],
|
||||
"tracker_nodes": [
|
||||
{
|
||||
"node_id": node.node_id,
|
||||
@@ -1394,32 +1458,63 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
}})
|
||||
return
|
||||
|
||||
# Find a live tracker-mode node for this model
|
||||
with server.lock:
|
||||
self._purge_expired_nodes()
|
||||
candidates = [
|
||||
n for n in server.registry.values()
|
||||
if n.tracker_mode and _node_matches_model(n, model)
|
||||
]
|
||||
|
||||
if not candidates:
|
||||
# Fall back: any node serving shard_start=0 for this model
|
||||
# US-030: optional pinned route — "route": [node_id, ...] uses those
|
||||
# nodes in order instead of auto-selection. Absent field: unchanged.
|
||||
pinned_ids = body.get("route")
|
||||
pinned_nodes: list[_NodeEntry] | None = None
|
||||
if pinned_ids is not None:
|
||||
if (
|
||||
not isinstance(pinned_ids, list)
|
||||
or not pinned_ids
|
||||
or not all(isinstance(nid, str) and nid for nid in pinned_ids)
|
||||
):
|
||||
self._send_json(400, {"error": {
|
||||
"message": "route must be a non-empty list of node id strings",
|
||||
"type": "invalid_request_error",
|
||||
"code": "invalid_route",
|
||||
}})
|
||||
return
|
||||
with server.lock:
|
||||
self._purge_expired_nodes()
|
||||
missing = [nid for nid in pinned_ids if nid not in server.registry]
|
||||
if missing:
|
||||
self._send_json(400, {"error": {
|
||||
"message": f"unknown node ids in route: {missing}",
|
||||
"type": "invalid_request_error",
|
||||
"code": "unknown_route_nodes",
|
||||
}})
|
||||
return
|
||||
pinned_nodes = [server.registry[nid] for nid in pinned_ids]
|
||||
|
||||
if pinned_nodes is not None:
|
||||
node = pinned_nodes[0]
|
||||
else:
|
||||
# Find a live tracker-mode node for this model
|
||||
with server.lock:
|
||||
self._purge_expired_nodes()
|
||||
candidates = [
|
||||
n for n in server.registry.values()
|
||||
if n.shard_start == 0 and _node_matches_model(n, model)
|
||||
if n.tracker_mode and _node_matches_model(n, model)
|
||||
]
|
||||
|
||||
if not candidates:
|
||||
self._send_json(503, {"error": {
|
||||
"message": f"no nodes available for model {model!r}",
|
||||
"type": "service_unavailable",
|
||||
"code": "model_not_available",
|
||||
}})
|
||||
return
|
||||
if not candidates:
|
||||
# Fall back: any node serving shard_start=0 for this model
|
||||
with server.lock:
|
||||
candidates = [
|
||||
n for n in server.registry.values()
|
||||
if n.shard_start == 0 and _node_matches_model(n, model)
|
||||
]
|
||||
|
||||
# Simple round-robin via list length modulo (stateless, good enough)
|
||||
node = candidates[int(time.time() * 1000) % len(candidates)]
|
||||
if not candidates:
|
||||
self._send_json(503, {"error": {
|
||||
"message": f"no nodes available for model {model!r}",
|
||||
"type": "service_unavailable",
|
||||
"code": "model_not_available",
|
||||
}})
|
||||
return
|
||||
|
||||
# Simple round-robin via list length modulo (stateless, good enough)
|
||||
node = candidates[int(time.time() * 1000) % len(candidates)]
|
||||
target_url = f"{node.endpoint}/v1/chat/completions"
|
||||
request_id = str(body.get("id") or f"req-{time.time_ns():x}")
|
||||
|
||||
@@ -1439,7 +1534,10 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
and n.shard_start is not None and n.num_layers is not None
|
||||
]
|
||||
rs, re = 0, (max((n.num_layers for n in all_nodes), default=1) - 1)
|
||||
route_nodes, _ = _select_route(all_nodes, rs, re)
|
||||
if pinned_nodes is not None:
|
||||
route_nodes = pinned_nodes
|
||||
else:
|
||||
route_nodes, _ = _select_route(all_nodes, rs, re)
|
||||
# Compute start_layer for each hop: each node begins where the previous ended + 1.
|
||||
# This allows overlapping shard registrations without double-computation.
|
||||
covered_up_to = rs - 1
|
||||
@@ -1616,6 +1714,19 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
if server.billing is None or api_key is None:
|
||||
return
|
||||
# Probationary period (issue 08): a wallet's first N jobs earn nothing —
|
||||
# its share stays in the protocol cut. Job counts live in the registry
|
||||
# contract, so this only applies when the tracker runs with contracts.
|
||||
if server.contracts is not None:
|
||||
adjusted: list[tuple[str | None, int]] = []
|
||||
for wallet, work in node_work:
|
||||
if wallet:
|
||||
in_probation = server.contracts.registry.probationary_jobs_remaining(wallet) > 0
|
||||
server.contracts.registry.record_completed_job(wallet)
|
||||
if in_probation:
|
||||
wallet = None
|
||||
adjusted.append((wallet, work))
|
||||
node_work = adjusted
|
||||
try:
|
||||
event = server.billing.charge_request(api_key, model, total_tokens, node_work)
|
||||
print(
|
||||
@@ -1984,6 +2095,47 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
return
|
||||
self._send_json(200, server.billing.snapshot())
|
||||
|
||||
def _handle_dashboard(self):
|
||||
"""Serve the read-only web dashboard (US-035). Any tracker in the
|
||||
mesh — leader or follower — serves it from its replicated state."""
|
||||
try:
|
||||
html = files("meshnet_tracker").joinpath("dashboard.html").read_text()
|
||||
except (FileNotFoundError, OSError):
|
||||
self._send_json(404, {"error": "dashboard asset missing"})
|
||||
return
|
||||
body = html.encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
try:
|
||||
self.wfile.write(body)
|
||||
except BrokenPipeError:
|
||||
pass
|
||||
|
||||
def _handle_registry_wallets(self):
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
if server.contracts is None:
|
||||
self._send_json(200, {"wallets": {}})
|
||||
return
|
||||
wallets = server.contracts.registry.list_wallets()
|
||||
self._send_json(200, {"wallets": {
|
||||
wallet: {
|
||||
"stake_balance": state.stake_balance,
|
||||
"strike_count": state.strike_count,
|
||||
"banned": state.banned,
|
||||
"completed_jobs": state.completed_job_count,
|
||||
}
|
||||
for wallet, state in wallets.items()
|
||||
}})
|
||||
|
||||
def _handle_billing_settlements(self):
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
if server.billing is None:
|
||||
self._send_json(404, {"error": "billing is not enabled on this tracker"})
|
||||
return
|
||||
self._send_json(200, {"settlements": server.billing.settlement_history()})
|
||||
|
||||
def _handle_billing_gossip(self):
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
body = self._read_json_body()
|
||||
@@ -1999,6 +2151,178 @@ class _TrackerHandler(http.server.BaseHTTPRequestHandler):
|
||||
applied = server.billing.apply_events([e for e in events if isinstance(e, dict)])
|
||||
self._send_json(200, {"applied": applied})
|
||||
|
||||
def _handle_billing_forfeit(self):
|
||||
"""Privileged: forfeit a node's pending balance + record a strike (US-034).
|
||||
|
||||
Auth is a header-presence stub (non-empty Authorization), matching the
|
||||
benchmark endpoints — real validator auth arrives with on-chain keys.
|
||||
"""
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
if not self.headers.get("Authorization"):
|
||||
self._send_json(401, {"error": "Authorization header required"})
|
||||
return
|
||||
if server.billing is None:
|
||||
self._send_json(404, {"error": "billing is not enabled on this tracker"})
|
||||
return
|
||||
body = self._read_json_body()
|
||||
if body is None:
|
||||
return
|
||||
wallet = body.get("wallet")
|
||||
if not wallet or not isinstance(wallet, str):
|
||||
self._send_json(400, {"error": "wallet is required"})
|
||||
return
|
||||
reason = body.get("reason") or "fraud"
|
||||
event = server.billing.forfeit_pending(wallet, reason=str(reason))
|
||||
strike_state: dict = {}
|
||||
if server.contracts is not None:
|
||||
server.contracts.registry.record_strike(wallet)
|
||||
wallet_state = server.contracts.registry.get_wallet(wallet)
|
||||
strike_state = {
|
||||
"strike_count": wallet_state.strike_count,
|
||||
"banned": wallet_state.banned,
|
||||
}
|
||||
print(
|
||||
f"[tracker] forfeited pending balance of {wallet}: "
|
||||
f"{event['amount']:.6f} USDT ({reason}) strikes={strike_state.get('strike_count', 'n/a')}",
|
||||
flush=True,
|
||||
)
|
||||
self._send_json(200, {"forfeited": event["amount"], **strike_state})
|
||||
|
||||
def _handle_wallet_register(self):
|
||||
"""Bind the caller's client wallet pubkey to their API key (US-032).
|
||||
|
||||
Deposits from that wallet into the treasury are then credited to the
|
||||
API key's ledger balance by the deposit watcher.
|
||||
"""
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
api_key = _api_key_from_headers(self.headers)
|
||||
if not api_key:
|
||||
self._send_json(401, {"error": "Authorization header required"})
|
||||
return
|
||||
if server.billing is None:
|
||||
self._send_json(404, {"error": "billing is not enabled on this tracker"})
|
||||
return
|
||||
body = self._read_json_body()
|
||||
if body is None:
|
||||
return
|
||||
wallet = body.get("wallet")
|
||||
if not wallet or not isinstance(wallet, str):
|
||||
self._send_json(400, {"error": "wallet is required"})
|
||||
return
|
||||
server.billing.bind_wallet(api_key, wallet)
|
||||
print(f"[tracker] wallet bound: {wallet} -> api_key …{api_key[-6:]}", flush=True)
|
||||
self._send_json(200, {"wallet": wallet, "bound": True})
|
||||
|
||||
def _handle_benchmark_hop_penalty(self):
|
||||
"""Privileged: run the same prompt through 1/2/3-node pinned routes (US-030).
|
||||
|
||||
Data collection only — the routing algorithm is unchanged. Per-hop
|
||||
latency is derived incrementally: the k-node route's final hop penalty
|
||||
is its total minus the (k-1)-node route's total.
|
||||
"""
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
auth = self.headers.get("Authorization")
|
||||
if not auth:
|
||||
self._send_json(401, {"error": "Authorization header required"})
|
||||
return
|
||||
body = self._read_json_body()
|
||||
if body is None:
|
||||
return
|
||||
model = body.get("model", "")
|
||||
if not model:
|
||||
self._send_json(400, {"error": "model is required"})
|
||||
return
|
||||
prompt = body.get("prompt") or "Benchmark: say OK."
|
||||
max_new_tokens = int(body.get("max_new_tokens", 64))
|
||||
|
||||
with server.lock:
|
||||
self._purge_expired_nodes()
|
||||
resolved = _nodes_and_bounds_for_model(server, model)
|
||||
if resolved is None or not resolved[0]:
|
||||
self._send_json(404, {"error": f"no nodes registered for model {model!r}"})
|
||||
return
|
||||
all_nodes, rs, re = resolved
|
||||
|
||||
self_url = f"http://127.0.0.1:{self.server.server_address[1]}"
|
||||
route_results: list[dict] = []
|
||||
prev_total_ms: float | None = None
|
||||
prev_per_hop: list[float] = []
|
||||
for hop_count in (1, 2, 3):
|
||||
combo = _find_pinned_route(all_nodes, rs, re, hop_count)
|
||||
if combo is None:
|
||||
continue # insufficient coverage for this hop count — skip
|
||||
request_body = json.dumps({
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_new_tokens,
|
||||
"route": [n.node_id for n in combo],
|
||||
}).encode()
|
||||
req = urllib.request.Request(
|
||||
f"{self_url}/v1/chat/completions",
|
||||
data=request_body,
|
||||
headers={"Content-Type": "application/json", "Authorization": auth},
|
||||
method="POST",
|
||||
)
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=300.0) as resp:
|
||||
payload = json.loads(resp.read())
|
||||
except Exception as exc:
|
||||
route_results.append({
|
||||
"route": [n.node_id for n in combo],
|
||||
"error": str(exc),
|
||||
})
|
||||
continue
|
||||
total_ms = (time.monotonic() - started) * 1000.0
|
||||
if prev_total_ms is not None and len(prev_per_hop) == hop_count - 1:
|
||||
per_hop_ms = prev_per_hop + [max(0.0, total_ms - prev_total_ms)]
|
||||
else:
|
||||
per_hop_ms = [total_ms / hop_count] * hop_count
|
||||
prev_total_ms = total_ms
|
||||
prev_per_hop = per_hop_ms
|
||||
route_results.append({
|
||||
"route": [n.node_id for n in combo],
|
||||
"total_ms": total_ms,
|
||||
"per_hop_ms": per_hop_ms,
|
||||
"tokens_generated": _usage_total_tokens(payload) or 0,
|
||||
})
|
||||
|
||||
record = {
|
||||
"timestamp": time.time(),
|
||||
"model": model,
|
||||
"prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:16],
|
||||
"routes": route_results,
|
||||
}
|
||||
with server.benchmark_lock:
|
||||
existing: list = []
|
||||
if os.path.exists(server.benchmark_results_path):
|
||||
try:
|
||||
with open(server.benchmark_results_path, encoding="utf-8") as fh:
|
||||
existing = json.load(fh)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
existing = []
|
||||
if not isinstance(existing, list):
|
||||
existing = []
|
||||
existing.append(record)
|
||||
with open(server.benchmark_results_path, "w", encoding="utf-8") as fh:
|
||||
json.dump(existing, fh, indent=2)
|
||||
self._send_json(200, record)
|
||||
|
||||
def _handle_benchmark_results(self):
|
||||
server: _TrackerHTTPServer = self.server # type: ignore[assignment]
|
||||
if not self.headers.get("Authorization"):
|
||||
self._send_json(401, {"error": "Authorization header required"})
|
||||
return
|
||||
results: list = []
|
||||
with server.benchmark_lock:
|
||||
if os.path.exists(server.benchmark_results_path):
|
||||
try:
|
||||
with open(server.benchmark_results_path, encoding="utf-8") as fh:
|
||||
results = json.load(fh)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
results = []
|
||||
self._send_json(200, {"results": results if isinstance(results, list) else []})
|
||||
|
||||
def _handle_assign(self, parsed: urllib.parse.ParseResult):
|
||||
"""Return an optimal shard assignment for a node given its hardware profile.
|
||||
|
||||
@@ -2430,6 +2754,13 @@ class TrackerServer:
|
||||
relay_url: str | None = None,
|
||||
billing: BillingLedger | None = None,
|
||||
billing_db: str | None = None,
|
||||
benchmark_results_path: str | None = None,
|
||||
treasury: Any | None = None,
|
||||
deposit_poll_interval: float = 15.0,
|
||||
settle_period: float = 86400.0,
|
||||
payout_threshold: float = 5.0,
|
||||
payout_dust_floor: float = 0.01,
|
||||
settlement_check_interval: float | None = None,
|
||||
) -> None:
|
||||
self._host = host
|
||||
self._requested_port = port
|
||||
@@ -2463,6 +2794,19 @@ class TrackerServer:
|
||||
billing = BillingLedger(db_path=billing_db, prices=preset_prices)
|
||||
self._billing: BillingLedger | None = billing
|
||||
self._billing_gossip_cursor = 0
|
||||
self._benchmark_results_path = benchmark_results_path
|
||||
self._treasury = treasury
|
||||
self._deposit_poll_interval = deposit_poll_interval
|
||||
self._deposit_stop = threading.Event()
|
||||
self._deposit_thread: threading.Thread | None = None
|
||||
self._settle_period = settle_period
|
||||
self._payout_threshold = payout_threshold
|
||||
self._payout_dust_floor = payout_dust_floor
|
||||
self._settlement_check_interval = settlement_check_interval or max(
|
||||
1.0, min(settle_period / 4.0, 15.0)
|
||||
)
|
||||
self._settlement_stop = threading.Event()
|
||||
self._settlement_thread: threading.Thread | None = None
|
||||
self.port: int | None = None
|
||||
|
||||
def start(self) -> int:
|
||||
@@ -2487,6 +2831,7 @@ class TrackerServer:
|
||||
relay_url=effective_relay_url,
|
||||
stats=self._stats,
|
||||
billing=self._billing,
|
||||
benchmark_results_path=self._benchmark_results_path,
|
||||
)
|
||||
self.port = self._server.server_address[1]
|
||||
|
||||
@@ -2512,8 +2857,107 @@ class TrackerServer:
|
||||
self._rebalance_thread.start()
|
||||
self._stats_thread = threading.Thread(target=self._stats_loop, daemon=True)
|
||||
self._stats_thread.start()
|
||||
if self._treasury is not None and self._billing is not None:
|
||||
self._deposit_stop.clear()
|
||||
self._deposit_thread = threading.Thread(target=self._deposit_loop, daemon=True)
|
||||
self._deposit_thread.start()
|
||||
self._settlement_stop.clear()
|
||||
self._settlement_thread = threading.Thread(target=self._settlement_loop, daemon=True)
|
||||
self._settlement_thread.start()
|
||||
return self.port
|
||||
|
||||
def _settlement_loop(self) -> None:
|
||||
"""Leader-only on-chain settlement (US-033, ADR-0015).
|
||||
|
||||
Pay a node when pending ≥ threshold OR its pending age ≥ max period,
|
||||
with a dust floor. Pending is debited (payout events, replicated)
|
||||
before the transaction is sent; unconfirmed batches are resent with
|
||||
the same settlement id, so retries never double-pay.
|
||||
"""
|
||||
billing = self._billing
|
||||
treasury = self._treasury
|
||||
assert billing is not None and treasury is not None
|
||||
while not self._settlement_stop.wait(self._settlement_check_interval):
|
||||
if self._raft is not None and not self._raft.is_leader:
|
||||
continue # followers replicate the ledger but never sign
|
||||
# resend batches whose transaction never confirmed
|
||||
for settlement_id, payouts in billing.unconfirmed_settlements().items():
|
||||
self._send_settlement(settlement_id, payouts)
|
||||
banned: set[str] = set()
|
||||
if self._server is not None and self._server.contracts is not None:
|
||||
registry = self._server.contracts.registry
|
||||
banned = {
|
||||
wallet for wallet, _ in billing.payables(
|
||||
threshold=0.0, max_period=0.0, dust_floor=0.0,
|
||||
)
|
||||
if registry.get_wallet(wallet).banned
|
||||
}
|
||||
due = billing.payables(
|
||||
threshold=self._payout_threshold,
|
||||
max_period=self._settle_period,
|
||||
dust_floor=self._payout_dust_floor,
|
||||
exclude=banned,
|
||||
)
|
||||
if not due:
|
||||
continue
|
||||
settlement_id = uuid.uuid4().hex
|
||||
for wallet, amount in due:
|
||||
billing.settle_node_payout(wallet, amount, reference=settlement_id)
|
||||
self._send_settlement(settlement_id, due)
|
||||
|
||||
def _send_settlement(self, settlement_id: str, payouts: list) -> None:
|
||||
billing = self._billing
|
||||
treasury = self._treasury
|
||||
assert billing is not None and treasury is not None
|
||||
try:
|
||||
signature = treasury.send_payouts([(w, a) for w, a in payouts])
|
||||
except Exception as exc:
|
||||
print(
|
||||
f"[tracker] settlement {settlement_id} failed (will retry): {exc}",
|
||||
flush=True,
|
||||
)
|
||||
return
|
||||
billing.confirm_settlement(settlement_id, signature)
|
||||
total = sum(a for _, a in payouts)
|
||||
print(
|
||||
f"[tracker] settled {settlement_id}: {len(payouts)} payout(s), "
|
||||
f"{total:.6f} USDT total (tx {signature})",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
def _deposit_loop(self) -> None:
|
||||
"""Poll the treasury token account and credit confirmed deposits (US-032)."""
|
||||
billing = self._billing
|
||||
treasury = self._treasury
|
||||
assert billing is not None and treasury is not None
|
||||
while not self._deposit_stop.wait(self._deposit_poll_interval):
|
||||
try:
|
||||
deposits = treasury.list_new_deposits(
|
||||
lambda sig: billing.has_event(f"deposit-{sig}")
|
||||
)
|
||||
except Exception as exc:
|
||||
print(f"[tracker] deposit poll failed: {exc}", flush=True)
|
||||
continue
|
||||
for deposit in deposits:
|
||||
api_key = billing.api_key_for_wallet(deposit.sender_wallet)
|
||||
if api_key is None:
|
||||
print(
|
||||
f"[tracker] unattributed deposit {deposit.signature}: "
|
||||
f"{deposit.amount_usdt} USDT from unbound wallet "
|
||||
f"{deposit.sender_wallet} — register via /v1/wallet/register",
|
||||
flush=True,
|
||||
)
|
||||
continue
|
||||
credited = billing.credit_deposit(
|
||||
api_key, deposit.amount_usdt, deposit.signature
|
||||
)
|
||||
if credited is not None:
|
||||
print(
|
||||
f"[tracker] deposit credited: {deposit.amount_usdt} USDT "
|
||||
f"-> api_key …{api_key[-6:]} (tx {deposit.signature})",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
def _raft_apply(self, command: str, payload: dict) -> None:
|
||||
"""Called by RaftNode when a log entry is committed — replicate to local registry."""
|
||||
if command != "register":
|
||||
@@ -2611,6 +3055,8 @@ class TrackerServer:
|
||||
return
|
||||
self._rebalance_stop.set()
|
||||
self._stats_stop.set()
|
||||
self._deposit_stop.set()
|
||||
self._settlement_stop.set()
|
||||
if self._stats is not None:
|
||||
self._stats.save_to_db()
|
||||
if self._billing is not None:
|
||||
@@ -2623,6 +3069,12 @@ class TrackerServer:
|
||||
self._rebalance_thread.join(timeout=1)
|
||||
if self._stats_thread is not None:
|
||||
self._stats_thread.join(timeout=1)
|
||||
if self._deposit_thread is not None:
|
||||
self._deposit_thread.join(timeout=1)
|
||||
self._deposit_thread = None
|
||||
if self._settlement_thread is not None:
|
||||
self._settlement_thread.join(timeout=1)
|
||||
self._settlement_thread = None
|
||||
self._server = None
|
||||
self._thread = None
|
||||
self._rebalance_thread = None
|
||||
|
||||
@@ -20,4 +20,4 @@ where = ["."]
|
||||
include = ["meshnet_tracker*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
meshnet_tracker = ["*.json"]
|
||||
meshnet_tracker = ["*.json", "*.html"]
|
||||
|
||||
46
packages/validator/README.md
Normal file
46
packages/validator/README.md
Normal file
@@ -0,0 +1,46 @@
|
||||
# meshnet-validator
|
||||
|
||||
Optimistic fraud detection (ADR-0003, penalty amended by ADR-0015): the
|
||||
validator re-runs a random ~5% sample of completed inference requests against
|
||||
a trusted reference node and, on divergence, submits a slash proof and
|
||||
forfeits the node's pending balance.
|
||||
|
||||
## Why the penalty deters cheating
|
||||
|
||||
There is no upfront stake. Settlement is periodic (US-033), so a node always
|
||||
has an unpaid **pending balance** — that balance *is* the collateral.
|
||||
|
||||
At a sampling rate `p`, a cheater is caught on average once every `1/p`
|
||||
fraudulent jobs, so cheating is unprofitable when:
|
||||
|
||||
```
|
||||
penalty > per_job_gain / p # p = 0.05 → penalty > 20 × per_job_gain
|
||||
```
|
||||
|
||||
With the production settlement period of 24h, the pending balance at any
|
||||
moment approximates a full day's earnings — hundreds to thousands of jobs —
|
||||
which is far above the 20× bar. Each catch also records a strike; three
|
||||
strikes ban the wallet (registration rejected, excluded from routes, unpaid
|
||||
pending never settled), and the probationary period (first N jobs unpaid)
|
||||
makes re-entry with a fresh wallet costly.
|
||||
|
||||
Two operational notes:
|
||||
|
||||
- Shortening the settlement period shrinks the collateral. Period changes
|
||||
must weigh chain overhead against deterrence.
|
||||
- A cheater immediately after a payout has little to forfeit — the
|
||||
strike/ban ladder covers that window.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
ValidatorProcess(
|
||||
contracts=contracts, # registry/validation boundary
|
||||
billing=ledger, # BillingLedger — enables forfeiture
|
||||
reference_node_url="http://...",
|
||||
sample_rate=0.05,
|
||||
)
|
||||
```
|
||||
|
||||
Remote validators can instead call the tracker's privileged
|
||||
`POST /v1/billing/forfeit` endpoint (non-empty Authorization header).
|
||||
@@ -1,159 +1,170 @@
|
||||
"""Optimistic fraud validator for completed inference requests."""
|
||||
|
||||
import json
|
||||
import math
|
||||
import random
|
||||
import threading
|
||||
import time
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
__version__ = "0.1.0"
|
||||
|
||||
|
||||
class ValidatorProcess:
|
||||
"""Separate validator loop that samples completed requests and submits slashes."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
contracts: Any,
|
||||
reference_node_url: str,
|
||||
sample_rate: float = 0.05,
|
||||
tolerance: float = 1e-6,
|
||||
slash_amount: int = 100,
|
||||
strike_threshold: int = 3,
|
||||
random_seed: int | None = None,
|
||||
webhook_url: str | None = None,
|
||||
interval_seconds: float = 1.0,
|
||||
) -> None:
|
||||
if not 0.0 <= sample_rate <= 1.0:
|
||||
raise ValueError("sample_rate must be between 0 and 1")
|
||||
if tolerance < 0:
|
||||
raise ValueError("tolerance must be non-negative")
|
||||
if slash_amount <= 0:
|
||||
raise ValueError("slash_amount must be positive")
|
||||
if strike_threshold <= 0:
|
||||
raise ValueError("strike_threshold must be positive")
|
||||
if interval_seconds <= 0:
|
||||
raise ValueError("interval_seconds must be positive")
|
||||
|
||||
self._contracts = contracts
|
||||
self._reference_node_url = reference_node_url.rstrip("/")
|
||||
self._sample_rate = sample_rate
|
||||
self._tolerance = tolerance
|
||||
self._slash_amount = slash_amount
|
||||
self._strike_threshold = strike_threshold
|
||||
self._webhook_url = webhook_url
|
||||
self._interval_seconds = interval_seconds
|
||||
self._random = random.Random(random_seed)
|
||||
self._last_event_index = -1
|
||||
self._running = False
|
||||
self._thread: threading.Thread | None = None
|
||||
self.sampled_count = 0
|
||||
|
||||
def validate_once(self) -> list[Any]:
|
||||
"""Run one validation cycle and return slash receipts submitted this cycle."""
|
||||
receipts: list[Any] = []
|
||||
events = self._contracts.validation.list_completed_inferences(
|
||||
after_index=self._last_event_index,
|
||||
)
|
||||
for event in events:
|
||||
self._last_event_index = max(self._last_event_index, event.index)
|
||||
if self._random.random() >= self._sample_rate:
|
||||
continue
|
||||
self.sampled_count += 1
|
||||
reference_output = self._run_reference(event.messages)
|
||||
if _outputs_match(event.observed_output, reference_output, self._tolerance):
|
||||
continue
|
||||
receipts.extend(self._slash_route(event.route_nodes, event.observed_output, reference_output))
|
||||
return receipts
|
||||
|
||||
def start(self) -> None:
|
||||
if self._running:
|
||||
raise RuntimeError("ValidatorProcess is already running")
|
||||
self._running = True
|
||||
self._thread = threading.Thread(target=self._run_loop, daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
self._running = False
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=2)
|
||||
self._thread = None
|
||||
|
||||
def _run_loop(self) -> None:
|
||||
while self._running:
|
||||
self.validate_once()
|
||||
time.sleep(self._interval_seconds)
|
||||
|
||||
def _run_reference(self, messages: list[dict]) -> str:
|
||||
response = _post_json(
|
||||
f"{self._reference_node_url}/v1/infer",
|
||||
{"messages": messages},
|
||||
)
|
||||
text = response.get("text")
|
||||
if not isinstance(text, str):
|
||||
raise ValueError("reference node response did not contain text")
|
||||
return text
|
||||
|
||||
def _slash_route(
|
||||
self,
|
||||
route_nodes: list[dict],
|
||||
observed_output: str,
|
||||
reference_output: str,
|
||||
) -> list[Any]:
|
||||
receipts: list[Any] = []
|
||||
node = _final_text_node(route_nodes)
|
||||
wallet_address = node.get("wallet_address") if node else None
|
||||
if not wallet_address:
|
||||
return receipts
|
||||
if self._contracts.registry.get_wallet(wallet_address).banned:
|
||||
return receipts
|
||||
receipts.append(self._contracts.registry.submit_slash_proof(
|
||||
wallet_address=wallet_address,
|
||||
slash_amount=self._slash_amount,
|
||||
strike_threshold=self._strike_threshold,
|
||||
reason=(
|
||||
"reference output diverged "
|
||||
f"(observed={observed_output!r}, reference={reference_output!r})"
|
||||
),
|
||||
webhook_url=self._webhook_url,
|
||||
))
|
||||
return receipts
|
||||
|
||||
|
||||
def _final_text_node(route_nodes: list[dict]) -> dict | None:
|
||||
if not route_nodes:
|
||||
return None
|
||||
return max(route_nodes, key=lambda node: int(node.get("shard_end", 0)))
|
||||
|
||||
|
||||
def _outputs_match(observed: str, reference: str, tolerance: float) -> bool:
|
||||
observed_float = _parse_float(observed)
|
||||
reference_float = _parse_float(reference)
|
||||
if observed_float is not None and reference_float is not None:
|
||||
return math.isclose(observed_float, reference_float, rel_tol=tolerance, abs_tol=tolerance)
|
||||
return observed == reference
|
||||
|
||||
|
||||
def _parse_float(value: str) -> float | None:
|
||||
try:
|
||||
return float(value)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _post_json(url: str, payload: dict, timeout: float = 5.0) -> dict:
|
||||
data = json.dumps(payload).encode()
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as response:
|
||||
return json.loads(response.read())
|
||||
|
||||
|
||||
__all__ = ["ValidatorProcess"]
|
||||
"""Optimistic fraud validator for completed inference requests."""
|
||||
|
||||
import json
|
||||
import math
|
||||
import random
|
||||
import threading
|
||||
import time
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
__version__ = "0.1.0"
|
||||
|
||||
|
||||
class ValidatorProcess:
|
||||
"""Separate validator loop that samples completed requests and submits slashes."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
contracts: Any,
|
||||
reference_node_url: str,
|
||||
sample_rate: float = 0.05,
|
||||
tolerance: float = 1e-6,
|
||||
slash_amount: int = 100,
|
||||
strike_threshold: int = 3,
|
||||
random_seed: int | None = None,
|
||||
webhook_url: str | None = None,
|
||||
interval_seconds: float = 1.0,
|
||||
billing: Any | None = None,
|
||||
) -> None:
|
||||
if not 0.0 <= sample_rate <= 1.0:
|
||||
raise ValueError("sample_rate must be between 0 and 1")
|
||||
if tolerance < 0:
|
||||
raise ValueError("tolerance must be non-negative")
|
||||
if slash_amount <= 0:
|
||||
raise ValueError("slash_amount must be positive")
|
||||
if strike_threshold <= 0:
|
||||
raise ValueError("strike_threshold must be positive")
|
||||
if interval_seconds <= 0:
|
||||
raise ValueError("interval_seconds must be positive")
|
||||
|
||||
self._contracts = contracts
|
||||
self._billing = billing
|
||||
self._reference_node_url = reference_node_url.rstrip("/")
|
||||
self._sample_rate = sample_rate
|
||||
self._tolerance = tolerance
|
||||
self._slash_amount = slash_amount
|
||||
self._strike_threshold = strike_threshold
|
||||
self._webhook_url = webhook_url
|
||||
self._interval_seconds = interval_seconds
|
||||
self._random = random.Random(random_seed)
|
||||
self._last_event_index = -1
|
||||
self._running = False
|
||||
self._thread: threading.Thread | None = None
|
||||
self.sampled_count = 0
|
||||
|
||||
def validate_once(self) -> list[Any]:
|
||||
"""Run one validation cycle and return slash receipts submitted this cycle."""
|
||||
receipts: list[Any] = []
|
||||
events = self._contracts.validation.list_completed_inferences(
|
||||
after_index=self._last_event_index,
|
||||
)
|
||||
for event in events:
|
||||
self._last_event_index = max(self._last_event_index, event.index)
|
||||
if self._random.random() >= self._sample_rate:
|
||||
continue
|
||||
self.sampled_count += 1
|
||||
reference_output = self._run_reference(event.messages)
|
||||
if _outputs_match(event.observed_output, reference_output, self._tolerance):
|
||||
continue
|
||||
receipts.extend(self._slash_route(event.route_nodes, event.observed_output, reference_output))
|
||||
return receipts
|
||||
|
||||
def start(self) -> None:
|
||||
if self._running:
|
||||
raise RuntimeError("ValidatorProcess is already running")
|
||||
self._running = True
|
||||
self._thread = threading.Thread(target=self._run_loop, daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
self._running = False
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=2)
|
||||
self._thread = None
|
||||
|
||||
def _run_loop(self) -> None:
|
||||
while self._running:
|
||||
self.validate_once()
|
||||
time.sleep(self._interval_seconds)
|
||||
|
||||
def _run_reference(self, messages: list[dict]) -> str:
|
||||
response = _post_json(
|
||||
f"{self._reference_node_url}/v1/infer",
|
||||
{"messages": messages},
|
||||
)
|
||||
text = response.get("text")
|
||||
if not isinstance(text, str):
|
||||
raise ValueError("reference node response did not contain text")
|
||||
return text
|
||||
|
||||
def _slash_route(
|
||||
self,
|
||||
route_nodes: list[dict],
|
||||
observed_output: str,
|
||||
reference_output: str,
|
||||
) -> list[Any]:
|
||||
receipts: list[Any] = []
|
||||
node = _final_text_node(route_nodes)
|
||||
wallet_address = node.get("wallet_address") if node else None
|
||||
if not wallet_address:
|
||||
return receipts
|
||||
if self._contracts.registry.get_wallet(wallet_address).banned:
|
||||
return receipts
|
||||
receipts.append(self._contracts.registry.submit_slash_proof(
|
||||
wallet_address=wallet_address,
|
||||
slash_amount=self._slash_amount,
|
||||
strike_threshold=self._strike_threshold,
|
||||
reason=(
|
||||
"reference output diverged "
|
||||
f"(observed={observed_output!r}, reference={reference_output!r})"
|
||||
),
|
||||
webhook_url=self._webhook_url,
|
||||
))
|
||||
# ADR-0015: the pending balance is the collateral — forfeit it in the
|
||||
# same validation cycle as the strike.
|
||||
if self._billing is not None:
|
||||
forfeit = self._billing.forfeit_pending(wallet_address, reason="fraud-divergence")
|
||||
print(
|
||||
f"[validator] forfeited pending balance of {wallet_address}: "
|
||||
f"{forfeit['amount']:.6f} USDT (fraud-divergence)",
|
||||
flush=True,
|
||||
)
|
||||
return receipts
|
||||
|
||||
|
||||
def _final_text_node(route_nodes: list[dict]) -> dict | None:
|
||||
if not route_nodes:
|
||||
return None
|
||||
return max(route_nodes, key=lambda node: int(node.get("shard_end", 0)))
|
||||
|
||||
|
||||
def _outputs_match(observed: str, reference: str, tolerance: float) -> bool:
|
||||
observed_float = _parse_float(observed)
|
||||
reference_float = _parse_float(reference)
|
||||
if observed_float is not None and reference_float is not None:
|
||||
return math.isclose(observed_float, reference_float, rel_tol=tolerance, abs_tol=tolerance)
|
||||
return observed == reference
|
||||
|
||||
|
||||
def _parse_float(value: str) -> float | None:
|
||||
try:
|
||||
return float(value)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def _post_json(url: str, payload: dict, timeout: float = 5.0) -> dict:
|
||||
data = json.dumps(payload).encode()
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as response:
|
||||
return json.loads(response.read())
|
||||
|
||||
|
||||
__all__ = ["ValidatorProcess"]
|
||||
|
||||
@@ -11,6 +11,9 @@ requires-python = ">=3.10"
|
||||
[project.optional-dependencies]
|
||||
dev = ["pytest>=8", "openai>=1", "langchain-openai>=0.1"]
|
||||
|
||||
[tool.setuptools]
|
||||
packages = []
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
markers = [
|
||||
|
||||
108
scripts/devnet_setup.py
Normal file
108
scripts/devnet_setup.py
Normal file
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Devnet custodial treasury setup (ADR-0015, US-032).
|
||||
|
||||
Creates the mock-USDT SPL mint (6 decimals, matching real USDT) and the
|
||||
treasury associated token account on Solana devnet, then prints/writes the
|
||||
.env.devnet values. Real USDT exists only on mainnet — the mint address is
|
||||
config, so mainnet cutover is a config change.
|
||||
|
||||
Usage:
|
||||
python scripts/devnet_setup.py # full setup
|
||||
python scripts/devnet_setup.py --mint <ADDR> # reuse existing mint
|
||||
python scripts/devnet_setup.py --mint <ADDR> --mint-to <WALLET> --amount 25
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "packages", "contracts"))
|
||||
|
||||
from solders.keypair import Keypair # noqa: E402
|
||||
|
||||
from meshnet_contracts.solana_adapter import SolanaCustodialTreasury # noqa: E402
|
||||
|
||||
DEFAULT_KEYPAIR = os.path.expanduser("~/.config/solana/meshnet-treasury.json")
|
||||
DEFAULT_RPC = "https://api.devnet.solana.com"
|
||||
# placeholder mint for bootstrapping the adapter before the real mint exists
|
||||
_SYSTEM_PROGRAM = "11111111111111111111111111111111"
|
||||
|
||||
|
||||
def _load_or_create_keypair(path: str) -> Keypair:
|
||||
if os.path.exists(path):
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
return Keypair.from_bytes(bytes(json.load(fh)))
|
||||
keypair = Keypair()
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as fh:
|
||||
json.dump(list(bytes(keypair)), fh)
|
||||
os.chmod(path, 0o600)
|
||||
print(f"generated treasury keypair: {path}")
|
||||
return keypair
|
||||
|
||||
|
||||
def _ensure_sol(treasury: SolanaCustodialTreasury, minimum_sol: float = 0.5) -> None:
|
||||
balance = treasury.get_sol_balance()
|
||||
if balance >= minimum_sol:
|
||||
print(f"treasury SOL balance: {balance:.4f}")
|
||||
return
|
||||
print(f"airdropping 2 SOL to {treasury.treasury_wallet} (balance {balance:.4f})…")
|
||||
signature = treasury.request_airdrop(2.0)
|
||||
for _ in range(30):
|
||||
time.sleep(2)
|
||||
if treasury.get_sol_balance() >= minimum_sol:
|
||||
print("airdrop confirmed")
|
||||
return
|
||||
sys.exit(
|
||||
f"airdrop {signature} not confirmed — devnet faucet may be rate-limited; retry later"
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--rpc-url", default=DEFAULT_RPC)
|
||||
parser.add_argument("--keypair", default=DEFAULT_KEYPAIR, help="Treasury keypair path")
|
||||
parser.add_argument("--mint", default=None, help="Reuse an existing mock-USDT mint address")
|
||||
parser.add_argument("--mint-to", default=None, metavar="WALLET",
|
||||
help="Mint mock USDT to a client wallet (creates their token account)")
|
||||
parser.add_argument("--amount", type=float, default=100.0, help="Amount for --mint-to")
|
||||
parser.add_argument("--env-out", default=".env.devnet", help="Env file to write")
|
||||
args = parser.parse_args()
|
||||
|
||||
keypair = _load_or_create_keypair(args.keypair)
|
||||
bootstrap = SolanaCustodialTreasury(args.rpc_url, args.mint or _SYSTEM_PROGRAM, keypair)
|
||||
print(f"treasury wallet: {bootstrap.treasury_wallet}")
|
||||
_ensure_sol(bootstrap)
|
||||
|
||||
if args.mint:
|
||||
treasury = SolanaCustodialTreasury(args.rpc_url, args.mint, keypair)
|
||||
mint_address = args.mint
|
||||
print(f"using existing mock-USDT mint: {mint_address}")
|
||||
else:
|
||||
treasury, mint_address = bootstrap.create_mock_usdt_mint()
|
||||
print(f"created mock-USDT mint (6 decimals): {mint_address}")
|
||||
time.sleep(2) # let the mint transaction confirm before creating ATAs
|
||||
|
||||
treasury.ensure_token_account(treasury.treasury_wallet)
|
||||
print(f"treasury token account: {treasury.treasury_token_account}")
|
||||
|
||||
if args.mint_to:
|
||||
signature = treasury.mint_mock_usdt(args.mint_to, args.amount)
|
||||
print(f"minted {args.amount} mock USDT to {args.mint_to} (tx {signature})")
|
||||
|
||||
env = (
|
||||
f"MESHNET_SOLANA_RPC_URL={args.rpc_url}\n"
|
||||
f"MESHNET_USDT_MINT={mint_address}\n"
|
||||
f"MESHNET_TREASURY_KEYPAIR={args.keypair}\n"
|
||||
f"MESHNET_TREASURY_WALLET={treasury.treasury_wallet}\n"
|
||||
f"MESHNET_TREASURY_TOKEN_ACCOUNT={treasury.treasury_token_account}\n"
|
||||
)
|
||||
with open(args.env_out, "w", encoding="utf-8") as fh:
|
||||
fh.write(env)
|
||||
print(f"\nwrote {args.env_out}:\n{env}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
60
tests/test_dashboard.py
Normal file
60
tests/test_dashboard.py
Normal file
@@ -0,0 +1,60 @@
|
||||
"""US-035: tracker web dashboard — served from any tracker, embedded asset."""
|
||||
|
||||
import json
|
||||
import urllib.request
|
||||
|
||||
from meshnet_contracts import LocalSolanaContracts
|
||||
from meshnet_tracker.billing import BillingLedger
|
||||
from meshnet_tracker.server import TrackerServer
|
||||
|
||||
PANELS = [
|
||||
"Tracker hive", "Nodes & coverage", "Client balances",
|
||||
"Node pending payouts", "Settlement history",
|
||||
"Strikes / bans / forfeitures", "Model usage",
|
||||
]
|
||||
|
||||
|
||||
def test_dashboard_served_with_all_panels():
|
||||
tracker = TrackerServer(billing=BillingLedger())
|
||||
port = tracker.start()
|
||||
try:
|
||||
html = urllib.request.urlopen(
|
||||
f"http://127.0.0.1:{port}/dashboard"
|
||||
).read().decode()
|
||||
for panel in PANELS:
|
||||
assert panel in html
|
||||
assert "<script>" in html # polling client embedded, no build step
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_dashboard_served_by_follower():
|
||||
"""A tracker that is not the leader (unreachable peers → never elected)
|
||||
still serves the dashboard from its own replicated state."""
|
||||
tracker = TrackerServer(
|
||||
billing=BillingLedger(),
|
||||
cluster_peers=["http://127.0.0.1:1", "http://127.0.0.1:2"],
|
||||
)
|
||||
port = tracker.start()
|
||||
try:
|
||||
response = urllib.request.urlopen(f"http://127.0.0.1:{port}/dashboard")
|
||||
assert response.status == 200
|
||||
assert "meshnet tracker" in response.read().decode()
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_registry_wallets_endpoint():
|
||||
contracts = LocalSolanaContracts()
|
||||
contracts.registry.submit_stake("wallet-a", 100)
|
||||
contracts.registry.record_strike("wallet-a")
|
||||
tracker = TrackerServer(contracts=contracts)
|
||||
port = tracker.start()
|
||||
try:
|
||||
data = json.loads(urllib.request.urlopen(
|
||||
f"http://127.0.0.1:{port}/v1/registry/wallets"
|
||||
).read())
|
||||
assert data["wallets"]["wallet-a"]["strike_count"] == 1
|
||||
assert data["wallets"]["wallet-a"]["banned"] is False
|
||||
finally:
|
||||
tracker.stop()
|
||||
223
tests/test_devnet_treasury.py
Normal file
223
tests/test_devnet_treasury.py
Normal file
@@ -0,0 +1,223 @@
|
||||
"""US-032: devnet custodial treasury — wallet binding + deposit watcher.
|
||||
|
||||
The deposit watcher polls the treasury token account and credits the sending
|
||||
wallet's bound API key exactly once per transaction signature. The watcher is
|
||||
exercised against a fake treasury (hermetic CI, ADR-0007); the real
|
||||
solana-test-validator flow runs only when the binary is installed.
|
||||
"""
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
from meshnet_tracker.billing import BillingLedger
|
||||
from meshnet_tracker.server import TrackerServer
|
||||
|
||||
|
||||
class _FakeDeposit:
|
||||
def __init__(self, signature, sender_wallet, amount_usdt):
|
||||
self.signature = signature
|
||||
self.sender_wallet = sender_wallet
|
||||
self.amount_usdt = amount_usdt
|
||||
|
||||
|
||||
class _FakeTreasury:
|
||||
"""Stands in for SolanaCustodialTreasury: a queue of confirmed deposits."""
|
||||
|
||||
def __init__(self):
|
||||
self.deposits: list[_FakeDeposit] = []
|
||||
|
||||
def list_new_deposits(self, is_seen, *, limit=200):
|
||||
return [d for d in self.deposits if not is_seen(d.signature)]
|
||||
|
||||
|
||||
def _post_json(url: str, payload: dict, headers: dict | None = None) -> dict:
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=json.dumps(payload).encode(),
|
||||
headers={"Content-Type": "application/json", **(headers or {})},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def watched_tracker():
|
||||
ledger = BillingLedger(starting_credit=0.0, default_price_per_1k=0.02)
|
||||
treasury = _FakeTreasury()
|
||||
tracker = TrackerServer(
|
||||
billing=ledger,
|
||||
treasury=treasury,
|
||||
deposit_poll_interval=0.1,
|
||||
)
|
||||
port = tracker.start()
|
||||
yield f"http://127.0.0.1:{port}", ledger, treasury
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def _wait_for(predicate, timeout=3.0):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(0.05)
|
||||
return False
|
||||
|
||||
|
||||
def test_wallet_register_requires_auth(watched_tracker):
|
||||
tracker_url, _, _ = watched_tracker
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
_post_json(f"{tracker_url}/v1/wallet/register", {"wallet": "So1anaWa11et111"})
|
||||
assert exc_info.value.code == 401
|
||||
|
||||
|
||||
def test_deposit_credits_bound_api_key_exactly_once(watched_tracker):
|
||||
tracker_url, ledger, treasury = watched_tracker
|
||||
reply = _post_json(
|
||||
f"{tracker_url}/v1/wallet/register",
|
||||
{"wallet": "So1anaWa11et111"},
|
||||
headers={"Authorization": "Bearer client-key-1"},
|
||||
)
|
||||
assert reply["bound"] is True
|
||||
|
||||
treasury.deposits.append(_FakeDeposit("sig-1", "So1anaWa11et111", 25.0))
|
||||
assert _wait_for(lambda: ledger.get_client_balance("client-key-1") > 0)
|
||||
assert ledger.get_client_balance("client-key-1") == pytest.approx(25.0)
|
||||
|
||||
# the same signature keeps being reported by the RPC — credited once only
|
||||
time.sleep(0.4)
|
||||
assert ledger.get_client_balance("client-key-1") == pytest.approx(25.0)
|
||||
|
||||
# replay through the ledger API directly is also a no-op
|
||||
assert ledger.credit_deposit("client-key-1", 25.0, "sig-1") is None
|
||||
|
||||
|
||||
def test_unbound_wallet_deposit_is_not_credited(watched_tracker):
|
||||
_, ledger, treasury = watched_tracker
|
||||
treasury.deposits.append(_FakeDeposit("sig-2", "UnknownWallet999", 10.0))
|
||||
time.sleep(0.4)
|
||||
assert ledger.snapshot()["clients"] == {}
|
||||
|
||||
|
||||
def test_binding_replicates_via_events():
|
||||
a = BillingLedger(starting_credit=0.0)
|
||||
b = BillingLedger(starting_credit=0.0)
|
||||
a.bind_wallet("key-9", "WalletNine")
|
||||
a.credit_deposit("key-9", 3.0, "sig-9")
|
||||
events, _ = a.events_since(0)
|
||||
b.apply_events(events)
|
||||
assert b.api_key_for_wallet("WalletNine") == "key-9"
|
||||
assert b.get_client_balance("key-9") == pytest.approx(3.0)
|
||||
# deposit event id embeds the signature → replicated replay is a no-op
|
||||
assert b.apply_events(events) == 0
|
||||
|
||||
|
||||
def test_solana_adapter_derives_treasury_accounts():
|
||||
"""Adapter smoke test without any RPC round-trip."""
|
||||
pytest.importorskip("solders")
|
||||
token_instructions = pytest.importorskip("spl.token.instructions")
|
||||
if not hasattr(token_instructions, "InitializeMintParams"):
|
||||
pytest.skip("installed spl.token lacks InitializeMintParams")
|
||||
from solders.keypair import Keypair
|
||||
|
||||
from meshnet_contracts.solana_adapter import SolanaCustodialTreasury
|
||||
|
||||
keypair = Keypair()
|
||||
treasury = SolanaCustodialTreasury(
|
||||
"http://127.0.0.1:8899",
|
||||
"Es9vMFrzaCERmJfrF4H2FYD4KCoNkY11McCe8BenwNYB", # real USDT mint address shape
|
||||
keypair,
|
||||
)
|
||||
assert treasury.treasury_wallet == str(keypair.pubkey())
|
||||
assert treasury.treasury_token_account # deterministic ATA derivation
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
shutil.which("solana-test-validator") is None,
|
||||
reason="solana-test-validator not installed",
|
||||
)
|
||||
def test_mint_deposit_credit_flow_against_local_validator(tmp_path):
|
||||
"""Full mint → deposit → credit flow on a local validator (US-032)."""
|
||||
import subprocess
|
||||
|
||||
from solders.keypair import Keypair
|
||||
|
||||
from meshnet_contracts.solana_adapter import RpcClient, SolanaCustodialTreasury
|
||||
|
||||
rpc_url = "http://127.0.0.1:8899"
|
||||
ledger_dir = tmp_path / "ledger"
|
||||
validator = subprocess.Popen(
|
||||
["solana-test-validator", "--ledger", str(ledger_dir), "--quiet",
|
||||
"--rpc-port", "8899", "--reset"],
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
try:
|
||||
assert _wait_for(lambda: _rpc_alive(RpcClient(rpc_url)), timeout=60)
|
||||
|
||||
treasury_kp, client_kp = Keypair(), Keypair()
|
||||
bootstrap = SolanaCustodialTreasury(
|
||||
rpc_url, "11111111111111111111111111111111", treasury_kp,
|
||||
)
|
||||
client_bootstrap = SolanaCustodialTreasury(
|
||||
rpc_url, "11111111111111111111111111111111", client_kp,
|
||||
)
|
||||
bootstrap.request_airdrop(2.0)
|
||||
client_bootstrap.request_airdrop(2.0)
|
||||
assert _wait_for(lambda: bootstrap.get_sol_balance() > 0, timeout=30)
|
||||
assert _wait_for(lambda: client_bootstrap.get_sol_balance() > 0, timeout=30)
|
||||
|
||||
treasury, mint = bootstrap.create_mock_usdt_mint()
|
||||
assert _wait_for(
|
||||
lambda: treasury._account_exists( # noqa: SLF001 — test-only poke
|
||||
__import__("solders.pubkey", fromlist=["Pubkey"]).Pubkey.from_string(mint)
|
||||
),
|
||||
timeout=30,
|
||||
)
|
||||
treasury.ensure_token_account(treasury.treasury_wallet)
|
||||
treasury.mint_mock_usdt(str(client_kp.pubkey()), 50.0)
|
||||
|
||||
# the client depositing into the treasury is just a transfer the
|
||||
# client signs — reuse send_payouts from an adapter bound to their key
|
||||
client_side = SolanaCustodialTreasury(rpc_url, mint, client_kp)
|
||||
assert _wait_for(
|
||||
lambda: _try_send(client_side, treasury.treasury_wallet, 25.0), timeout=30,
|
||||
)
|
||||
|
||||
ledger = BillingLedger(starting_credit=0.0)
|
||||
ledger.bind_wallet("api-key-e2e", str(client_kp.pubkey()))
|
||||
|
||||
def _credited():
|
||||
for dep in treasury.list_new_deposits(
|
||||
lambda sig: ledger.has_event(f"deposit-{sig}")
|
||||
):
|
||||
key = ledger.api_key_for_wallet(dep.sender_wallet)
|
||||
if key:
|
||||
ledger.credit_deposit(key, dep.amount_usdt, dep.signature)
|
||||
return ledger.get_client_balance("api-key-e2e") > 0
|
||||
|
||||
assert _wait_for(_credited, timeout=60)
|
||||
assert ledger.get_client_balance("api-key-e2e") == pytest.approx(25.0)
|
||||
finally:
|
||||
validator.terminate()
|
||||
validator.wait(timeout=10)
|
||||
|
||||
|
||||
def _try_send(adapter, wallet: str, amount: float) -> bool:
|
||||
try:
|
||||
adapter.send_payouts([(wallet, amount)])
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _rpc_alive(rpc) -> bool:
|
||||
try:
|
||||
return rpc.call("getHealth") == "ok"
|
||||
except Exception:
|
||||
return False
|
||||
261
tests/test_forfeiture_penalty.py
Normal file
261
tests/test_forfeiture_penalty.py
Normal file
@@ -0,0 +1,261 @@
|
||||
"""US-034: hardened proof-of-work — pending-balance forfeiture penalty.
|
||||
|
||||
The pending balance is the fraud collateral (ADR-0015): a confirmed divergence
|
||||
forfeits it in full to the protocol cut alongside the strike; three strikes ban
|
||||
the wallet, whose remaining pending is never paid out.
|
||||
"""
|
||||
|
||||
import json
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
from meshnet_contracts import LocalSolanaContracts
|
||||
from meshnet_node.server import StubNodeServer
|
||||
from meshnet_tracker.billing import BillingLedger
|
||||
from meshnet_tracker.server import TrackerServer
|
||||
from meshnet_validator import ValidatorProcess
|
||||
|
||||
MODEL = "stub-model"
|
||||
|
||||
|
||||
def _record_event(contracts, session_id: str, output: str, wallet: str) -> None:
|
||||
contracts.validation.record_completed_inference(
|
||||
session_id=session_id,
|
||||
model=MODEL,
|
||||
messages=[{"role": "user", "content": "2+2"}],
|
||||
observed_output=output,
|
||||
route_nodes=[{"wallet_address": wallet, "shard_end": 31}],
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def reference_node():
|
||||
node = StubNodeServer(shard_start=0, shard_end=31, is_last_shard=True)
|
||||
port = node.start()
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
node.stop()
|
||||
|
||||
|
||||
def _reference_output(reference_url: str) -> str:
|
||||
"""What the reference stub answers for the probe messages."""
|
||||
req = urllib.request.Request(
|
||||
f"{reference_url}/v1/infer",
|
||||
data=json.dumps({"messages": [{"role": "user", "content": "2+2"}]}).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req) as r:
|
||||
return json.loads(r.read())["text"]
|
||||
|
||||
|
||||
def test_divergence_forfeits_pending_and_strikes(reference_node):
|
||||
contracts = LocalSolanaContracts()
|
||||
contracts.registry.submit_stake("wallet-bad", 500)
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-bad", 12)])
|
||||
pending_before = ledger.get_node_pending("wallet-bad")
|
||||
assert pending_before > 0
|
||||
|
||||
validator = ValidatorProcess(
|
||||
contracts=contracts,
|
||||
billing=ledger,
|
||||
reference_node_url=reference_node,
|
||||
sample_rate=1.0,
|
||||
slash_amount=100,
|
||||
random_seed=7,
|
||||
)
|
||||
_record_event(contracts, "sess-1", "fraudulent nonsense", "wallet-bad")
|
||||
receipts = validator.validate_once()
|
||||
|
||||
assert len(receipts) == 1
|
||||
assert ledger.get_node_pending("wallet-bad") == pytest.approx(0.0)
|
||||
snapshot = ledger.snapshot()
|
||||
# forfeited amount landed in the protocol cut on top of the original 10%
|
||||
assert snapshot["protocol_cut"] == pytest.approx(0.02 * 0.10 + pending_before)
|
||||
assert contracts.registry.get_wallet("wallet-bad").strike_count == 1
|
||||
|
||||
|
||||
def test_matching_output_forfeits_nothing(reference_node):
|
||||
contracts = LocalSolanaContracts()
|
||||
contracts.registry.submit_stake("wallet-good", 500)
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-good", 12)])
|
||||
pending_before = ledger.get_node_pending("wallet-good")
|
||||
|
||||
validator = ValidatorProcess(
|
||||
contracts=contracts,
|
||||
billing=ledger,
|
||||
reference_node_url=reference_node,
|
||||
sample_rate=1.0,
|
||||
random_seed=7,
|
||||
)
|
||||
_record_event(contracts, "sess-1", _reference_output(reference_node), "wallet-good")
|
||||
receipts = validator.validate_once()
|
||||
|
||||
assert receipts == []
|
||||
assert ledger.get_node_pending("wallet-good") == pytest.approx(pending_before)
|
||||
assert contracts.registry.get_wallet("wallet-good").strike_count == 0
|
||||
|
||||
|
||||
def test_three_strikes_bans_and_bad_node_loses_everything(reference_node):
|
||||
"""Deliberately-bad node: every job is fraudulent, checks always sample.
|
||||
|
||||
Earn → caught → forfeit, three times over; the third strike bans the
|
||||
wallet, and the tracker rejects its registration.
|
||||
"""
|
||||
contracts = LocalSolanaContracts()
|
||||
contracts.registry.submit_stake("wallet-bad", 500)
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
validator = ValidatorProcess(
|
||||
contracts=contracts,
|
||||
billing=ledger,
|
||||
reference_node_url=reference_node,
|
||||
sample_rate=1.0,
|
||||
strike_threshold=3,
|
||||
random_seed=7,
|
||||
)
|
||||
|
||||
for i in range(3):
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-bad", 12)])
|
||||
_record_event(contracts, f"sess-{i}", f"fraud-{i}", "wallet-bad")
|
||||
validator.validate_once()
|
||||
|
||||
wallet = contracts.registry.get_wallet("wallet-bad")
|
||||
assert wallet.strike_count == 3
|
||||
assert wallet.banned
|
||||
assert ledger.get_node_pending("wallet-bad") == pytest.approx(0.0)
|
||||
|
||||
# banned wallet cannot register with the tracker
|
||||
tracker = TrackerServer(contracts=contracts)
|
||||
port = tracker.start()
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
f"http://127.0.0.1:{port}/v1/nodes/register",
|
||||
data=json.dumps({
|
||||
"endpoint": "http://127.0.0.1:9",
|
||||
"shard_start": 0,
|
||||
"shard_end": 31,
|
||||
"hardware_profile": {},
|
||||
"wallet_address": "wallet-bad",
|
||||
"score": 1.0,
|
||||
}).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
urllib.request.urlopen(req)
|
||||
assert exc_info.value.code == 403
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_forfeit_endpoint_requires_auth_and_forfeits():
|
||||
contracts = LocalSolanaContracts()
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-x", 12)])
|
||||
pending = ledger.get_node_pending("wallet-x")
|
||||
|
||||
tracker = TrackerServer(contracts=contracts, billing=ledger)
|
||||
port = tracker.start()
|
||||
url = f"http://127.0.0.1:{port}/v1/billing/forfeit"
|
||||
body = json.dumps({"wallet": "wallet-x", "reason": "fraud"}).encode()
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
url, data=body, headers={"Content-Type": "application/json"}, method="POST",
|
||||
)
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
urllib.request.urlopen(req)
|
||||
assert exc_info.value.code == 401
|
||||
|
||||
req = urllib.request.Request(
|
||||
url, data=body,
|
||||
headers={"Content-Type": "application/json", "Authorization": "Bearer validator"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req) as r:
|
||||
result = json.loads(r.read())
|
||||
assert result["forfeited"] == pytest.approx(pending)
|
||||
assert result["strike_count"] == 1
|
||||
assert result["banned"] is False
|
||||
assert ledger.get_node_pending("wallet-x") == pytest.approx(0.0)
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_probation_earns_nothing_then_earning_begins():
|
||||
"""First N jobs accrue no pending balance; job N+1 earns (issue 08)."""
|
||||
contracts = LocalSolanaContracts(probationary_job_count=2)
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
|
||||
tracker = TrackerServer(
|
||||
model_presets={
|
||||
MODEL: {
|
||||
"layers_start": 0,
|
||||
"layers_end": 11,
|
||||
"bytes_per_layer": {"bfloat16": 30 * 1024 * 1024},
|
||||
}
|
||||
},
|
||||
contracts=contracts,
|
||||
billing=ledger,
|
||||
)
|
||||
port = tracker.start()
|
||||
tracker_url = f"http://127.0.0.1:{port}"
|
||||
|
||||
stub = StubNodeServer(
|
||||
shard_start=0, shard_end=11, is_last_shard=True, model=MODEL, tracker_mode=True,
|
||||
)
|
||||
stub_port = stub.start()
|
||||
reg = json.dumps({
|
||||
"endpoint": f"http://127.0.0.1:{stub_port}",
|
||||
"shard_start": 0,
|
||||
"shard_end": 11,
|
||||
"model": MODEL,
|
||||
"hardware_profile": {},
|
||||
"score": 1.0,
|
||||
"tracker_mode": True,
|
||||
"wallet_address": "wallet-new",
|
||||
}).encode()
|
||||
|
||||
def _chat():
|
||||
req = urllib.request.Request(
|
||||
f"{tracker_url}/v1/chat/completions",
|
||||
data=json.dumps({
|
||||
"model": MODEL,
|
||||
"messages": [{"role": "user", "content": "hi"}],
|
||||
}).encode(),
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": "Bearer probation-client",
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req) as r:
|
||||
r.read()
|
||||
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
f"{tracker_url}/v1/nodes/register",
|
||||
data=reg,
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req) as r:
|
||||
r.read()
|
||||
|
||||
_chat() # job 1: probation
|
||||
_chat() # job 2: probation
|
||||
assert ledger.get_node_pending("wallet-new") == pytest.approx(0.0)
|
||||
assert contracts.registry.probationary_jobs_remaining("wallet-new") == 0
|
||||
|
||||
_chat() # job 3: earning begins (tokens are 0 in the stub, so the
|
||||
# charge event exists but the amount is 0 — assert via event log)
|
||||
events, _ = ledger.events_since(0)
|
||||
charges = [e for e in events if e["type"] == "charge"]
|
||||
assert len(charges) == 3
|
||||
assert charges[0]["shares"] == {} and charges[1]["shares"] == {}
|
||||
assert "wallet-new" in charges[2]["shares"]
|
||||
finally:
|
||||
stub.stop()
|
||||
tracker.stop()
|
||||
154
tests/test_manual_route_benchmark.py
Normal file
154
tests/test_manual_route_benchmark.py
Normal file
@@ -0,0 +1,154 @@
|
||||
"""US-030: manual route selection + hop-penalty benchmarking.
|
||||
|
||||
Pinned "route" in the chat body overrides auto-selection; the privileged
|
||||
benchmark endpoint fans the same prompt through 1/2/3-node routes and appends
|
||||
per-hop latency records to benchmark_results.json.
|
||||
"""
|
||||
|
||||
import json
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
from meshnet_node.server import StubNodeServer
|
||||
from meshnet_tracker.server import TrackerServer
|
||||
|
||||
MODEL = "openai-community/gpt2"
|
||||
|
||||
|
||||
def _post_json(url: str, payload: dict, headers: dict | None = None) -> dict:
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=json.dumps(payload).encode(),
|
||||
headers={"Content-Type": "application/json", **(headers or {})},
|
||||
method="POST",
|
||||
)
|
||||
with urllib.request.urlopen(req) as r:
|
||||
return json.loads(r.read())
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def benchmark_setup(tmp_path):
|
||||
"""Tracker + one full-coverage node and one 0-5/6-11 pair, with node ids."""
|
||||
results_path = str(tmp_path / "benchmark_results.json")
|
||||
tracker = TrackerServer(
|
||||
model_presets={
|
||||
MODEL: {
|
||||
"layers_start": 0,
|
||||
"layers_end": 11,
|
||||
"bytes_per_layer": {"bfloat16": 30 * 1024 * 1024},
|
||||
}
|
||||
},
|
||||
benchmark_results_path=results_path,
|
||||
)
|
||||
port = tracker.start()
|
||||
tracker_url = f"http://127.0.0.1:{port}"
|
||||
|
||||
stubs = []
|
||||
node_ids = {}
|
||||
for name, (start, end, prefix, last) in {
|
||||
"full": (0, 11, "full:", True),
|
||||
"head": (0, 5, "head:", False),
|
||||
"tail": (6, 11, "tail:", True),
|
||||
}.items():
|
||||
stub = StubNodeServer(
|
||||
shard_start=start, shard_end=end, is_last_shard=last,
|
||||
response_prefix=prefix, model=MODEL, tracker_mode=(start == 0),
|
||||
)
|
||||
sport = stub.start()
|
||||
stubs.append(stub)
|
||||
reply = _post_json(f"{tracker_url}/v1/nodes/register", {
|
||||
"endpoint": f"http://127.0.0.1:{sport}",
|
||||
"shard_start": start,
|
||||
"shard_end": end,
|
||||
"model": MODEL,
|
||||
"hardware_profile": {},
|
||||
"score": 1.0,
|
||||
"tracker_mode": start == 0,
|
||||
"node_id": f"node-{name}",
|
||||
})
|
||||
node_ids[name] = reply.get("node_id", f"node-{name}")
|
||||
|
||||
yield tracker_url, node_ids, results_path
|
||||
|
||||
for stub in stubs:
|
||||
stub.stop()
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def _chat(tracker_url: str, route: list[str] | None = None) -> dict:
|
||||
body: dict = {
|
||||
"model": MODEL,
|
||||
"messages": [{"role": "user", "content": "ping"}],
|
||||
}
|
||||
if route is not None:
|
||||
body["route"] = route
|
||||
return _post_json(f"{tracker_url}/v1/chat/completions", body)
|
||||
|
||||
|
||||
def test_pinned_route_uses_named_node(benchmark_setup):
|
||||
tracker_url, node_ids, _ = benchmark_setup
|
||||
reply = _chat(tracker_url, route=[node_ids["full"]])
|
||||
content = reply["choices"][0]["message"]["content"]
|
||||
assert content.startswith("full:")
|
||||
|
||||
|
||||
def test_unknown_route_node_is_400(benchmark_setup):
|
||||
tracker_url, _, _ = benchmark_setup
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
_chat(tracker_url, route=["no-such-node"])
|
||||
assert exc_info.value.code == 400
|
||||
detail = json.loads(exc_info.value.read())
|
||||
assert "no-such-node" in detail["error"]["message"]
|
||||
|
||||
|
||||
def test_invalid_route_shape_is_400(benchmark_setup):
|
||||
tracker_url, _, _ = benchmark_setup
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
_chat(tracker_url, route=[])
|
||||
assert exc_info.value.code == 400
|
||||
|
||||
|
||||
def test_clients_without_route_are_unaffected(benchmark_setup):
|
||||
tracker_url, _, _ = benchmark_setup
|
||||
reply = _chat(tracker_url)
|
||||
assert reply["choices"][0]["message"]["content"]
|
||||
|
||||
|
||||
def test_benchmark_requires_auth(benchmark_setup):
|
||||
tracker_url, _, _ = benchmark_setup
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
_post_json(f"{tracker_url}/v1/benchmark/hop-penalty", {"model": MODEL})
|
||||
assert exc_info.value.code == 401
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
urllib.request.urlopen(f"{tracker_url}/v1/benchmark/results")
|
||||
assert exc_info.value.code == 401
|
||||
|
||||
|
||||
def test_benchmark_records_one_and_two_node_routes(benchmark_setup):
|
||||
tracker_url, _, results_path = benchmark_setup
|
||||
record = _post_json(
|
||||
f"{tracker_url}/v1/benchmark/hop-penalty",
|
||||
{"model": MODEL, "prompt": "2+2", "max_new_tokens": 8},
|
||||
headers={"Authorization": "Bearer bench"},
|
||||
)
|
||||
by_hops = {len(r["route"]): r for r in record["routes"] if "total_ms" in r}
|
||||
assert 1 in by_hops and 2 in by_hops # 3-node coverage impossible → skipped
|
||||
assert 3 not in by_hops
|
||||
assert len(by_hops[1]["per_hop_ms"]) == 1
|
||||
assert len(by_hops[2]["per_hop_ms"]) == 2
|
||||
assert by_hops[2]["total_ms"] > 0
|
||||
|
||||
stored = json.loads(open(results_path, encoding="utf-8").read())
|
||||
assert isinstance(stored, list) and len(stored) == 1
|
||||
assert stored[0]["model"] == MODEL
|
||||
assert stored[0]["prompt_hash"]
|
||||
|
||||
fetched = json.loads(
|
||||
urllib.request.urlopen(urllib.request.Request(
|
||||
f"{tracker_url}/v1/benchmark/results",
|
||||
headers={"Authorization": "Bearer bench"},
|
||||
)).read()
|
||||
)
|
||||
assert len(fetched["results"]) == 1
|
||||
169
tests/test_settlement_loop.py
Normal file
169
tests/test_settlement_loop.py
Normal file
@@ -0,0 +1,169 @@
|
||||
"""US-033: leader-only settlement loop with batched USDT payouts.
|
||||
|
||||
Mining-pool trigger (threshold OR max-period, dust floor), pending debited
|
||||
before the transaction is sent, unconfirmed batches resent by settlement id
|
||||
(never double-paying), banned wallets skipped, history queryable over HTTP.
|
||||
"""
|
||||
|
||||
import json
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
import pytest
|
||||
|
||||
from meshnet_contracts import LocalSolanaContracts
|
||||
from meshnet_tracker.billing import BillingLedger
|
||||
from meshnet_tracker.server import TrackerServer
|
||||
|
||||
MODEL = "stub-model"
|
||||
|
||||
|
||||
class _FakePayoutTreasury:
|
||||
"""Records payout batches; can be told to fail the next N sends."""
|
||||
|
||||
def __init__(self):
|
||||
self.batches: list[list[tuple[str, float]]] = []
|
||||
self.fail_next = 0
|
||||
self._counter = 0
|
||||
|
||||
def list_new_deposits(self, is_seen, *, limit=200):
|
||||
return []
|
||||
|
||||
def send_payouts(self, payouts):
|
||||
if self.fail_next > 0:
|
||||
self.fail_next -= 1
|
||||
raise RuntimeError("rpc timeout (simulated)")
|
||||
self.batches.append(list(payouts))
|
||||
self._counter += 1
|
||||
return f"fake-tx-{self._counter}"
|
||||
|
||||
|
||||
def _wait_for(predicate, timeout=5.0):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return True
|
||||
time.sleep(0.05)
|
||||
return False
|
||||
|
||||
|
||||
def _make_tracker(ledger, treasury, *, threshold=0.01, period=3600.0,
|
||||
dust=0.001, contracts=None, cluster_peers=None):
|
||||
return TrackerServer(
|
||||
billing=ledger,
|
||||
treasury=treasury,
|
||||
contracts=contracts,
|
||||
deposit_poll_interval=30.0,
|
||||
settle_period=period,
|
||||
payout_threshold=threshold,
|
||||
payout_dust_floor=dust,
|
||||
settlement_check_interval=0.05,
|
||||
cluster_peers=cluster_peers,
|
||||
)
|
||||
|
||||
|
||||
def test_threshold_triggers_payout_and_zeroes_pending():
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-a", 12)]) # 0.018 pending
|
||||
treasury = _FakePayoutTreasury()
|
||||
tracker = _make_tracker(ledger, treasury, threshold=0.01)
|
||||
port = tracker.start()
|
||||
try:
|
||||
assert _wait_for(lambda: treasury.batches)
|
||||
assert treasury.batches[0] == [("wallet-a", pytest.approx(0.018))]
|
||||
assert ledger.get_node_pending("wallet-a") == pytest.approx(0.0)
|
||||
|
||||
history = json.loads(urllib.request.urlopen(
|
||||
f"http://127.0.0.1:{port}/v1/billing/settlements"
|
||||
).read())["settlements"]
|
||||
assert len(history) == 1
|
||||
assert history[0]["signature"] == "fake-tx-1"
|
||||
assert history[0]["payouts"] == [
|
||||
{"wallet": "wallet-a", "amount": pytest.approx(0.018)}
|
||||
]
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_period_triggers_payout_below_threshold():
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-a", 12)])
|
||||
treasury = _FakePayoutTreasury()
|
||||
tracker = _make_tracker(ledger, treasury, threshold=100.0, period=0.2)
|
||||
tracker.start()
|
||||
try:
|
||||
assert _wait_for(lambda: treasury.batches)
|
||||
assert ledger.get_node_pending("wallet-a") == pytest.approx(0.0)
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_dust_floor_blocks_tiny_payouts():
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 10, [("wallet-a", 12)]) # 0.00018 pending
|
||||
treasury = _FakePayoutTreasury()
|
||||
tracker = _make_tracker(ledger, treasury, threshold=0.0, period=0.1, dust=0.001)
|
||||
tracker.start()
|
||||
try:
|
||||
time.sleep(0.5)
|
||||
assert treasury.batches == []
|
||||
assert ledger.get_node_pending("wallet-a") > 0
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_failed_transaction_retries_without_double_pay():
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-a", 12)])
|
||||
treasury = _FakePayoutTreasury()
|
||||
treasury.fail_next = 3
|
||||
tracker = _make_tracker(ledger, treasury, threshold=0.01)
|
||||
tracker.start()
|
||||
try:
|
||||
assert _wait_for(lambda: treasury.batches)
|
||||
time.sleep(0.3) # give the loop room to (incorrectly) double-send
|
||||
assert len(treasury.batches) == 1
|
||||
assert treasury.batches[0] == [("wallet-a", pytest.approx(0.018))]
|
||||
assert ledger.get_node_pending("wallet-a") == pytest.approx(0.0)
|
||||
history = ledger.settlement_history()
|
||||
assert len(history) == 1 # one settlement id across all retries
|
||||
assert history[0]["signature"] == "fake-tx-1"
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_banned_wallet_is_never_paid():
|
||||
contracts = LocalSolanaContracts()
|
||||
contracts.registry.ban_wallet("wallet-banned")
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-banned", 6), ("wallet-ok", 6)])
|
||||
treasury = _FakePayoutTreasury()
|
||||
tracker = _make_tracker(ledger, treasury, threshold=0.001, contracts=contracts)
|
||||
tracker.start()
|
||||
try:
|
||||
assert _wait_for(lambda: treasury.batches)
|
||||
time.sleep(0.3)
|
||||
paid_wallets = {w for batch in treasury.batches for w, _ in batch}
|
||||
assert paid_wallets == {"wallet-ok"}
|
||||
assert ledger.get_node_pending("wallet-banned") > 0 # held, never paid
|
||||
finally:
|
||||
tracker.stop()
|
||||
|
||||
|
||||
def test_non_leader_never_signs():
|
||||
"""A tracker in a cluster whose peers are unreachable never wins an
|
||||
election, so it must never send a payout transaction."""
|
||||
ledger = BillingLedger(starting_credit=10.0, default_price_per_1k=0.02)
|
||||
ledger.charge_request("client", MODEL, 1000, [("wallet-a", 12)])
|
||||
treasury = _FakePayoutTreasury()
|
||||
tracker = _make_tracker(
|
||||
ledger, treasury, threshold=0.001,
|
||||
cluster_peers=["http://127.0.0.1:1", "http://127.0.0.1:2"],
|
||||
)
|
||||
tracker.start()
|
||||
try:
|
||||
time.sleep(0.8)
|
||||
assert treasury.batches == []
|
||||
assert ledger.get_node_pending("wallet-a") > 0
|
||||
finally:
|
||||
tracker.stop()
|
||||
@@ -1,8 +1,9 @@
|
||||
"""US-014 integration test: tracker-as-first-layer-node inference entry point.
|
||||
"""US-014 integration test: head-worker inference entry point.
|
||||
|
||||
Two stub tracker-nodes (shard 0-5, tracker_mode=True) + two mid-shard stub nodes
|
||||
(shard 6-11) for openai-community/gpt2. Ten requests via gateway assert round-robin
|
||||
load distribution across tracker-nodes and valid OpenAI-format responses.
|
||||
Two stub head workers (shard 0-5, tracker_mode=True for legacy wire
|
||||
compatibility) + two mid-shard stub nodes (shard 6-11) for openai-community/gpt2.
|
||||
Ten requests via gateway assert round-robin load distribution across head
|
||||
workers and valid OpenAI-format responses.
|
||||
"""
|
||||
|
||||
import json
|
||||
@@ -51,7 +52,7 @@ def _register_node(
|
||||
|
||||
@pytest.fixture
|
||||
def tracker_node_setup():
|
||||
"""Start tracker, two tracker-nodes (shard 0-5), two mid-shard nodes (shard 6-11),
|
||||
"""Start tracker, two head workers (shard 0-5), two mid-shard nodes (shard 6-11),
|
||||
and a gateway. Yields (gateway_url, tracker_node_a, tracker_node_b)."""
|
||||
|
||||
tracker = TrackerServer(
|
||||
@@ -66,12 +67,12 @@ def tracker_node_setup():
|
||||
tracker_port = tracker.start()
|
||||
tracker_url = f"http://127.0.0.1:{tracker_port}"
|
||||
|
||||
# Two tracker-nodes: serve shard 0-5, expose /v1/chat/completions
|
||||
# Two head workers: serve shard 0-5, expose /v1/chat/completions.
|
||||
tracker_node_a = StubNodeServer(
|
||||
shard_start=0,
|
||||
shard_end=5,
|
||||
is_last_shard=False,
|
||||
response_prefix="tracker-node-A:",
|
||||
response_prefix="head-worker-A:",
|
||||
model=GPT2_MODEL,
|
||||
tracker_mode=True,
|
||||
)
|
||||
@@ -81,7 +82,7 @@ def tracker_node_setup():
|
||||
shard_start=0,
|
||||
shard_end=5,
|
||||
is_last_shard=False,
|
||||
response_prefix="tracker-node-B:",
|
||||
response_prefix="head-worker-B:",
|
||||
model=GPT2_MODEL,
|
||||
tracker_mode=True,
|
||||
)
|
||||
@@ -106,7 +107,7 @@ def tracker_node_setup():
|
||||
)
|
||||
mid_port_b = mid_node_b.start()
|
||||
|
||||
# Register all nodes with tracker (tracker-nodes get tracker_mode=True)
|
||||
# Register all nodes with tracker (head workers keep legacy tracker_mode=True).
|
||||
_register_node(tracker_url, f"http://127.0.0.1:{port_a}", 0, 5, tracker_mode=True)
|
||||
_register_node(tracker_url, f"http://127.0.0.1:{port_b}", 0, 5, tracker_mode=True)
|
||||
_register_node(tracker_url, f"http://127.0.0.1:{mid_port_a}", 6, 11)
|
||||
@@ -155,22 +156,22 @@ def test_all_responses_valid_openai_format(tracker_node_setup):
|
||||
|
||||
|
||||
def test_both_tracker_nodes_receive_load(tracker_node_setup):
|
||||
"""Both tracker-nodes handle at least one request each out of ten."""
|
||||
"""Both head workers handle at least one request each out of ten."""
|
||||
gateway_url, tracker_node_a, tracker_node_b = tracker_node_setup
|
||||
for i in range(10):
|
||||
_send_chat_request(gateway_url, f"message {i}")
|
||||
assert tracker_node_a.chat_completion_count >= 1, (
|
||||
f"tracker-node-A received no requests (count={tracker_node_a.chat_completion_count})"
|
||||
f"head-worker-A received no requests (count={tracker_node_a.chat_completion_count})"
|
||||
)
|
||||
assert tracker_node_b.chat_completion_count >= 1, (
|
||||
f"tracker-node-B received no requests (count={tracker_node_b.chat_completion_count})"
|
||||
f"head-worker-B received no requests (count={tracker_node_b.chat_completion_count})"
|
||||
)
|
||||
total = tracker_node_a.chat_completion_count + tracker_node_b.chat_completion_count
|
||||
assert total == 10, f"total requests handled by tracker-nodes: {total}, expected 10"
|
||||
assert total == 10, f"total requests handled by head workers: {total}, expected 10"
|
||||
|
||||
|
||||
def test_tracker_nodes_endpoint_returns_registered_nodes(tracker_node_setup):
|
||||
"""GET /v1/tracker-nodes/<model> on the tracker returns both registered tracker-nodes."""
|
||||
"""GET /v1/tracker-nodes/<model> remains as a legacy alias for head workers."""
|
||||
_, tracker_node_a, tracker_node_b = tracker_node_setup
|
||||
# Find the tracker URL by inspecting the fixture indirectly
|
||||
# We need the tracker URL — use the gateway's tracker_url
|
||||
@@ -181,13 +182,13 @@ def test_tracker_nodes_endpoint_returns_registered_nodes(tracker_node_setup):
|
||||
|
||||
|
||||
def test_load_is_distributed_evenly(tracker_node_setup):
|
||||
"""With 10 requests and round-robin, each tracker-node gets exactly 5."""
|
||||
"""With 10 requests and round-robin, each head worker gets exactly 5."""
|
||||
gateway_url, tracker_node_a, tracker_node_b = tracker_node_setup
|
||||
for i in range(10):
|
||||
_send_chat_request(gateway_url, f"round-robin test {i}")
|
||||
assert tracker_node_a.chat_completion_count == 5, (
|
||||
f"expected 5 requests on tracker-node-A, got {tracker_node_a.chat_completion_count}"
|
||||
f"expected 5 requests on head-worker-A, got {tracker_node_a.chat_completion_count}"
|
||||
)
|
||||
assert tracker_node_b.chat_completion_count == 5, (
|
||||
f"expected 5 requests on tracker-node-B, got {tracker_node_b.chat_completion_count}"
|
||||
f"expected 5 requests on head-worker-B, got {tracker_node_b.chat_completion_count}"
|
||||
)
|
||||
|
||||
45
tests/test_tracker_control_plane.py
Normal file
45
tests/test_tracker_control_plane.py
Normal file
@@ -0,0 +1,45 @@
|
||||
"""Tracker control-plane boundaries."""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
|
||||
|
||||
def test_tracker_startup_does_not_import_or_load_model_backends():
|
||||
"""The public tracker is a router/API endpoint, not an inference worker."""
|
||||
code = textwrap.dedent(
|
||||
"""
|
||||
import builtins
|
||||
import sys
|
||||
|
||||
blocked_roots = {"meshnet_node", "torch", "transformers"}
|
||||
real_import = builtins.__import__
|
||||
|
||||
def guarded_import(name, globals=None, locals=None, fromlist=(), level=0):
|
||||
root = name.split(".", 1)[0]
|
||||
if root in blocked_roots:
|
||||
raise AssertionError(f"tracker imported model backend dependency: {name}")
|
||||
return real_import(name, globals, locals, fromlist, level)
|
||||
|
||||
builtins.__import__ = guarded_import
|
||||
|
||||
from meshnet_tracker.server import TrackerServer
|
||||
|
||||
tracker = TrackerServer()
|
||||
port = tracker.start()
|
||||
assert port > 0
|
||||
tracker.stop()
|
||||
|
||||
imported = blocked_roots & set(sys.modules)
|
||||
assert not imported, f"unexpected model modules imported: {sorted(imported)}"
|
||||
"""
|
||||
)
|
||||
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", code],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
Reference in New Issue
Block a user