neuron-tai/CONTEXT.md

Distributed Inference Network

A volunteer GPU network where nodes independently load model shards, a tracker routes inference through optimal node chains, and contributors earn tokens for serving compute.

Language

Nodes & compute

Node: A volunteer machine that runs the node client, holds one or more shards on disk, and serves inference requests for those shards. Avoid: worker, peer, miner, server

Shard: A contiguous range of transformer layers from a model that a node loads and serves. Shards are the unit of storage, assignment, and reward. Avoid: partition, slice, chunk, segment

Shard Swarm: The P2P group of nodes that collectively seed and download a specific shard. One swarm exists per shard. Avoid: torrent, cluster, pool

Inference Route: An ordered sequence of nodes whose shards together cover all layers of a model. The tracker selects the optimal route per request. Avoid: pipeline, chain, path

Route Session: An active inference request bound to one Inference Route and one stable session id while the request is being served. Avoid: conversation, job, token stream

Activation Seam: The boundary between two adjacent shard executions where hidden states pass from one node to the next. Avoid: handoff, layer gap, boundary hop

Hot KV State: The live attention/cache state a node holds for its own shard during a Route Session. Avoid: centralized KV cache, global cache, remote cache

Prefix Snapshot: A persisted copy of reusable Route Session state for a prompt prefix, used for reuse, retry, or failover. Avoid: hot cache, CDN cache, active KV

Model Artifact: A versioned model file or tokenizer file that nodes download, verify, and keep locally to serve a Model Preset. Avoid: model blob, weights dump, asset

Artifact Manifest: The canonical record that identifies the Model Artifacts, their integrity checks, and the model parts they support. Avoid: torrent file, metadata JSON, download list

Gateway: The network entry point that accepts client requests (OpenAI-compatible HTTP), selects an inference route from the tracker, and streams results and progress to the client when possible. Avoid: proxy, relay, orchestrator, primary

Generation Telemetry: Realtime progress information for an active Route Session, including phase, generated token count, and tokens-per-second speed. Avoid: logs, debug output

Tracker

Tracker: The coordinator service that maintains the node registry, scores nodes by throughput/latency, and assigns inference routes. Runs as a centralized service with a P2P gossip fallback. Avoid: coordinator, scheduler, director

Tracker Node: A node that serves at least the first-layer shard (layers[0..k]) for a model and acts as the inference entry point for that model. Tracker nodes own the tokenizer and embed_tokens, receive client requests directly, select the onward route from the coverage map, and stream results and progress when possible. Any node advertising a new model to the network becomes its tracker node. Avoid: primary node, master node, gateway node

Coverage Map: The tracker's per-model mapping of layer ranges to node counts: [(start_layer, end_layer, node_count), ...]. A layer range with node_count=0 is a coverage gap — the model is unroutable until the gap is filled. Coverage-first bin-packing fills all gaps before adding redundancy. Avoid: shard map, assignment table, coverage report

Rebalance Directive: A LOAD_SHARD or DROP_SHARD instruction the tracker issues to a node when the coverage map changes (node joins, node leaves, or load-balance reoptimization). Delivered as part of the node's heartbeat response. Avoid: rebalance command, shard instruction, migration order

Node Score: A throughput/latency rating the tracker maintains per node, used for route selection. Updated continuously from inference telemetry. Avoid: reputation, rating, rank

Payments & fraud

Stake: Collateral a node stands to lose for fraud. In the current design the node's Pending Balance serves as stake — no upfront deposit is required. An optional USDT/TAI deposit may return later for routing priority. Avoid: deposit, bond, escrow

Treasury: The single project-owned Solana wallet that custodially holds client deposits, pays node payouts, and accumulates the Protocol Cut. Its keypair is loaded only on settlement-capable trackers. Avoid: escrow, vault, hot wallet

Pending Balance: A node's accrued, not-yet-paid USDT earnings on the tracker ledger. Doubles as the node's fraud collateral: it is forfeited in full when a validator catches a divergent output. Avoid: unpaid rewards, accrual, balance due

Settlement Period: The dynamic interval driving on-chain payouts: a node is paid when its Pending Balance exceeds the Payout Threshold or the period elapses, whichever comes first. Short in development (seconds), long in production (daily), configurable to grow with volume. Avoid: epoch, payout cycle, billing cycle

Payout Threshold: The minimum Pending Balance that triggers an immediate payout before the Settlement Period elapses. Includes a dust floor so payouts are never smaller than they are worth. Avoid: minimum payout, dust limit

Protocol Cut: The 10% of inference fees retained by the project for infrastructure; the remaining 90% goes to the nodes that served the request. Accumulates in the Treasury as the future TAI liquidity reserve. Avoid: spread, commission, house fee

Deposit Watcher: The tracker component that observes the Treasury's on-chain USDT deposits and credits the sending client's API-key ledger balance. Avoid: payment listener, chain scanner

Mock USDT: The self-created 6-decimal SPL mint that stands in for USDT on devnet, where real USDT does not exist. The mint address is configuration, so mainnet cutover is a config change. Avoid: test token, fake USDT, devnet dollar

Tax: The share of caller payments distributed to compute nodes as rewards. Taxes are weighted by completed work and historical node speed so faster, larger nodes earn proportionally more. Avoid: fee, toll, commission

Caller Credit: Free starting balance granted to a new caller/API key so they can try the network before topping up. Avoid: signup bonus, faucet, airdrop

Free Compute Job: Work a compute node performs without earning immediate rewards, usually during probation or bootstrap phases. Avoid: unpaid labor, warmup request

Slash: The penalty for a proven fraud incident: the node's entire Pending Balance is forfeited to the Treasury and a Strike is recorded. Avoid: penalize, burn, fine, forfeit

Strike: A fraud incident recorded on-chain against a node. Enough strikes result in a ban. Avoid: infraction, violation, flag

Ban: Permanent exclusion of a wallet from the network after exceeding the strike threshold. Recorded on-chain. Avoid: blacklist, block, suspension

Probationary Period: The first N jobs a new wallet must complete without earning, to raise the cost of re-entering after a ban. Avoid: trial period, warmup, grace period

Token: TAI, our native Solana SPL token. Deferred (ADR-0015): nodes are currently paid directly in USDT; TAI returns as the reward/upside layer once volume exists, funded by the accumulated Protocol Cut. Clients never need to hold it. Avoid: coin, reward token, native token

Contract Boundary: The Python interface in packages/contracts that represents registry, payment, and settlement behavior. During the prototype it is implemented by deterministic local wrappers; later the same boundary is backed by real Solana programs. Avoid: mock contract, fake chain, temporary hack

Validator: A trusted node (or the tracker itself) that re-runs a sample of inference requests to detect fraud. Avoid: auditor, checker, referee

Validation Event: A completed inference record that contains enough information for a validator to decide whether to sample and re-run the request: session id, model preset, messages, inference route, node wallets, and observed output. Avoid: audit log, trace, receipt

Slash Proof: The record submitted by a validator when a sampled re-run diverges from the observed output beyond tolerance. In the prototype this is deterministic local contract state; later it maps to an on-chain proof transaction. Avoid: accusation, report, claim

Client-facing

Client: Any application or user that sends inference requests to the gateway. Prepays USDT into the Treasury; each request is metered against the resulting ledger balance at a per-1K-tokens price set per model. Avoid: user, caller, consumer

Model Preset: A named, versioned model available on the network (e.g. llama-3-70b). The tracker knows which nodes hold which shards for each preset. Avoid: model, checkpoint, version