neuron-tai/.scratch/alpha-hardening/issues/23-dynamic-hf-pricing_completed.md

Status: done

Scoped 2026-07-06 during alpha-launch-readiness grilling session. High priority, ship-soon for launch — not an alpha-release blocker (unlike issue 21): a stale/static price is a revenue/business-model risk, not a safety risk, so the friends + hired-VPS launch may proceed on the current static default while this lands in parallel.

23 — Dynamic per-model pricing benchmarked against HuggingFace inference rates

What to build

Client-facing price per model should track the market: 80% of the cheapest comparable provider rate on HuggingFace's inference marketplace (https://huggingface.co/inference/models), refreshed daily, auto-adjusting so served models stay competitively priced as the market moves. Nodes are unaffected by this loop (per launch design: clients are the only party spending real money; node payouts come from the 90/10 split of whatever price is charged, per ADR-0015/packages/validator/README.md).

Current state (confirmed by code read 2026-07-06): pricing is 100% static today. DEFAULT_PRICE_PER_1K_TOKENS = 0.02 (packages/tracker/meshnet_tracker/billing.py:21) is the fallback nearly every model hits, since model_presets.json currently has no price_per_1k_tokens key for any preset. BillingLedger.set_price(model, price) (billing.py:67-69) is the only write path and already exists — no CLI/admin route calls it yet. No external HTTP/market-data integration exists anywhere in the tracker.

Data source: https://huggingface.co/inference/models aggregates multiple providers (novita, together, fireworks-ai, deepinfra, etc.) with per-model, per-provider $/1M input and output token pricing; the "cheapest" badge already identifies the lowest-cost provider per model on the page itself. It supports a GET query param for filtering, e.g. ?search=GLM. No confirmed public JSON API was found during this session's fetch — the page reads as a rendered table. Owner's suggestion: try a plain requests + BeautifulSoup scrape first; if the pricing table turns out to be client-rendered (not present in the initial HTML), that's the fallback signal to escalate to a headless-browser fetch (e.g. Playwright) — confirm which is needed during implementation before building the full pipeline around it. Another data source is acceptable if more convenient/stable, owner is not wedded to this specific page.

Deliverables

• Live-fetch attempt (requests + BeautifulSoup against the HF page with ?search=<model-family>, or an equivalent stable source) as the primary path — confirm during implementation whether the pricing table is present in the raw HTML or requires a headless-browser fetch, and note which in the PR
• Extend model_presets.json per model with: hf_aliases (curated list of comparable HF model+provider IDs — human-verified, not auto-discovered), hf_verified_match_note (free text: params count + quantization confirmation, so a human signs off once per alias that it is a fair comparable before it's used for auto-pricing), hf_last_price_per_1k (derived from the $/1M rate), hf_last_updated (ISO date)
• Daily refresh job reusing the tracker's existing daemon-thread pattern (_settlement_loop/_deposit_loop in server.py, threading.Event().wait(interval) loop) — for each preset with a non-empty hf_aliases list, fetch current pricing for those aliases, compute 0.8 × cheapest matched alias price, call set_price(), and update hf_last_price_per_1k/hf_last_updated
• Every price change logged (old price, new price, source alias, timestamp) — needed for dispute auditability if a client questions a charge
• Fallback behavior: empty/missing hf_aliases, fetch failure, or no verified match → silently keep the existing static default price. Never error the pricing path, never zero-price a model

Acceptance criteria

• At least one model preset has a working end-to-end refresh (alias → live fetch → 80% computed price → set_price() called → metadata updated) demonstrated in a test
• Models without a curated/verified alias continue to use the static default, unaffected by this feature
• Fetch failures (network error, page structure change, no match found) degrade gracefully — logged, not raised to the request path
• Price-change log is queryable/inspectable (doesn't need a UI yet — a log line or table row is sufficient for alpha)
• Note in the runbook/issue on which fetch mechanism (plain HTTP scrape vs. headless browser) was actually required, so the next person doesn't have to rediscover it

Implementation notes (2026-07-06)

Fetch mechanism confirmed: plain HTTP scrape, no headless browser needed. Live-fetched https://huggingface.co/inference/models?search=GLM this session — the pricing table is server-rendered into the initial HTML response (SvelteKit SSR), confirmed by grepping the raw response for cheapest/$-prefixed price cells before any JS runs. A stdlib urllib.request GET + html.parser.HTMLParser-based table walk is sufficient; no requests/bs4/Playwright dependency was added, matching this package's existing zero-new-HTTP-dependency convention (gossip.py/raft.py/server.py all use urllib.request only). Each row's most stable extraction anchor turned out to be the <a href="/<org>/<repo>/?inference_api=true&inference_provider=<provider>"> link, not the display text (which duplicates the repo id at two responsive breakpoints and is easy to mis-parse).

What shipped: new packages/tracker/meshnet_tracker/hf_pricing.py — pure HTML parser (parse_hf_pricing_table), alias matching (cheapest_matching_quote, supports both org/repo and org/repo::provider forms so a human can pin a specific provider's deployment), a pure per-preset computation function (refresh_preset_price, never raises), and HfPricingLog (SQLite-backed change log, same shape as billing.py/calibration.py). TrackerServer gained an opt-in (enable_hf_pricing=True / --enable-hf-pricing) daily daemon thread (_hf_pricing_loop, same threading.Event().wait(interval) shape as _settlement_loop) and GET /v1/pricing/hf/history (admin/validator-gated, mirrors /v1/calibration/toploc/results). model_presets.json's kimi-k2.7 preset now carries the hf_aliases/hf_verified_match_note schema fields, left as an empty list pending a human sign-off on a genuinely comparable HF listing (params count + quantization) — per this issue's own "human-verified, not auto-discovered" requirement, an agent should not fabricate that sign-off. This also means the shipped default config demonstrates the required "no alias → static price, unaffected" fallback for a real production preset; the alias→live-fetch→80%→set_price() path is demonstrated end-to-end against an injected fetch backend in tests/test_hf_pricing_dispatch.py (the fetch_html=/hf_pricing_fetch_html= injection point mirrors this codebase's backend= convention for anything that would otherwise hit the network in tests).

Bug caught and fixed while wiring this in: TrackerServer previously did dict(DEFAULT_MODEL_PRESETS) when no explicit model_presets was passed — a shallow copy that aliases every preset's inner dict to the shared module-level global. Writing hf_last_price_per_1k/hf_last_updated in place would have leaked across every other TrackerServer instance in the same process (real risk in the test suite, and in any future multi-tracker-in-one-process embedding). Fixed with a _clone_model_presets() helper that also shallow-copies each preset dict.

Follow-up for a human (not a completion blocker): populate real hf_aliases/hf_verified_match_note entries for production presets once someone has confirmed a genuinely comparable HF-listed deployment (params + quantization) — that activates dynamic pricing for that model on the next refresh tick. Until then every preset safely stays on its static price.

Tests: tests/test_hf_pricing.py (11 tests: parsing, blended-price math, alias matching incl. provider-scoped aliases, all three fallback paths, log persistence) + tests/test_hf_pricing_dispatch.py (5 tests: full TrackerServer end-to-end refresh, unaffected-without-alias, history auth gating, history content, history model filter). Full suite (pytest tests/ -q -k "not integration"): 346 passed, 2 skipped.

ADR links

• ADR-0015 — settlement/pricing this touches (90/10 split, per-model pricing)

Blocked by

None — independent of the alpha-hardening trust-boundary work; touches billing.py/server.py pricing paths only.

Blocks

None — ship-soon for launch quality, not a release gate (see status note above).