54 lines
8.6 KiB
Markdown
54 lines
8.6 KiB
Markdown
Status: done
|
||
|
||
Scoped 2026-07-06 during alpha-launch-readiness grilling session. High priority, ship-soon for launch — **not** an alpha-release blocker (unlike issue 21): a stale/static price is a revenue/business-model risk, not a safety risk, so the friends + hired-VPS launch may proceed on the current static default while this lands in parallel.
|
||
|
||
# 23 — Dynamic per-model pricing benchmarked against HuggingFace inference rates
|
||
|
||
## What to build
|
||
|
||
Client-facing price per model should track the market: **80% of the cheapest comparable provider rate on HuggingFace's inference marketplace** (`https://huggingface.co/inference/models`), refreshed daily, auto-adjusting so served models stay competitively priced as the market moves. Nodes are unaffected by this loop (per launch design: clients are the only party spending real money; node payouts come from the 90/10 split of whatever price is charged, per ADR-0015/`packages/validator/README.md`).
|
||
|
||
**Current state (confirmed by code read 2026-07-06):** pricing is 100% static today. `DEFAULT_PRICE_PER_1K_TOKENS = 0.02` (`packages/tracker/meshnet_tracker/billing.py:21`) is the fallback nearly every model hits, since `model_presets.json` currently has no `price_per_1k_tokens` key for any preset. `BillingLedger.set_price(model, price)` (`billing.py:67-69`) is the only write path and already exists — no CLI/admin route calls it yet. No external HTTP/market-data integration exists anywhere in the tracker.
|
||
|
||
**Data source:** `https://huggingface.co/inference/models` aggregates multiple providers (novita, together, fireworks-ai, deepinfra, etc.) with per-model, per-provider $/1M input and output token pricing; the "cheapest" badge already identifies the lowest-cost provider per model on the page itself. It supports a GET query param for filtering, e.g. `?search=GLM`. **No confirmed public JSON API was found** during this session's fetch — the page reads as a rendered table. Owner's suggestion: try a plain `requests` + BeautifulSoup scrape first; if the pricing table turns out to be client-rendered (not present in the initial HTML), that's the fallback signal to escalate to a headless-browser fetch (e.g. Playwright) — confirm which is needed during implementation before building the full pipeline around it. Another data source is acceptable if more convenient/stable, owner is not wedded to this specific page.
|
||
|
||
## Deliverables
|
||
|
||
- [x] Live-fetch attempt (requests + BeautifulSoup against the HF page with `?search=<model-family>`, or an equivalent stable source) as the primary path — confirm during implementation whether the pricing table is present in the raw HTML or requires a headless-browser fetch, and note which in the PR
|
||
- [x] Extend `model_presets.json` per model with: `hf_aliases` (curated list of comparable HF model+provider IDs — **human-verified, not auto-discovered**), `hf_verified_match_note` (free text: params count + quantization confirmation, so a human signs off once per alias that it is a fair comparable before it's used for auto-pricing), `hf_last_price_per_1k` (derived from the $/1M rate), `hf_last_updated` (ISO date)
|
||
- [x] Daily refresh job reusing the tracker's existing daemon-thread pattern (`_settlement_loop`/`_deposit_loop` in `server.py`, `threading.Event().wait(interval)` loop) — for each preset with a non-empty `hf_aliases` list, fetch current pricing for those aliases, compute `0.8 × cheapest matched alias price`, call `set_price()`, and update `hf_last_price_per_1k`/`hf_last_updated`
|
||
- [x] Every price change logged (old price, new price, source alias, timestamp) — needed for dispute auditability if a client questions a charge
|
||
- [x] Fallback behavior: empty/missing `hf_aliases`, fetch failure, or no verified match → silently keep the existing static default price. Never error the pricing path, never zero-price a model
|
||
|
||
## Acceptance criteria
|
||
|
||
- [x] At least one model preset has a working end-to-end refresh (alias → live fetch → 80% computed price → `set_price()` called → metadata updated) demonstrated in a test
|
||
- [x] Models without a curated/verified alias continue to use the static default, unaffected by this feature
|
||
- [x] Fetch failures (network error, page structure change, no match found) degrade gracefully — logged, not raised to the request path
|
||
- [x] Price-change log is queryable/inspectable (doesn't need a UI yet — a log line or table row is sufficient for alpha)
|
||
- [x] Note in the runbook/issue on which fetch mechanism (plain HTTP scrape vs. headless browser) was actually required, so the next person doesn't have to rediscover it
|
||
|
||
## Implementation notes (2026-07-06)
|
||
|
||
**Fetch mechanism confirmed: plain HTTP scrape, no headless browser needed.** Live-fetched `https://huggingface.co/inference/models?search=GLM` this session — the pricing table is server-rendered into the initial HTML response (SvelteKit SSR), confirmed by grepping the raw response for `cheapest`/`$`-prefixed price cells before any JS runs. A stdlib `urllib.request` GET + `html.parser.HTMLParser`-based table walk is sufficient; no `requests`/`bs4`/Playwright dependency was added, matching this package's existing zero-new-HTTP-dependency convention (`gossip.py`/`raft.py`/`server.py` all use `urllib.request` only). Each row's most stable extraction anchor turned out to be the `<a href="/<org>/<repo>/?inference_api=true&inference_provider=<provider>">` link, not the display text (which duplicates the repo id at two responsive breakpoints and is easy to mis-parse).
|
||
|
||
**What shipped:** new `packages/tracker/meshnet_tracker/hf_pricing.py` — pure HTML parser (`parse_hf_pricing_table`), alias matching (`cheapest_matching_quote`, supports both `org/repo` and `org/repo::provider` forms so a human can pin a specific provider's deployment), a pure per-preset computation function (`refresh_preset_price`, never raises), and `HfPricingLog` (SQLite-backed change log, same shape as `billing.py`/`calibration.py`). `TrackerServer` gained an opt-in (`enable_hf_pricing=True` / `--enable-hf-pricing`) daily daemon thread (`_hf_pricing_loop`, same `threading.Event().wait(interval)` shape as `_settlement_loop`) and `GET /v1/pricing/hf/history` (admin/validator-gated, mirrors `/v1/calibration/toploc/results`). `model_presets.json`'s `kimi-k2.7` preset now carries the `hf_aliases`/`hf_verified_match_note` schema fields, left as an empty list pending a human sign-off on a genuinely comparable HF listing (params count + quantization) — per this issue's own "human-verified, not auto-discovered" requirement, an agent should not fabricate that sign-off. This also means the shipped default config demonstrates the required "no alias → static price, unaffected" fallback for a real production preset; the alias→live-fetch→80%→set_price() path is demonstrated end-to-end against an injected fetch backend in `tests/test_hf_pricing_dispatch.py` (the `fetch_html=`/`hf_pricing_fetch_html=` injection point mirrors this codebase's `backend=` convention for anything that would otherwise hit the network in tests).
|
||
|
||
**Bug caught and fixed while wiring this in:** `TrackerServer` previously did `dict(DEFAULT_MODEL_PRESETS)` when no explicit `model_presets` was passed — a shallow copy that aliases every preset's inner dict to the shared module-level global. Writing `hf_last_price_per_1k`/`hf_last_updated` in place would have leaked across every other `TrackerServer` instance in the same process (real risk in the test suite, and in any future multi-tracker-in-one-process embedding). Fixed with a `_clone_model_presets()` helper that also shallow-copies each preset dict.
|
||
|
||
**Follow-up for a human (not a completion blocker):** populate real `hf_aliases`/`hf_verified_match_note` entries for production presets once someone has confirmed a genuinely comparable HF-listed deployment (params + quantization) — that activates dynamic pricing for that model on the next refresh tick. Until then every preset safely stays on its static price.
|
||
|
||
Tests: `tests/test_hf_pricing.py` (11 tests: parsing, blended-price math, alias matching incl. provider-scoped aliases, all three fallback paths, log persistence) + `tests/test_hf_pricing_dispatch.py` (5 tests: full TrackerServer end-to-end refresh, unaffected-without-alias, history auth gating, history content, history model filter). Full suite (`pytest tests/ -q -k "not integration"`): 346 passed, 2 skipped.
|
||
|
||
## ADR links
|
||
|
||
- [ADR-0015](../../docs/adr/0015-usdt-custodial-settlement.md) — settlement/pricing this touches (90/10 split, per-model pricing)
|
||
|
||
## Blocked by
|
||
|
||
None — independent of the alpha-hardening trust-boundary work; touches `billing.py`/`server.py` pricing paths only.
|
||
|
||
## Blocks
|
||
|
||
None — ship-soon for launch quality, not a release gate (see status note above).
|