neuron-tai/docs/issues/45-dual-rate-billing.md

US-045 — Dual-rate billing: separate input and output token prices

Status: in progress Priority: High (billing correctness before friends test; providers all price this way) Stage: Designed

Context

Today the ledger has one price_per_1k_tokens per model, and the two proxy paths don't even agree on what they count:

• Non-streaming bills usage.total_tokens (prompt + completion) at the blended rate (_billable_non_stream_tokens).
• Streaming bills min(observed output deltas, reported total) — output only in practice (_billable_stream_tokens).
• The HF pricing refresher (issue 23) averages a provider's input/output rates 50/50 (blended_price_per_1k_tokens), which misprices asymmetric models — e.g. Qwen3.6-35B-A3B on deepinfra is $0.15/1M in, $0.95/1M out.

Decision (user, 2026-07-06): charge both input and output tokens, at two separate rates, same as other providers.

Design

1. BillingLedger stores {model: (input_per_1k, output_per_1k)}.
   • set_prices(model, input_per_1k, output_per_1k) (new); set_price(model, p) keeps working and sets both.
   • prices_for(model) -> (input, output) (new); price_for(model) returns the blended average for back-compat (estimators/logs).
   • charge_request(...) gains keyword input_tokens/output_tokens; when provided, cost = in_rate·in/1k + out_rate·out/1k and the event records the split. Without them, legacy behavior (blended × total) — old events and gossip replicas replay unchanged (cost stays the applied field).
2. Token counting (server.py):
   • Non-stream: prefer usage.prompt_tokens/completion_tokens; fall back to content estimates (_estimate_prompt_tokens, observed completion), capped by max_tokens bounds as today.
   • Stream (direct + relay): output = observed deltas as today; input = usage.prompt_tokens when a usage chunk appears, else the prompt estimate from the request body. _stream_line_tokens returns the parsed usage triple instead of just the total.
3. Presets: input_price_per_1k_tokens / output_price_per_1k_tokens (dual keys win; price_per_1k_tokens alone still means "both rates"). Qwen3.6-35B-A3B: input 0.00012, output 0.00076 (80% of deepinfra).
4. HF refresher: applies 80% of each side separately via set_prices (all alias keys); change log keeps recording the blended pair for history continuity.
5. Spend cap (--max-charge-per-request): estimate = in_rate·prompt_estimate + out_rate·completion_limit.

Acceptance criteria

• Streamed and non-streamed requests for the same exchange bill the same split (input charged in both)
• A model with asymmetric provider rates bills input and output differently; usage_for / billing events expose the split
• Old persisted billing events replay byte-identically (balances unchanged)
• HF refresh sets both rates from the marketplace row, not the average
• Spend cap uses the dual rates
• python -m pytest passes from repo root