61 lines
3.0 KiB
Markdown
61 lines
3.0 KiB
Markdown
# US-045 — Dual-rate billing: separate input and output token prices
|
||
|
||
Status: in progress
|
||
Priority: High (billing correctness before friends test; providers all price this way)
|
||
Stage: Designed
|
||
|
||
## Context
|
||
|
||
Today the ledger has one `price_per_1k_tokens` per model, and the two proxy
|
||
paths don't even agree on what they count:
|
||
|
||
- **Non-streaming** bills `usage.total_tokens` (prompt + completion) at the
|
||
blended rate (`_billable_non_stream_tokens`).
|
||
- **Streaming** bills `min(observed output deltas, reported total)` — output
|
||
only in practice (`_billable_stream_tokens`).
|
||
- The HF pricing refresher (issue 23) averages a provider's input/output
|
||
rates 50/50 (`blended_price_per_1k_tokens`), which misprices asymmetric
|
||
models — e.g. Qwen3.6-35B-A3B on deepinfra is $0.15/1M in, $0.95/1M out.
|
||
|
||
Decision (user, 2026-07-06): charge **both** input and output tokens, at
|
||
**two separate rates**, same as other providers.
|
||
|
||
## Design
|
||
|
||
1. **`BillingLedger`** stores `{model: (input_per_1k, output_per_1k)}`.
|
||
- `set_prices(model, input_per_1k, output_per_1k)` (new);
|
||
`set_price(model, p)` keeps working and sets both.
|
||
- `prices_for(model) -> (input, output)` (new); `price_for(model)` returns
|
||
the blended average for back-compat (estimators/logs).
|
||
- `charge_request(...)` gains keyword `input_tokens`/`output_tokens`; when
|
||
provided, `cost = in_rate·in/1k + out_rate·out/1k` and the event records
|
||
the split. Without them, legacy behavior (blended × total) — old events
|
||
and gossip replicas replay unchanged (`cost` stays the applied field).
|
||
2. **Token counting** (`server.py`):
|
||
- Non-stream: prefer `usage.prompt_tokens`/`completion_tokens`; fall back
|
||
to content estimates (`_estimate_prompt_tokens`, observed completion),
|
||
capped by `max_tokens` bounds as today.
|
||
- Stream (direct + relay): output = observed deltas as today; input =
|
||
`usage.prompt_tokens` when a usage chunk appears, else the prompt
|
||
estimate from the request body. `_stream_line_tokens` returns the parsed
|
||
usage triple instead of just the total.
|
||
3. **Presets**: `input_price_per_1k_tokens` / `output_price_per_1k_tokens`
|
||
(dual keys win; `price_per_1k_tokens` alone still means "both rates").
|
||
Qwen3.6-35B-A3B: input 0.00012, output 0.00076 (80% of deepinfra).
|
||
4. **HF refresher**: applies 80% of each side separately via `set_prices`
|
||
(all alias keys); change log keeps recording the blended pair for history
|
||
continuity.
|
||
5. **Spend cap** (`--max-charge-per-request`): estimate =
|
||
`in_rate·prompt_estimate + out_rate·completion_limit`.
|
||
|
||
## Acceptance criteria
|
||
|
||
- Streamed and non-streamed requests for the same exchange bill the same
|
||
split (input charged in both)
|
||
- A model with asymmetric provider rates bills input and output differently;
|
||
`usage_for` / billing events expose the split
|
||
- Old persisted billing events replay byte-identically (balances unchanged)
|
||
- HF refresh sets both rates from the marketplace row, not the average
|
||
- Spend cap uses the dual rates
|
||
- `python -m pytest` passes from repo root
|