DEV Community

Samuel Ochaba
Samuel Ochaba

Posted on AI-assisted

This LLM cost function looks right. Can you spot the bug?

#llm #openai #ai #python

OpenAI's reasoning models bill you for thinking you never see. So it seems careful to add that thinking to your cost tracking, like this:

from decimal import Decimal

# Made-up prices, in USD per million tokens.
INPUT_RATE = Decimal("0.50")
OUTPUT_RATE = Decimal("2.00")


def estimate_cost(usage):
    """Estimated USD cost of one OpenAI Responses API call."""
    reasoning = usage["output_tokens_details"]["reasoning_tokens"]
    billed_output = usage["output_tokens"] + reasoning
    return (usage["input_tokens"] * INPUT_RATE
            + billed_output * OUTPUT_RATE) / 1_000_000
Enter fullscreen mode Exit fullscreen mode

It runs, and it returns a believable number. It's also wrong for every response that used reasoning. Have a guess at why before you read on.

The answer: the reasoning is already in output_tokens

OpenAI's Responses API reports everything the model generates as output_tokens. Its token counting guide says: "Reported output token usage includes all tokens generated by the model, not only the text visible in a response." The reasoning_tokens value under output_tokens_details breaks that number down: it tells you how many of the output tokens were reasoning. So estimate_cost counts the reasoning twice.

You can see it in the example response in OpenAI's reasoning guide:

{
  "input_tokens": 75,
  "input_tokens_details": { "cached_tokens": 0 },
  "output_tokens": 1186,
  "output_tokens_details": { "reasoning_tokens": 1024 },
  "total_tokens": 1261
}
Enter fullscreen mode Exit fullscreen mode

The total is 75 + 1,186 = 1,261, so the 1,024 reasoning tokens are already inside the 1,186. Add them again and your log says 2,285 tokens for a response that reported 1,261. The same guide confirms the billing: "While reasoning tokens are not visible via the API, they still occupy space in the model's context window and are billed as output tokens."

The fix, and what the bug costs

Use output_tokens as it is. Here's the fixed function, with a made-up response to run both versions on:

def estimate_cost_fixed(usage):
    """Same estimate; output_tokens already includes the reasoning tokens."""
    billed_output = usage["output_tokens"]
    return (usage["input_tokens"] * INPUT_RATE
            + billed_output * OUTPUT_RATE) / 1_000_000


# A made-up response in the shape of OpenAI's Responses API usage object.
usage = {
    "input_tokens": 1200,
    "output_tokens": 900,
    "output_tokens_details": {"reasoning_tokens": 700},
}

for label, cost in [("Counted once: ", estimate_cost_fixed(usage)),
                    ("Counted twice:", estimate_cost(usage))]:
    print(f"{label} ${cost:.4f} per request, ${cost * 10_000:.2f} per 10,000")
Enter fullscreen mode Exit fullscreen mode

Put both blocks in one file and run it:

Counted once:  $0.0024 per request, $24.00 per 10,000
Counted twice: $0.0038 per request, $38.00 per 10,000
Enter fullscreen mode Exit fullscreen mode

On this made-up response, the buggy version is 58% too high. Both numbers look perfectly plausible, which is why the mistake is easy to miss.

Gemini goes the other way

Google's Gemini API splits it differently. In the generateContent reference, thoughtsTokenCount is the "Number of tokens of thoughts for thinking models", and totalTokenCount is "prompt + tool-use prompt + thoughts + response candidates". The thinking tokens sit outside candidatesTokenCount, so if you treat candidatesTokenCount as all of the output, they go missing.

No single formula covers both APIs. Adding the reasoning field counts tokens twice on OpenAI, and leaving it out drops tokens on Gemini. Read each provider's definitions before you write the formula.

Cached tokens hide the same trap

Some providers report cached tokens inside the input count. Gemini's promptTokenCount, for example, "includes the number of tokens in the cached content", so adding the cached count on top counts those tokens twice.

What to do today

  1. Store the raw usage object with each request, field names and all.
  2. For each provider you use, write down which fields contain which. A comment next to your cost function is enough.
  3. Test your cost code against a usage record whose answer you already know, like the one above.
  4. Recheck the provider's usage docs when you change models or providers, because the definitions can change.

Did you spot the bug before the answer? And if you call more than one provider, how do you keep their usage fields straight? Tell me in the comments.

Sources

Top comments (4)

Collapse
 
eqvero profile image
Eqvero •

Another variation of this bug hits when teams track streaming responses. In standard completion calls, the usage block arrives cleanly on the final object. In streaming mode, that usage chunk is delivered only in the terminal frame. If an upstream network glitch or client timeout severs the socket midway, the client logs zero output tokens, but the provider billing gateway meters and bills every generated token up to the drop. When reconciling internal pipeline cost logs against the monthly provider invoice, client-side token logs systematically undercount spend by 3 to 7 percent.

The other trap is Anthropic prompt caching accounting. cache_read_input_tokens is discounted at 10 percent of base input rate, while cache_creation_input_tokens incurs a 25 percent surcharge. If your cost estimator treats input_tokens as a flat rate or adds the cache buckets on top without subtracting from the base count, the math drifts in both directions depending on cache hit ratio. Storing the provider exact payload and delegating cost calculations to an isolated pricing table with provider-specific schemas is the only setup that survives monthly API updates.

Collapse
 
samuel_ochaba_eb9c875fa89 profile image
Samuel Ochaba •

Thanks, both of these are real traps.

On streaming, agreed. Usage arrives at the end of the stream. OpenAI puts it in the final event (on Chat Completions, only if you set stream_options: {"include_usage": true}), and its SDK reference warns that an interrupted stream may never deliver it. Anthropic sends the input count in message_start and the cumulative output count in message_delta. If the stream dies first, I'd log the output as unavailable rather than zero, since the provider may still bill what it generated, and flag those requests when reconciling against the provider's own usage data. Is the 3 to 7 percent from your own reconciliations? I'd be curious what traffic produced it.

On Anthropic caching, one correction to the arithmetic. There, input_tokens counts only the uncached tokens after the last cache breakpoint, and the docs give the total as cache_read_input_tokens + cache_creation_input_tokens + input_tokens. So you add the cache buckets on top, each at its own rate, and subtract nothing. That's the opposite of OpenAI and Gemini, where cached tokens sit inside the input count. The rates vary too: cache writes cost 1.25x base input with the 5-minute TTL and 2x with the 1-hour TTL, and reads are 0.1x on most models but lower on some newer ones.

Fully agree on the pricing table: raw payload in, one schema per provider, and a test against a known usage record, so a docs change breaks a test instead of the budget.

Collapse
 
aiapikey profile image
于侃 •

Worth adding the number that makes this more than an arithmetic slip: in the example you quote, reasoning is 1,024 of 1,186 output tokens - 86% of what gets billed is text the user never sees.

So the fix is not only "do not double count". It changes the unit worth reporting. If you quote cost per million output tokens, a reasoning-heavy call reads about 7x cheaper than it is per visible token (1,186 billed vs 162 visible). A dashboard that pairs spend with words returned will flag that as a price change and chase the wrong thing.

Which raises the practical question: do you cap reasoning tokens per call? That cap is the only lever that actually moves the 86%.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

The bug is in the sum, not the rate. As I read the two usage objects, OpenAI's output_tokens already includes the reasoning tokens, so adding them again counts them twice. Gemini reports thinking tokens on their own line, so a function that reads only the candidate count misses them. Does your Gemini branch add the thinking tokens to the output price?