3 ms·

I've always wondered about that. LLM providers could easily decimate the cost of inference if they got the models to just stop emitting so much hot air. I don't

I've always wondered about that. LLM providers could easily decimate the cost of inference if they got the models to just stop emitting so much hot air. I don't understand why OpenAI wants to pay 3x the cost to generate a response when two thirds of those tokens are meaningless noise.

sambaumann 8mo ago
because for API users they get to charge for 3x the tokens for the same requests
CamperBob2 8mo ago
The 'hot air' is apparently more important than it appears at first, because those initial tokens are the substrate that the transformer uses for computation. Karpathy talks a little about this in some of his introductory lectures on YouTube.
- Terr_ 8mo ago
  Related are "reasoning" models, where there's a stream of "hot air" that's not being shown to the end-user. I analogize it as a film noir script document: The hardboiled detective character has unspoken text, and if you ask some agent to "make this document longer", there's extra continuity to work with.
ben_w 8mo ago
Because they don't yet know how to "just stop emitting so much hot air" without also removing their ability to do anything like "thinking" (or whatever you want to call the transcript mode), which is hard because knowing which tokens are hot air is the hard problem itself. They basically only started doing this because someone noticed you got better performance from the early models by straight up writing "think step by step" in your prompt.
- Terr_ 8mo ago
  IMO it supports the framing that it's all just a "make document longer" problem, where our human brains are primed for a kind of illusion, where we perceive/infer a mind because, traditionally, that's been the only thing that makes such fitting language.
  - ben_w 8mo ago
    To an extent. Even though they're clearly improving*, they also definitely look better than they actually are. * this time last year they couldn't write compilable source code for a compiler for a toy language, I know because I tried
    - hansvm 8mo ago
      This time last year they could definitely write compilable source code for a compiler for a toy language if you bootstrapped the implementation. If you, e.g., had it write an interpreter and use the source code as a comptime argument (I used Zig as the backend -- Futamura transforms and all that), everything worked swimmingly. I wasn't even using agents; ChatGPT with a big context window was sufficient to write most of the compiler for some language for embedded tensor shenanigans I was hacking on.
      - ben_w 8mo ago
        Used to need the "if", now SOTA doesn't. SOTA today has a different set of caveats, of course.
- mikepurvis 8mo ago
  I would guess that by the time a response is being emitted, 90% of the actual work is done. The response has been thought out, planned, drafted, the individual elements researched and placed. It would actually take more work to condense that long response into a terse one, particularly if the condensing was user specific, like "based on what you know about me from our interactions, reduce your response to the 200 words most relevant to my immediate needs, and wait for me to ask for more details if I require them."
  - tbossanova 8mo ago
    “Sorry for the long letter, I would have written a shorter one but I didn’t have the time.”
observationist 8mo ago
This is an active research topic - two papers on this have come out over the last few days, one cutting half of the tokens and actually boosting performance overall. I'd hazard a guess that they could get another 40% reduction, if they can come up with better reasoning scaffolding. Each advance over the last 4 years, from RLHF to o1 reasoning to multi-agent, multi-cluster parallelized CoT, has resulted in a new engineering scope, and the low hanging fruit in each place gets explored over the course of 8-12 months. We still probably have a year or 2 of low hanging fruit and hacking on everything htat makes up current frontier models. It'll be interesting if there's any architectural upsets in the near future. All the money and time invested into transformers could get ditched in favor of some other new king of the hill(climbers). https://arxiv.org/abs/2602.02828 https://arxiv.org/abs/2602.02828 https://arxiv.org/abs/2503.16419 https://arxiv.org/abs/2503.16419 https://arxiv.org/abs/2508.05988 https://arxiv.org/abs/2508.05988 Current LLMs are going to get really sleek and highly tuned, but I have a feeling they're going to be relegated to a component status, or maybe even abandoned when the next best thing comes along and blows the performance away.
mattclarkdotnet 8mo ago
Because inference costs are negligible compared to training costs
zahlman 8mo ago
My assumption has been that emitting those tokens is part of the inference, analogous to humans "thinking out loud".
- abustamam 8mo ago
  You're absolutely right!
tempestn 8mo ago
The one that always gets me is how they're insistent on giving 17-step instructions to any given problem, even when each step is conditional and requires feedback. So in practice you need to do the first step, then report the results, and have it adapt, at which point it will repeat steps 2-16. IME it's almost impossible to reliably prevent it from doing this, however you ask, at least without severely degrading the value of the response.
mitthrowaway2 8mo ago
I can only imagine that someone's KPIs are tied to increasing rather than decreasing token usage.
ferris-booler 8mo ago
An LLM uses constant compute per output token (one forward pass through the model), so the only computational mechanism to increase 'thinking' quantity is to emit more tokens. Hence why reasoning models produce many intermediary tokens that are not shown to the user, as mentioned in other replies here. This is also why the accuracy of "reasoning traces" is hotly debated; the words themselves may not matter so much as simply providing a compute scratch space. Alternative approaches like "reasoning in the latent space" are active research areas, but have not yet found major success.

sambaumann 8mo ago

because for API users they get to charge for 3x the tokens for the same requests

CamperBob2 8mo ago

The 'hot air' is apparently more important than it appears at first, because those initial tokens are the substrate that the transformer uses for computation. Karpathy talks a little about this in some of his introductory lectures on YouTube.

Terr_ 8mo ago
Related are "reasoning" models, where there's a stream of "hot air" that's not being shown to the end-user. I analogize it as a film noir script document: The hardboiled detective character has unspoken text, and if you ask some agent to "make this document longer", there's extra continuity to work with.

ben_w 8mo ago

Because they don't yet know how to "just stop emitting so much hot air" without also removing their ability to do anything like "thinking" (or whatever you want to call the transcript mode), which is hard because knowing which tokens are hot air is the hard problem itself. They basically only started doing this because someone noticed you got better performance from the early models by straight up writing "think step by step" in your prompt.

Terr_ 8mo ago
IMO it supports the framing that it's all just a "make document longer" problem, where our human brains are primed for a kind of illusion, where we perceive/infer a mind because, traditionally, that's been the only thing that makes such fitting language.
- ben_w 8mo ago
  To an extent. Even though they're clearly improving*, they also definitely look better than they actually are. * this time last year they couldn't write compilable source code for a compiler for a toy language, I know because I tried
  - hansvm 8mo ago
    This time last year they could definitely write compilable source code for a compiler for a toy language if you bootstrapped the implementation. If you, e.g., had it write an interpreter and use the source code as a comptime argument (I used Zig as the backend -- Futamura transforms and all that), everything worked swimmingly. I wasn't even using agents; ChatGPT with a big context window was sufficient to write most of the compiler for some language for embedded tensor shenanigans I was hacking on.
    - ben_w 8mo ago
      Used to need the "if", now SOTA doesn't. SOTA today has a different set of caveats, of course.
mikepurvis 8mo ago
I would guess that by the time a response is being emitted, 90% of the actual work is done. The response has been thought out, planned, drafted, the individual elements researched and placed. It would actually take more work to condense that long response into a terse one, particularly if the condensing was user specific, like "based on what you know about me from our interactions, reduce your response to the 200 words most relevant to my immediate needs, and wait for me to ask for more details if I require them."
- tbossanova 8mo ago
  “Sorry for the long letter, I would have written a shorter one but I didn’t have the time.”

observationist 8mo ago

This is an active research topic - two papers on this have come out over the last few days, one cutting half of the tokens and actually boosting performance overall. I'd hazard a guess that they could get another 40% reduction, if they can come up with better reasoning scaffolding. Each advance over the last 4 years, from RLHF to o1 reasoning to multi-agent, multi-cluster parallelized CoT, has resulted in a new engineering scope, and the low hanging fruit in each place gets explored over the course of 8-12 months. We still probably have a year or 2 of low hanging fruit and hacking on everything htat makes up current frontier models. It'll be interesting if there's any architectural upsets in the near future. All the money and time invested into transformers could get ditched in favor of some other new king of the hill(climbers). https://arxiv.org/abs/2602.02828 https://arxiv.org/abs/2602.02828 https://arxiv.org/abs/2503.16419 https://arxiv.org/abs/2503.16419 https://arxiv.org/abs/2508.05988 https://arxiv.org/abs/2508.05988 Current LLMs are going to get really sleek and highly tuned, but I have a feeling they're going to be relegated to a component status, or maybe even abandoned when the next best thing comes along and blows the performance away.

mattclarkdotnet 8mo ago

Because inference costs are negligible compared to training costs

zahlman 8mo ago

My assumption has been that emitting those tokens is part of the inference, analogous to humans "thinking out loud".

abustamam 8mo ago
You're absolutely right!

tempestn 8mo ago

The one that always gets me is how they're insistent on giving 17-step instructions to any given problem, even when each step is conditional and requires feedback. So in practice you need to do the first step, then report the results, and have it adapt, at which point it will repeat steps 2-16. IME it's almost impossible to reliably prevent it from doing this, however you ask, at least without severely degrading the value of the response.

mitthrowaway2 8mo ago

I can only imagine that someone's KPIs are tied to increasing rather than decreasing token usage.

ferris-booler 8mo ago

An LLM uses constant compute per output token (one forward pass through the model), so the only computational mechanism to increase 'thinking' quantity is to emit more tokens. Hence why reasoning models produce many intermediary tokens that are not shown to the user, as mentioned in other replies here. This is also why the accuracy of "reasoning traces" is hotly debated; the words themselves may not matter so much as simply providing a compute scratch space. Alternative approaches like "reasoning in the latent space" are active research areas, but have not yet found major success.