4 ms·

I guess gigawatts is how we roughly measure computing capacity at the datacenter scale? Also saw something similar here: > Costs and pricing are expressed per

I guess gigawatts is how we roughly measure computing capacity at the datacenter scale? Also saw something similar here: > Costs and pricing are expressed per “token”, but the published data immediately seems to admit that this is a bad choice of unit because it costs a lot more to output a token than input one. It seems to me that the actual marginal quantity being produced and consumed is “processing power”, which is apparently measured in gigawatt hours these days. In any case, I think more than anything this vindicates my original decision not to get too precise. [...] https://backofmind.substack.com/p/new-new-rules-for-the-new-new-economy https://backofmind.substack.com/p/new-new-rules-for-the-new-... Is it priced that way, though? I assume next-gen TPU's will be more efficient?

brokencode 7mo ago
Gigawatts seems like more a statement of the power supply and dissipation of the actual facility. I’m assuming you can cram more chips in there if you have more efficient chips to make use of spare capacity? Trying to measure the actual compute is a moving target since you’d be upgrading things over time, whereas the power aspects are probably more fixed by fire code, building size, and utilities.
- delichon 7mo ago
  Measuring data centers in watts is like measuring cars in horsepower. Power isn't a direct measure of performance, but of the primary constraint on performance. When in doubt choose the thermodynamic perspective.
  - pepperoni_pizza 7mo ago
    Gigawatts are units of power, gigawatthours are units of energy. The equivalent of cars would be pricing by how much gas you burned, not horsepower.
    - delichon 7mo ago
      1 horsepower = 745.7 watts
      - pepperoni_pizza 7mo ago
        Yes, and that is both units of power, not energy.
- stingraycharles 7mo ago
  I mean a single nuclear reactor delivers around 1GW, so if a single datacenter consumes multiple of those, it gives a reasonably accurate idea of the scale.
- deleted 7mo ago
  [deleted]
- franktankbank 7mo ago
  This conversation is confusing because OP didn't use the same units as the person in the quote.
nomel 7mo ago
> but the published data immediately seems to admit that this is a bad choice of unit because it costs a lot more to output a token than input one And, that's silly, because API pricing is more expensive for output than input tokens, 5x so for Anthropic [1], and 6x so for OpenAI! [1] https://platform.claude.com/docs/en/about-claude/pricing https://platform.claude.com/docs/en/about-claude/pricing [2] https://openai.com/api/pricing https://openai.com/api/pricing
- AlphaSite 7mo ago
  I think for the same model wall time is probably a more intuitive metric; at the end of the day what you’re doing is renting GPU time slices. Large outputs dominate compute time so are more expensive. IMO input and output token counts are actually still a bad metric since they linearise non linear cost increases and I suspect we’ll see another change in the future where they bucket by context length. XL output contexts may be 20x more expensive instead of 10x.
  - nsomaru 7mo ago
    They already bucket when context goes above 200k
    - refulgentis 7mo ago
      No longer
  - nomel 7mo ago
    As a customer, it's nice that I can quantize and count the units of cost in an understandable way. For Anthropic, as a business bleeding money, it's probably nice to have value-based pricing, for the tokens, so innovation (like computation efficiency improvements) can result in some extra margin. If they exposed the more direct computation cost, they could never financially benefit from any improved efficiency, including faster hardware!
  - yencabulator 7mo ago
    > I think for the same model wall time is probably a more intuitive metric; at the end of the day what you’re doing is renting GPU time slices This is a bit too much of a simplification. The LLM provider batches multiple customer requests into one GPU/TPU pass over the weights, with minimal latency increase. The LLM provider may in fact be renting GPUs by the second, but the end user isn't. We the end users are essentially timesharing a pool of GPUs without any dedicated "1 vGPU" style resource allocation. In such a setting, charging by "GPU tick" sounds valid, and the various categories of token costs are an approximation of cost+margin.
twoodfin 7mo ago
That these data centers can turn electricity + a little bit of fairly simple software directly into consumer and business value is pretty much the whole story. Compare what you need to add to AWS EC2 to get the same result, above and beyond the electricity.
- zozbot234 7mo ago
  That's a convenient story, but most consumers' and businesses' use of AI is light enough that they could easily run local models on their existing silicon. Resorting to proprietary AI running in the datacenter would only add a tiny fraction of incremental value over that, and at a significant cost.
  - twoodfin 7mo ago
    Sure but where the puck is going is long-running reasoning agents where local models are (for the moment) significantly constrained relative to a Claude Opus 4.6.
  - astral_drama 7mo ago
    I'm looking forward to running a Gemma 4 turboquant on my 24GB GPU. The perf looks impressive for how compact it is. I often get a 10x more cost effective run processing on my local hardware. Still reaching for frontier models for coding, but find the hosted models on open router good enough for simple work. Feels like we are jumping to warp on flops. My cores are throttled and the fiber is lit.
jillesvangurp 7mo ago
It's not really a stable measure of compute, but it's a good indication of burn rate as energy cost is something we closely track in economies and it actually dominates a lot of the cost of operating data centers. At least short term. Over time we'll get more tokens per energy unit and less dollars for the hardware needed per energy unit. Tokens currently is too abstract for a lot of people. They have no concept of the relation ship of numbers of tokens per time unit and cost. Long term there's going to be a big shift from op-ex to cap-ex for energy usage as we shift from burning methane and coal to using renewables with storage.
- amelius 7mo ago
  We need a Moore's law for tokens, and energy.

brokencode 7mo ago

Gigawatts seems like more a statement of the power supply and dissipation of the actual facility. I’m assuming you can cram more chips in there if you have more efficient chips to make use of spare capacity? Trying to measure the actual compute is a moving target since you’d be upgrading things over time, whereas the power aspects are probably more fixed by fire code, building size, and utilities.

delichon 7mo ago
Measuring data centers in watts is like measuring cars in horsepower. Power isn't a direct measure of performance, but of the primary constraint on performance. When in doubt choose the thermodynamic perspective.
- pepperoni_pizza 7mo ago
  Gigawatts are units of power, gigawatthours are units of energy. The equivalent of cars would be pricing by how much gas you burned, not horsepower.
  - delichon 7mo ago
    1 horsepower = 745.7 watts
    - pepperoni_pizza 7mo ago
      Yes, and that is both units of power, not energy.
stingraycharles 7mo ago
I mean a single nuclear reactor delivers around 1GW, so if a single datacenter consumes multiple of those, it gives a reasonably accurate idea of the scale.
deleted 7mo ago
[deleted]
franktankbank 7mo ago
This conversation is confusing because OP didn't use the same units as the person in the quote.

nomel 7mo ago

> but the published data immediately seems to admit that this is a bad choice of unit because it costs a lot more to output a token than input one And, that's silly, because API pricing is more expensive for output than input tokens, 5x so for Anthropic [1], and 6x so for OpenAI! [1] https://platform.claude.com/docs/en/about-claude/pricing https://platform.claude.com/docs/en/about-claude/pricing [2] https://openai.com/api/pricing https://openai.com/api/pricing

AlphaSite 7mo ago
I think for the same model wall time is probably a more intuitive metric; at the end of the day what you’re doing is renting GPU time slices. Large outputs dominate compute time so are more expensive. IMO input and output token counts are actually still a bad metric since they linearise non linear cost increases and I suspect we’ll see another change in the future where they bucket by context length. XL output contexts may be 20x more expensive instead of 10x.
- nsomaru 7mo ago
  They already bucket when context goes above 200k
  - refulgentis 7mo ago
    No longer
- nomel 7mo ago
  As a customer, it's nice that I can quantize and count the units of cost in an understandable way. For Anthropic, as a business bleeding money, it's probably nice to have value-based pricing, for the tokens, so innovation (like computation efficiency improvements) can result in some extra margin. If they exposed the more direct computation cost, they could never financially benefit from any improved efficiency, including faster hardware!
- yencabulator 7mo ago
  > I think for the same model wall time is probably a more intuitive metric; at the end of the day what you’re doing is renting GPU time slices This is a bit too much of a simplification. The LLM provider batches multiple customer requests into one GPU/TPU pass over the weights, with minimal latency increase. The LLM provider may in fact be renting GPUs by the second, but the end user isn't. We the end users are essentially timesharing a pool of GPUs without any dedicated "1 vGPU" style resource allocation. In such a setting, charging by "GPU tick" sounds valid, and the various categories of token costs are an approximation of cost+margin.

twoodfin 7mo ago

That these data centers can turn electricity + a little bit of fairly simple software directly into consumer and business value is pretty much the whole story. Compare what you need to add to AWS EC2 to get the same result, above and beyond the electricity.

zozbot234 7mo ago
That's a convenient story, but most consumers' and businesses' use of AI is light enough that they could easily run local models on their existing silicon. Resorting to proprietary AI running in the datacenter would only add a tiny fraction of incremental value over that, and at a significant cost.
- twoodfin 7mo ago
  Sure but where the puck is going is long-running reasoning agents where local models are (for the moment) significantly constrained relative to a Claude Opus 4.6.
- astral_drama 7mo ago
  I'm looking forward to running a Gemma 4 turboquant on my 24GB GPU. The perf looks impressive for how compact it is. I often get a 10x more cost effective run processing on my local hardware. Still reaching for frontier models for coding, but find the hosted models on open router good enough for simple work. Feels like we are jumping to warp on flops. My cores are throttled and the fiber is lit.

jillesvangurp 7mo ago

It's not really a stable measure of compute, but it's a good indication of burn rate as energy cost is something we closely track in economies and it actually dominates a lot of the cost of operating data centers. At least short term. Over time we'll get more tokens per energy unit and less dollars for the hardware needed per energy unit. Tokens currently is too abstract for a lot of people. They have no concept of the relation ship of numbers of tokens per time unit and cost. Long term there's going to be a big shift from op-ex to cap-ex for energy usage as we shift from burning methane and coal to using renewables with storage.

amelius 7mo ago
We need a Moore's law for tokens, and energy.