3 ms·
> as costs go down Huh? Why would that happen? Indications are that costs will likely go up, especially if currently vendors are selling tokens at a loss.
by jcgrillo 6mo ago
> as costs go down
Huh? Why would that happen? Indications are that costs will likely go up, especially if currently vendors are selling tokens at a loss.
- twoodfin 6mo agoThe main operational expense of a million LLM tokens is pennies of electricity. Even if you generously depreciate the GPU and other hardware, it’s hard to believe inference at scale in April 2026 isn’t highly profitable.
- jurgenburgen 6mo ago> The main operational expense of a million LLM tokens is pennies of electricity. I think you meant dollars of electricity.
- twoodfin 6mo agoI don’t think so. https://www.theregister.com/2024/03/18/nvidia_turns_up_the_ai/ https://www.theregister.com/2024/03/18/nvidia_turns_up_the_a... A Blackwell 8X node consumes about 15kw, let’s up that to 50kw to generously account for cooling and everything else. A US kWh is something like $0.20, so running that node for an hour costs ~$10. Nvidia got 30,000 parallel TPS out of DeepSeek-R1 on that node: https://developer.nvidia.com/blog/nvidia-blackwell-delivers-world-record-deepseek-r1-inference-performance/ https://developer.nvidia.com/blog/nvidia-blackwell-delivers-... So that $10 buys you over 100M tokens or … pennies per million. I’m sure these numbers are off, but not by an aggregate two orders of magnitude.