4 ms·
The power calculation is way off. Sure, a local model may use a maximum of 65 watts during inference. However, an H100 runs on 350 watts. Even accounting for
by Filligree 2y ago
The power calculation is way off.
Sure, a local model may use a maximum of 65 watts during inference.
However, an H100 runs on 350 watts. Even accounting for the fact that a larger model will need more than one slice, it’s so much faster that this almost balances out.
That’s not the largest difference, though. The H100 can run multiple user queries in parallel, and because LLM inference is limited by memory bandwidth, using large batch sizes this way will dramatically improve efficiency. In short, it takes less power to run it in the datacenter.
- rcthompson 2y agoI think they're also confusing power and energy, or at least not clearly distinguishing between them, with phrases like "enough to power X houses".
- NathanKP 2y agoTo put it in perspective I've previously calculated that eating a single beef burger is worse for the environment in terms of CO2 produced than using a high token I/O "reasoning" AI model all day long. The "AI uses a lot of electricity and creates a lot of carbon" crowd are typically way off when it comes to understanding the environmental impact of AI compared to the environmental impact of other typical daily activities.