26 ms·
the median prompt [...] consumes 0.24 watt-hours of electricity In layman's terms, that is (approximately) - one second of running a toaster, or - 1/80th of
by threeducks 1y ago
the median prompt [...] consumes 0.24 watt-hours of electricity
In layman's terms, that is (approximately)
- one second of running a toaster, or
- 1/80th of a phone charge,
- lifting 100 pounds to a height of 6 feet,
- muzzle energy of a 9mm bullet,
- driving 6 feet with a Tesla.
- 1gn15 1y agoThat's surprisingly energy efficient, for a technology that seemed like magic back in 2022.
- Capricorn2481 1y agoHow in the world do they claim to have 0.24Wh per query? MIT estimates the LLama 3.1 405B model uses 1.86Wh. GPT-4 is over a trillion parameters. Is there any reason to think they have 2.5x more parameters but somehow use 6x less energy?
- logicprog 1y agoMixture of Experts. Llama 3.1 405B is a dense model, so to evaluate the next token, the context has to go through literally every parameter in its neural network. Whereas with mixture of experts, it's usually like a sixth to a tenth or even less of the neural network parameters that actually get evaluated for every token. Also, they don't use GPT-4 or 4.5 anymore iirc, which may have been dense (and that's why they were so expensive), 4.1 and 4o are much different models.
- myaccountonhn 1y agoThe median prompt being a google search, not agentic coding for example. I also think this misses a bit the forest from the trees, emissions need to go down across all industries and tech is actively increasing theirs.