3 ms·
How in the world do they claim to have 0.24Wh per query? MIT estimates the LLama 3.1 405B model uses 1.86Wh. GPT-4 is over a trillion parameters. Is there any
by Capricorn2481 1y ago
How in the world do they claim to have 0.24Wh per query? MIT estimates the LLama 3.1 405B model uses 1.86Wh.
GPT-4 is over a trillion parameters. Is there any reason to think they have 2.5x more parameters but somehow use 6x less energy?
- logicprog 1y agoMixture of Experts. Llama 3.1 405B is a dense model, so to evaluate the next token, the context has to go through literally every parameter in its neural network. Whereas with mixture of experts, it's usually like a sixth to a tenth or even less of the neural network parameters that actually get evaluated for every token. Also, they don't use GPT-4 or 4.5 anymore iirc, which may have been dense (and that's why they were so expensive), 4.1 and 4o are much different models.