3 ms·
Depends. For CoT models inference is significantly more costly (compared to regular models). Also, >Brent Thill of Jefferies, an analyst, estimates that [infe
by gruez 1y ago
Depends. For CoT models inference is significantly more costly (compared to regular models).
Also,
>Brent Thill of Jefferies, an analyst, estimates that [inference] accounts for 96% of the overall energy consumed in data centres used by the AI industry.
https://archive.is/GJs5n https://archive.is/GJs5n
- jnieswl 1y agoForeword Author here. I agree, even early estimates e.g. from Meta (2022) suggested 20% Training, 10% Experiments, 70% inference. And adoption is rising from month to month.
- Zacharias030 1y agoThose must have been about other things than LLMs though. Meta has huge inference loads for other types of models.