4 ms·
I would be very interested in the breakdown of how they conclude it's 0.01kWh (10Wh) per request, and what that does and doesn't include. I expect that if you
by Syonyk 2y ago
I would be very interested in the breakdown of how they conclude it's 0.01kWh (10Wh) per request, and what that does and doesn't include.
I expect that if you were to calculate "incremental energy per request" - how much "extra CPU compute" each request adds, you could probably get to that sort of value.
But odds are good that figure ignored all the training data collection, all the processing on that, storage of that digested information, retrieval of it, etc, and that sort of number tends to also skip things like "storage systems running to have the information available."
If I've got an entire datacenter running to provide services, and the request consumes, say, 3 GPU-minutes of time across all the nodes, sure. This is a sane value. It just ignores a lot of the other resources dedicated to the task at various points.
Microsoft isn't using 30% more energy than their current footprint on 10Wh/request AI answers.
Math like this is very much a "Tell me what answer you'd like, and I'll make it work out!" sort of scenario. An increase in data center use by 30% is harder to fudge.
- bruce511 2y agoTo put your follow upquestion differently, 'Is 10w/h per request the marginal cost of a request? Or does it factor in the fixed energy cost of the whole facility? Does it include training costs' I'm inclined to lean towards it including at least some fixed costs. It seems rather high to be marginal cost. I have no gut feel for training costs though. So, assuming it does include at least some fixed cost, using it more will reduce 'cost per use' while at the same time driving up actual consumption.