2 ms·

So the model is buy some of the worlds most expensive hardware and let it sit idle for half the time? If I want to save the same throughput I need to buy at lea

So the model is buy some of the worlds most expensive hardware and let it sit idle for half the time? If I want to save the same throughput I need to buy at least twice the hardware! Load throttling is one of those ideas that seems great as long as someone else is doing it.

anonymousDan 1y ago
Ha, that's a great point. I guess if you have some latency sensitive inference workload the capacity will effectively be dynamic, but that is likely uncorrelated with local energy prices I imagine.