3 ms·
An idle GPU consumes almost nothing, a loaded (server-class) GPU can consume over 2kW. Admittedly a single request isn't a full load, but claiming that a reque
by tredre3 6mo ago
An idle GPU consumes almost nothing, a loaded (server-class) GPU can consume over 2kW.
Admittedly a single request isn't a full load, but claiming that a request makes no difference vs idle is misguided, in my opinion.
- bijowo1676 6mo agoOpenAI GPU wont be idle for long because they have all other requests to serve. Over time there will be a certain % of idle GPUs, amortized across all hundreds of millions of requests they receive.
- aoeusnth1 6mo agoAnd idle% is causally connected to whether you make a request or not, surely? I don't understand how your mental model works.