4 ms·
I have a similar system. When idle, nvtop reports about 11-12W/4090. When doing bs=1 inference, the default PL for my cards is 450W each, although I've found th
by lhl 3y ago
I have a similar system. When idle, nvtop reports about 11-12W/4090. When doing bs=1 inference, the default PL for my cards is 450W each, although I've found that I can keep 97% of prefill and 99%+ of inference performance lowering the power limit to 400W.
With system overhead, basically I'd guess 50W when idle, 900W when actively working.
- MuffinFlavored 3y ago> 900W when actively working. I'm trying to think of my typical ChatGPT conversations and the flow of them. I probably ask it anywhere between 150 and 1,000 tokens, and then I wait for it to respond to me about 150 to 3,000 tokens At 30 tokens a second, responses can be "900W" for about 5s to 100s Then as a human you have to sit there and read what it says (system doesn't need to generate tokens then as you interpret them). I'm trying to think what that usage graph looks like/what it translates roughly to in terms of full usage for an hour so you can estimate daily usage in kilowatt hour to quantify how intensive peak 900w load really is in real world use...