6 ms·
Yes, one AI GPU uses more energy than one GPU used for gaming. The one AI GPU however is shared between a large number of users, while the one gaming GPU is use
by elpocko 1y ago
Yes, one AI GPU uses more energy than one GPU used for gaming. The one AI GPU however is shared between a large number of users, while the one gaming GPU is used by one player.
Apparently a single gaming GPU can be used to run an LLM that serves hundreds of concurrent requests.
> Benchmarking Llama 3.1 8B (fp16) on our 1x RTX 3090 instance suggests that it can support apps with thousands of users by achieving reasonable tokens per second at 100+ concurrent requests.
https://backprop.co/environments/vllm https://backprop.co/environments/vllm
- jesus_666 1y agoBut that's a tiny model; it's the smallest version of Llama 3.1. The commercially marketed models are way bigger - e.g. GPT-4 has been estimated to use about 1.76 trillion parameters, 220 times more than the Llama build you mentioned. Their resource and performance requirements are vastly different. You're essentially arguing that shipping naval diesel aggregates must be trivial because you can fit a dozen moped motors on the bed of your pickup truck just fine.
- elpocko 1y agoOkay but these tiny models are being used by people and businesses instead of GPT-4. My point was that they consume less energy per user than a rig used for gaming. I have no insight into how many GPT-4 users are served per GPU, but I would assume OpenAI heavily optimizes for that, considering the cost to run that thing. It's probably in the same ballpark: hundreds-thousands of concurrent user requests per GPU. Still better than one GPU per gamer, even if it requires 10x the energy.