3 ms·
Err… no. They seem to be using the power consumption of an entire data centre as if the whole building was used for just your email. That’s not how that works!
by jiggawatts 2y ago
Err… no. They seem to be using the power consumption of an entire data centre as if the whole building was used for just your email. That’s not how that works!
GPT-4 needs roughly 8x GPUs and can generate 100 words in about a second or two. Let’s say it’s a single DGX server running for 2 seconds at 6.5kW. This is just 13 kJ, which can be cooled with about 5mL of water maximum… not 3 bottles!
In reality it’s much more efficient than that because during inference multiple requests are processed together, typically something like 8 to 32 at once depending on the model and the hardware. This means that a single email needs something like a tiny drop of water… which isn’t destroyed, merely evaporated.
This is histerical reporting by a scientifically illiterate arts major worried about being replaced by an AI next year.
- Atotalnoob 2y agoSource for gpt4 requiring 8 gpus?
- jiggawatts 2y agoRumours are that the original base model was a 16-way mixture of experts over 1 TB in size. With quantisation that would fit into 8x 80GB cards = 640 GB total. That is similar in size to competitive models. There’s also a practical benefit of being able to use 100% of a common platform such as a DGX server. Splitting across multiple servers is much more complex and less efficient.
- throwaway48540 2y agoDo you need to run all experts at once?
- jiggawatts 2y agoNo, but different ones are invoked for each word so typically they’d all be preloaded into VRAM. Also, because of batching, different users would use different experts concurrently. There’s no detailed information about how GPT4 is hosted, but we can guess from competing models and the leaked info.