3 ms·
"Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them." An H100 has 80GB of VRAM. The H
by narrator 4y ago
"Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them."
An H100 has 80GB of VRAM. The Highest end system I can find is 8xH100. Is a 640GB model is the biggest model you can run on a single system? Already GPT-4 is throttled and has a waiting list and they haven't even released the image processing or integrations to a wide audience. Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4?
- moralestapia 4y ago>Maybe they are just unable to do inference in a cost-efficient manner and at an acceptable speed on anything bigger than GPT-4? Yes, that's the whole thing. As others have pointed out, GPT-4 seems like an optimum point that balances cost, ROI, etc... it won't improve much just by throwing more data at it.
- forrest2 4y agoYou might not see it available via self serve interfaces, but I think the h100 cards can do 256x to a single host.
- narrator 4y agoYou're right! You can nvlink 32 systems with 8 h100s! That's 20 terabytes of VRAM total!