4 ms·
What is so special about OpenAI's cloud hardware that one can't build themselves a similar server to run AI models of similar size?
by grishka 3y ago
What is so special about OpenAI's cloud hardware that one can't build themselves a similar server to run AI models of similar size?
- objectivetruth 3y ago[dead]
- baq 3y agoNothing stopping you from buying an H100 and putting it in your desktop. As for me, I’ve got other uses for $45k.
- dcchambers 3y agoIf you wanted to buy one...are you even able to order one? Aren't they massively backordered by big companies? Is there a black marker for H100 cards?
- baq 3y agoI took the quote from eBay. Not sure which shade of black that would be ;)
- airspresso 3y agoThe hardware is primarily standard Nvidia GPUs (A100s, H100s), but the scale of the infrastructure is on another level entirely. These models currently need clusters of GPU-powered servers to make predictions fast enough. Which explains why OpenAI partnered with Microsoft and got billions in funding to spend on compute. You can run (much) smaller LLM models on consumer-grade GPUs though. A single Nvidia GPU with 8 GB RAM is enough to get started with models like Zephyr, Mistral or Llama2 in their smallest versions (7B parameters). But it will be both slower and lower quality than anything OpenAI currently offers.
- qeternity 3y ago> But it will be both slower and lower quality than anything OpenAI currently offers. It will definitely not be slower. Local inference with a 7b model on a 3090/4090 will outpace 3.5-turbo and smoke 4-turbo.