4 ms·
Haven't they left the hardware stuff to Azure/Microsoft?
by nisegami 3y ago
Haven't they left the hardware stuff to Azure/Microsoft?
- rich_sasha 3y agoI would imagine at such scale this abstraction starts to fail. Eg say they wrecked Microsoft's GPUs, MS probs can't just provision more somewhere else at a blink of an eye.
- ben_w 3y agoThere's only so much you can leave to Azure. I have no idea how many tokens/second OpenAI gets[0], but 100 M users, each generating an average of (to pick an arbitrary number for a Fermi estimate) 1000 tokens per day, is 1e11 tokens/day or ~1,157,407/s, which could be anywhere from 36k severs to just 830 depending on which value[0] I use as the anchor to guess how many tokens/s their specific servers can generate with their specific models. At the upper end, they may have bought about 8% of all H100s currently in existence. The Microsoft investment (1e10 USD) correlates better with the higher range than the lower range of my estimates for how many machines are in their cluster ((36k * H100) * $30k/H100 ~= 1e9 USD). [0] a quick google said 30/s in the automated answers section (which is citing a May post on Reddit), while one of the links says ~300/s on an H100, and the Llama.cpp github page says ~1400/s on an M2 Ultra (but for a relatively small model).