3 ms·

Possibly ridiculous question, but does GPU lifespan decrease with more intense use? I could imagine, if so, that they bought some hardware, which needs replaci

Possibly ridiculous question, but does GPU lifespan decrease with more intense use? I could imagine, if so, that they bought some hardware, which needs replacing sooner than they thought, and the running costs are catching up with them. Or not. Nigh impossible to speculate!

singularity2001 3y ago
Given the energy and heat involved the answer is most certainly yes
nisegami 3y ago
Haven't they left the hardware stuff to Azure/Microsoft?
- rich_sasha 3y ago
  I would imagine at such scale this abstraction starts to fail. Eg say they wrecked Microsoft's GPUs, MS probs can't just provision more somewhere else at a blink of an eye.
- ben_w 3y ago
  There's only so much you can leave to Azure. I have no idea how many tokens/second OpenAI gets[0], but 100 M users, each generating an average of (to pick an arbitrary number for a Fermi estimate) 1000 tokens per day, is 1e11 tokens/day or ~1,157,407/s, which could be anywhere from 36k severs to just 830 depending on which value[0] I use as the anchor to guess how many tokens/s their specific servers can generate with their specific models. At the upper end, they may have bought about 8% of all H100s currently in existence. The Microsoft investment (1e10 USD) correlates better with the higher range than the lower range of my estimates for how many machines are in their cluster ((36k * H100) * $30k/H100 ~= 1e9 USD). [0] a quick google said 30/s in the automated answers section (which is citing a May post on Reddit), while one of the links says ~300/s on an H100, and the Llama.cpp github page says ~1400/s on an M2 Ultra (but for a relatively small model).
Yizahi 3y ago
Mosfets have lifetime depending on the temperature, so yes. But on the other hand vendor can install components which won't fail for some long time even under max load. If I remember correctly this was the reason for the "New World destroys GPUs", because some vendors didn't read the Nvidia spec carefully and installed less robust mosfets, so the max load (which happened to be New World game) pushed them over the expected lifetime fast. But with proper design this seems not to be a case.
rsynnott 3y ago
> but does GPU lifespan decrease with more intense use? In principle, yes, could do (solid state electronics do wear to some extent) but in practice data centre GPUs will be specced for a low failure rate under constant use.

singularity2001 3y ago

Given the energy and heat involved the answer is most certainly yes

nisegami 3y ago

Haven't they left the hardware stuff to Azure/Microsoft?

rich_sasha 3y ago
I would imagine at such scale this abstraction starts to fail. Eg say they wrecked Microsoft's GPUs, MS probs can't just provision more somewhere else at a blink of an eye.
ben_w 3y ago
There's only so much you can leave to Azure. I have no idea how many tokens/second OpenAI gets[0], but 100 M users, each generating an average of (to pick an arbitrary number for a Fermi estimate) 1000 tokens per day, is 1e11 tokens/day or ~1,157,407/s, which could be anywhere from 36k severs to just 830 depending on which value[0] I use as the anchor to guess how many tokens/s their specific servers can generate with their specific models. At the upper end, they may have bought about 8% of all H100s currently in existence. The Microsoft investment (1e10 USD) correlates better with the higher range than the lower range of my estimates for how many machines are in their cluster ((36k * H100) * $30k/H100 ~= 1e9 USD). [0] a quick google said 30/s in the automated answers section (which is citing a May post on Reddit), while one of the links says ~300/s on an H100, and the Llama.cpp github page says ~1400/s on an M2 Ultra (but for a relatively small model).

Yizahi 3y ago

Mosfets have lifetime depending on the temperature, so yes. But on the other hand vendor can install components which won't fail for some long time even under max load. If I remember correctly this was the reason for the "New World destroys GPUs", because some vendors didn't read the Nvidia spec carefully and installed less robust mosfets, so the max load (which happened to be New World game) pushed them over the expected lifetime fast. But with proper design this seems not to be a case.

rsynnott 3y ago

> but does GPU lifespan decrease with more intense use? In principle, yes, could do (solid state electronics do wear to some extent) but in practice data centre GPUs will be specced for a low failure rate under constant use.