4 ms·
We don't know the parameters but it probably takes at least a H100 and possibly several to run a SOTA model. Given the pricing (25+k per H100 + hardware to run
by scheme271 5mo ago
We don't know the parameters but it probably takes at least a H100 and possibly several to run a SOTA model. Given the pricing (25+k per H100 + hardware to run it) and power (700W per H100 + hardware to run it), I don't see how anyone except for a largish company can afford to run this.
- sshumaker 5mo agoAre you serious? It’s multiple nodes to run a frontier model (a node is 8x GPUs), and they aren’t running on H100s. You are looking at 32+ GPUs.
- scheme271 5mo agoI was being pretty generous to the comment I was replying to. Needing 32+ H100s just strengthens my argument that people aren't going to run frontier models locally anytime soon.