3 ms·
I don't think that's really a fair comparison. Like if you wanted to sell ball bearings you could spend $50 and start dripping lead into a bucket of water, or y
by recursivecaveat 2y ago
I don't think that's really a fair comparison. Like if you wanted to sell ball bearings you could spend $50 and start dripping lead into a bucket of water, or you could spend millions on a factory. The latter is definitely the better business model. What maters is quality of result (latency and tokens/s/user) and throughput per watt or capex dollar, not your minimum capex expense so long as you're running at any kind of scale.
- elorant 2y agoHow is throughput per watt affordable? Each card has a TDP of 275 watts, and a maximum draw of 375. If you try to run a big model like Llama-3 400B you're looking at a cluster with 1,8k cards, and a total power draw of nearly 500KWs per hour.