3 ms·
They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo. I imagine Mixtral by itself would only
by LoganDark 3y ago
They need 568 LPUs to load both Mixtral 8x7B and LLaMA 70B, because they need both those models available for the demo.
I imagine Mixtral by itself would only take something like 200-300 LPUs
- wmf 3y agoOnly $5M then.
- LoganDark 3y agoI'm pretty sure $20,000 per LPU isn't actually the cost of these LPUs. I saw someone else on HN asking if $20,000 could get them something and an employee said to reach out. Which makes me think $20,000 is enough to get some sort of model running at least, even if it's not necessarily an LLM.
- int_19h 3y ago$5M once, upfront. But given the significantly increased throughput, how fast does that pay for itself?
- fzzzy 3y agoYou need computers for all of them and megawatts of power, power supplies, cooling, and power distribution.
- int_19h 3y agoNaturally, but you need that for GPUs as well, no? What is the actual difference when running, when measured per token generated?
- gessha 3y agoDepends on power usage. I’m curious how power hungry those are compared to server/workstation cards.