5 ms·
Holy smokes. I had guessed that one would need a H100 to run this. I had no idea you would need multiple. With how scarce those are, the running costs for th
by wing-_-nuts 3y ago
Holy smokes. I had guessed that one would need a H100 to run this. I had no idea you would need multiple. With how scarce those are, the running costs for this must be immense!
- deleted 3y ago[deleted]
- logicchains 3y agoYou should be able to run it quantised with much less RAM. 256GB for 8bit, 128GB RAM for 4bit quantisation.
- wing-_-nuts 3y agoonly 128GB, lol
- logicchains 3y agoOnly costs a few hundred bucks for CPU ram. Sure it's slow, but for creative work it's competitive in speed with a human.
- uoaei 3y agoHumans run at approximately 100W, for 2 H100s you're looking at 600W-1400W. Plus humans have a much larger variety of capabilities. And they're more fun. So you're paying ~10x the power costs to get worse, unverified, illogical answers faster when using LLMs vs humans. Which then have to be checked and revised by humans anyway.
- TrueDuality 3y agoThis is a pretty cool and neat comparison that I haven't seen before. Probably worth including the rest of the server required to run 2 H100s because those aren't trivial either... I think the 100W might just be for an estimate of the human brain so maybe it is an equivalent example. I know this isn't the spirit you meant it in, but I'm also impressed with humanity that we've managed to develop something as capable as it is (admittedly significantly less reliable and capable than a person) at only an order of magnitude difference in power consumption.
- GaggiX 3y ago>So you're paying ~10x the power costs to get worse I mean I don't usually plug myself into an electrical outlet, isn't food much more expensive for the same amount of energy?
- espadrine 3y agoThat is an interesting question. Where I live, the cost of electricity is 0.2276 €/KWh. So the two H100, at 1KW, cost 0.2276×24 = €5.5 ($6) per day, which is nearly my groceries average. (My meals are powering all of my body though, which is five times the consumption that my brain requires, so all in all, it seems a bit more power-efficient than the GPU still.)
- orbital-decay 3y ago> Humans run at approximately 100W I believe it's many times less for the brain. There's no way it dissipates anything close to 100W without cooking itself.
- jpk 3y agoSure, but brains aren't useful without the rest of the human.
- gsuuon 3y agoWhat a deeply unnerving thread..
- uoaei 3y agoBrains are about 20W alone, but needs the rest of the body to run properly.
- Taek 3y agoThe brain is literally liquid cooled (blood), it can totally dispense of 100w of heat if it has to. (that said, other commenters are correct, the brain used 20w)
- logicchains 3y agoFor a human as smart as Falcon 180B you'd probably need to pay at least $100k per year in the US.
- alchemist1e9 3y agoAnd even then to get them to actually work as quickly would be impossible.
- alchemist1e9 3y agoHowever the LLM doing inference on 2 H100s can easily exceed 10x the content generation rate of a single human. Regarding quality of the output, it obviously depends on the task, but we are benchmarking the models on real world metrics and they are beating most humans already.
- deleted 3y ago[deleted]
- politelemon 3y agoHow are they able to afford this, are they currently simply burning through vc money?