35 ms·

It's neat that Huggingface lets you test-drive these models right in your browser. This particular model says it needs 640GB of memory just for inference. Ass

It's neat that Huggingface lets you test-drive these models right in your browser. This particular model says it needs 640GB of memory just for inference. Assuming Huggingface also has other large models loaded, and wants to also make them available to a non-trivial number of concurrent users -- I wonder how many GPUs they have just to power this test-drive feature.

wing-_-nuts 3y ago
Holy smokes. I had guessed that one would need a H100 to run this. I had no idea you would need multiple. With how scarce those are, the running costs for this must be immense!
- deleted 3y ago
  [deleted]
- logicchains 3y ago
  You should be able to run it quantised with much less RAM. 256GB for 8bit, 128GB RAM for 4bit quantisation.
  - wing-_-nuts 3y ago
    only 128GB, lol
    - logicchains 3y ago
      Only costs a few hundred bucks for CPU ram. Sure it's slow, but for creative work it's competitive in speed with a human.
      - uoaei 3y ago
        Humans run at approximately 100W, for 2 H100s you're looking at 600W-1400W. Plus humans have a much larger variety of capabilities. And they're more fun. So you're paying ~10x the power costs to get worse, unverified, illogical answers faster when using LLMs vs humans. Which then have to be checked and revised by humans anyway.
        TrueDuality 3y ago
        This is a pretty cool and neat comparison that I haven't seen before. Probably worth including the rest of the server required to run 2 H100s because those aren't trivial either... I think the 100W might just be for an estimate of the human brain so maybe it is an equivalent example. I know this isn't the spirit you meant it in, but I'm also impressed with humanity that we've managed to develop something as capable as it is (admittedly significantly less reliable and capable than a person) at only an order of magnitude difference in power consumption.
        GaggiX 3y ago
        >So you're paying ~10x the power costs to get worse I mean I don't usually plug myself into an electrical outlet, isn't food much more expensive for the same amount of energy?
        espadrine 3y ago
        That is an interesting question. Where I live, the cost of electricity is 0.2276 €/KWh. So the two H100, at 1KW, cost 0.2276×24 = €5.5 ($6) per day, which is nearly my groceries average. (My meals are powering all of my body though, which is five times the consumption that my brain requires, so all in all, it seems a bit more power-efficient than the GPU still.)
        orbital-decay 3y ago
        > Humans run at approximately 100W I believe it's many times less for the brain. There's no way it dissipates anything close to 100W without cooking itself.
        jpk 3y ago
        Sure, but brains aren't useful without the rest of the human.
        gsuuon 3y ago
        What a deeply unnerving thread..
- deleted 3y ago
  [deleted]
- politelemon 3y ago
  How are they able to afford this, are they currently simply burning through vc money?
ilaksh 3y ago
It's 8 x A100 for inference. Huggingface has a lot more than 8 A100s.

wing-_-nuts 3y ago

Holy smokes. I had guessed that one would need a H100 to run this. I had no idea you would need multiple. With how scarce those are, the running costs for this must be immense!

deleted 3y ago
[deleted]
logicchains 3y ago
You should be able to run it quantised with much less RAM. 256GB for 8bit, 128GB RAM for 4bit quantisation.
- wing-_-nuts 3y ago
  only 128GB, lol
  - logicchains 3y ago
    Only costs a few hundred bucks for CPU ram. Sure it's slow, but for creative work it's competitive in speed with a human.
    - uoaei 3y ago
      Humans run at approximately 100W, for 2 H100s you're looking at 600W-1400W. Plus humans have a much larger variety of capabilities. And they're more fun. So you're paying ~10x the power costs to get worse, unverified, illogical answers faster when using LLMs vs humans. Which then have to be checked and revised by humans anyway.
      - TrueDuality 3y ago
        This is a pretty cool and neat comparison that I haven't seen before. Probably worth including the rest of the server required to run 2 H100s because those aren't trivial either... I think the 100W might just be for an estimate of the human brain so maybe it is an equivalent example. I know this isn't the spirit you meant it in, but I'm also impressed with humanity that we've managed to develop something as capable as it is (admittedly significantly less reliable and capable than a person) at only an order of magnitude difference in power consumption.
      - GaggiX 3y ago
        >So you're paying ~10x the power costs to get worse I mean I don't usually plug myself into an electrical outlet, isn't food much more expensive for the same amount of energy?
        espadrine 3y ago
        That is an interesting question. Where I live, the cost of electricity is 0.2276 €/KWh. So the two H100, at 1KW, cost 0.2276×24 = €5.5 ($6) per day, which is nearly my groceries average. (My meals are powering all of my body though, which is five times the consumption that my brain requires, so all in all, it seems a bit more power-efficient than the GPU still.)
      - orbital-decay 3y ago
        > Humans run at approximately 100W I believe it's many times less for the brain. There's no way it dissipates anything close to 100W without cooking itself.
        jpk 3y ago
        Sure, but brains aren't useful without the rest of the human.
        gsuuon 3y ago
        What a deeply unnerving thread..
deleted 3y ago
[deleted]
politelemon 3y ago
How are they able to afford this, are they currently simply burning through vc money?

ilaksh 3y ago

It's 8 x A100 for inference. Huggingface has a lot more than 8 A100s.