4 ms·

>Still, we are looking at GPT3 level of performance on one server rack. That says something when less than a year ago, such AI was literally magic and only run

>Still, we are looking at GPT3 level of performance on one server rack. That says something when less than a year ago, such AI was literally magic and only run on a massive datacenter. I'm not sure what you mean by this. You've always been able to run GPT3 on a single server (your typical 8xA100).

blovescoffee 3y ago
Am I missing something or how do you know this? Also I think the OP was talking about a single card not multiple but that was just my reading.
- redox99 3y ago
  Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)
  - axiom92 3y ago
    > And basically all servers will have 8xA100 for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).
    - doctorpangloss 3y ago
      The servers the commenter is talking about are DGX machines from NVIDIA. It doesn’t really make sense to BTO. What you gain economically you lose in the science you can do. But nobody could have anticipated this.
      - _zoltan_ 3y ago
        you could also get HGX from any of the vendors.
        bobboies 3y ago
        Wtf does HGX mean? God enough with the acronyms people. Please take an extra ten seconds to speak in proper human language! You could save on the worlds carbon footprint by reducing the number of times humans have to search for “what is NVIDIA hgx?” or is it “what is AMD HGX” and then subsequently visiting the websites to see if that’s right or not.
        Vvector 3y ago
        What does Wft mean? God enough with the acronyms people. /s
        bobboies 3y ago
        You got me there hahaha However, there’s a difference between an acronym known to the broader public versus some single shot, context-specific one!
    - _zoltan_ 3y ago
      who's norm? I assure you it's the norm. :)
  - loxias 3y ago
    > And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one) wishing, or guessing something, without actual experience, doesn't make it true.
brucethemoose2 3y ago
8xA100 is technically a single server, but I think OP is talking about affordable and plentiful CPU hosts, or even relatively modest single GPU instances. DGX boxes do not grow on trees, especially these days

blovescoffee 3y ago

Am I missing something or how do you know this? Also I think the OP was talking about a single card not multiple but that was just my reading.

redox99 3y ago
Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)
- axiom92 3y ago
  > And basically all servers will have 8xA100 for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).
  - doctorpangloss 3y ago
    The servers the commenter is talking about are DGX machines from NVIDIA. It doesn’t really make sense to BTO. What you gain economically you lose in the science you can do. But nobody could have anticipated this.
    - _zoltan_ 3y ago
      you could also get HGX from any of the vendors.
      - bobboies 3y ago
        Wtf does HGX mean? God enough with the acronyms people. Please take an extra ten seconds to speak in proper human language! You could save on the worlds carbon footprint by reducing the number of times humans have to search for “what is NVIDIA hgx?” or is it “what is AMD HGX” and then subsequently visiting the websites to see if that’s right or not.
        Vvector 3y ago
        What does Wft mean? God enough with the acronyms people. /s
        bobboies 3y ago
        You got me there hahaha However, there’s a difference between an acronym known to the broader public versus some single shot, context-specific one!
  - _zoltan_ 3y ago
    who's norm? I assure you it's the norm. :)
- loxias 3y ago
  > And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one) wishing, or guessing something, without actual experience, doesn't make it true.

brucethemoose2 3y ago

8xA100 is technically a single server, but I think OP is talking about affordable and plentiful CPU hosts, or even relatively modest single GPU instances. DGX boxes do not grow on trees, especially these days