4 ms·

Am I missing something or how do you know this? Also I think the OP was talking about a single card not multiple but that was just my reading.

redox99 3y ago
Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)
- axiom92 3y ago
  > And basically all servers will have 8xA100 for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).
  - doctorpangloss 3y ago
    The servers the commenter is talking about are DGX machines from NVIDIA. It doesn’t really make sense to BTO. What you gain economically you lose in the science you can do. But nobody could have anticipated this.
    - _zoltan_ 3y ago
      you could also get HGX from any of the vendors.
      - bobboies 3y ago
        Wtf does HGX mean? God enough with the acronyms people. Please take an extra ten seconds to speak in proper human language! You could save on the worlds carbon footprint by reducing the number of times humans have to search for “what is NVIDIA hgx?” or is it “what is AMD HGX” and then subsequently visiting the websites to see if that’s right or not.
        Vvector 3y ago
        What does Wft mean? God enough with the acronyms people. /s
        bobboies 3y ago
        You got me there hahaha However, there’s a difference between an acronym known to the broader public versus some single shot, context-specific one!
  - _zoltan_ 3y ago
    who's norm? I assure you it's the norm. :)
- loxias 3y ago
  > And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one) wishing, or guessing something, without actual experience, doesn't make it true.

redox99 3y ago

Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)

axiom92 3y ago
> And basically all servers will have 8xA100 for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).
- doctorpangloss 3y ago
  The servers the commenter is talking about are DGX machines from NVIDIA. It doesn’t really make sense to BTO. What you gain economically you lose in the science you can do. But nobody could have anticipated this.
  - _zoltan_ 3y ago
    you could also get HGX from any of the vendors.
    - bobboies 3y ago
      Wtf does HGX mean? God enough with the acronyms people. Please take an extra ten seconds to speak in proper human language! You could save on the worlds carbon footprint by reducing the number of times humans have to search for “what is NVIDIA hgx?” or is it “what is AMD HGX” and then subsequently visiting the websites to see if that’s right or not.
      - Vvector 3y ago
        What does Wft mean? God enough with the acronyms people. /s
        bobboies 3y ago
        You got me there hahaha However, there’s a difference between an acronym known to the broader public versus some single shot, context-specific one!
- _zoltan_ 3y ago
  who's norm? I assure you it's the norm. :)
loxias 3y ago
> And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one) wishing, or guessing something, without actual experience, doesn't make it true.