4 ms·

There's also a problem of dataset size. the 7900XTX maxes out at 24GB when a lot of new models require the higher-end 80GB A100 variants. The A100 is also a ser

There's also a problem of dataset size. the 7900XTX maxes out at 24GB when a lot of new models require the higher-end 80GB A100 variants. The A100 is also a server-grade card, so even spec-for-spec it's not a fair comparison.

Const-me 4y ago
> a lot of new models require the higher-end 80GB A100 variants For training, yeah. But at least in some cases, inference can be optimized to use less memory. For their large Whisper model, OpenAI says they need 10GB of VRAM: https://github.com/openai/whisper#available-models-and-languages https://github.com/openai/whisper#available-models-and-langu... My DirectCompute port of that uses slightly over 4GB of VRAM, see the last column in that table: https://github.com/Const-me/Whisper/blob/master/SampleClips/summary.tsv https://github.com/Const-me/Whisper/blob/master/SampleClips/... I haven’t actually optimized memory usage I only did the DirectCompute port, the memory savings were achieved by Georgi Gerganov in whisper.cpp project, which I ported to Windows.
- anentropic 4y ago
  apparently GPT-J needs ~25GB RAM min (for inference only), i.e. a 32GB card ...and GPT-3 is an order of magnitude bigger again https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-gpt-neo.html https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-gp... though TBH just making cards with more RAM to fit these kind of models seems easier than being at absolute cutting edge of performance I also wonder how this metric translates to something like Apple Silicon 'unified memory'... latest MBP have 32GB RAM, I wonder if anyone managed to run GPT-J there yet?
  - anentropic 4y ago
    There's a float16 version of GPT-J rewritten in C++ here that apparently fits into 16GB... https://github.com/ggerganov/ggml/tree/master/examples/gpt-j https://github.com/ggerganov/ggml/tree/master/examples/gpt-j it's CPU-only and runs inference at "about ~6 words per second" on an M1 MBP
    - smoldesu 4y ago
      That's for the max-size model (6B parameters). GPT-Neo has pruned models 125m and 1.3b that fit into 1.5gb and 6gb of memory respectively. These models are obviously nerfed, but you can run them on budget ARM hardware and get legible, paragraph-length responses in less than 10 seconds.
      - flangola7 4y ago
        There's a 20B model now
        smoldesu 4y ago
        The 20B model was the original I believe, and the other models were all pruned from it.
        atylerrice 4y ago
        This is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.
    - mark_l_watson 4y ago
      They have a link to the bottom for something similar, but with GPT-2. The Hugging Face docs have hints for using FP16, and avoiding double memory requirements for model and checkpoint data.
    - atylerrice 4y ago
      You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)
  - trifurcate 4y ago
    I am running GPT-J-6B inference on 13GB of VRAM. FP16 is a must at this scale. I've also loaded GPT-NeoX-20B using int8 quantization onto a single RTX 3090 before (24GB VRAM) however inference latency was quite high.
  - andruby 4y ago
    Actually, the MacBook Pro with M2 Max CPU can be configured with 96GB of ram. The Mac Studio with M1 Ultra can have up to 128GB of ram.
    - anentropic 4y ago
      Ah great, I hadn't seen that... seems like that should be plenty for some of these models!
- TOMDM 4y ago
  I've been running the stock implementation that OpenAI put out, I didn't know people had made so many improvements! Thank you for your work.
godelski 4y ago
As someone who trains large models, this is pretty much the only number I care about. Batching saves more time, and gets better results, than clock speed. For the A100, A6000s, and H100s you are paying for the memory and the upcharge for being a server card. AMD isn't really competing in this space, but they are moving in that direction if you look at the exascale machines[0,1] [0] https://www.top500.org/system/180047/ https://www.top500.org/system/180047/ [1] https://www.amd.com/en/products/server-accelerators/instinct-mi250x https://www.amd.com/en/products/server-accelerators/instinct...
- erichocean 4y ago
  > Batching saves more time, and gets better results, than clock speed. How much time is saved? If you've got a 12GB card at 5x speed vs. a 96GB card at 1x speed (so 5x slower), is the former still competitive? How much faster does the 12GB card need to be to get the same quality results with less memory (and smaller batches)? What if the faster card has 24GB of memory?
  - atylerrice 4y ago
    Well the batch size also helps with making training more stable. So a larger batch size computes the gradient across more examples so it ends up being more stable than only using a batch size of day one. A batch size of one is only fixing the error on one example rather than many. a larger Batch size can help keep the gpu fed but usually that’s not a problem with these larger models.
  - godelski 4y ago
    Hard to say tbh. Could be infinite, could be zero. In classification batch is going to help a ton (including stability) and you may not even get top results without a high batch, so infinite in that case. More realistically, consider that batch also helps with GPU I/O. I/O slows things down quite a lot, so if you're moving stuff in and out of the accelerator a lot then you can easily be I/O bound. For generation the conversation gets more complicated. For GANs the smaller card could do better if you can fit a proper batch size there (probably not tbh but often closer to 16 or 24 so that may be better here). Here distributed tends to help more. For VAEs, Diffusion, Autoregressive, etc are more stable and can also hugely benefit from large batches. So the truth is an unsatisfying "it depends." I know that may not be the answer you're looking for, but it is the real one. It takes a lot of work and tricks to scale (as anyone working in non-ML HPC will tell you very similar stories about scaling). But also note that the clock speeds aren't as big of a gap as you note though the memory gaps are.
TallGuyShort 4y ago
Could you educate me on what's really different about a card that is "server-grade" vs. one that is not?
- jmalicki 4y ago
  Two of the biggest differences are ECC memory and virtualization support.
  - crabbone 4y ago
    Yeah, also while ECC is mostly transparent to the software layer, virtualization support is in itself a huge software layer that comes with this kind of equipment. MIG ( https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ ), there's also software made to expose GPU inside containers. This software is mostly supplied by vendor, and it mostly cares about vendor's tech, and will usually require special features exposed by the hardware, so, won't work with consumer-grade GPUs.
  - m463 4y ago
    well, and it's a specialized die. A100 ~ 2x size of 3090
- ericpauley 4y ago
  For these GPUs, it partially comes down to form factor (being able to drop 8 in a chassis with unified cooling). I’d say that’s the primary difference here. This applies maybe a little more noticeably to the rest of the computer around the a100s, with ECC ram, hot swappable components, and redundant power supplies. The 8x a100 rigs are absolute beasts, difficult to move even by two people without removing some components. There is also price discrimination here where manufacturers aim for higher margins on server components (as nvidia clearly does). Between the two mentioned cards, though, it really does come down to performance. The cards are simply not interchangeable for large scale model training.

Const-me 4y ago

> a lot of new models require the higher-end 80GB A100 variants For training, yeah. But at least in some cases, inference can be optimized to use less memory. For their large Whisper model, OpenAI says they need 10GB of VRAM: https://github.com/openai/whisper#available-models-and-languages https://github.com/openai/whisper#available-models-and-langu... My DirectCompute port of that uses slightly over 4GB of VRAM, see the last column in that table: https://github.com/Const-me/Whisper/blob/master/SampleClips/summary.tsv https://github.com/Const-me/Whisper/blob/master/SampleClips/... I haven’t actually optimized memory usage I only did the DirectCompute port, the memory savings were achieved by Georgi Gerganov in whisper.cpp project, which I ported to Windows.

anentropic 4y ago
apparently GPT-J needs ~25GB RAM min (for inference only), i.e. a 32GB card ...and GPT-3 is an order of magnitude bigger again https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-gpt-neo.html https://nlpcloud.com/gpt-3-open-source-alternatives-gpt-j-gp... though TBH just making cards with more RAM to fit these kind of models seems easier than being at absolute cutting edge of performance I also wonder how this metric translates to something like Apple Silicon 'unified memory'... latest MBP have 32GB RAM, I wonder if anyone managed to run GPT-J there yet?
- anentropic 4y ago
  There's a float16 version of GPT-J rewritten in C++ here that apparently fits into 16GB... https://github.com/ggerganov/ggml/tree/master/examples/gpt-j https://github.com/ggerganov/ggml/tree/master/examples/gpt-j it's CPU-only and runs inference at "about ~6 words per second" on an M1 MBP
  - smoldesu 4y ago
    That's for the max-size model (6B parameters). GPT-Neo has pruned models 125m and 1.3b that fit into 1.5gb and 6gb of memory respectively. These models are obviously nerfed, but you can run them on budget ARM hardware and get legible, paragraph-length responses in less than 10 seconds.
    - flangola7 4y ago
      There's a 20B model now
      - smoldesu 4y ago
        The 20B model was the original I believe, and the other models were all pruned from it.
        atylerrice 4y ago
        This is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.
  - mark_l_watson 4y ago
    They have a link to the bottom for something similar, but with GPT-2. The Hugging Face docs have hints for using FP16, and avoiding double memory requirements for model and checkpoint data.
  - atylerrice 4y ago
    You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)
- trifurcate 4y ago
  I am running GPT-J-6B inference on 13GB of VRAM. FP16 is a must at this scale. I've also loaded GPT-NeoX-20B using int8 quantization onto a single RTX 3090 before (24GB VRAM) however inference latency was quite high.
- andruby 4y ago
  Actually, the MacBook Pro with M2 Max CPU can be configured with 96GB of ram. The Mac Studio with M1 Ultra can have up to 128GB of ram.
  - anentropic 4y ago
    Ah great, I hadn't seen that... seems like that should be plenty for some of these models!
TOMDM 4y ago
I've been running the stock implementation that OpenAI put out, I didn't know people had made so many improvements! Thank you for your work.

godelski 4y ago

As someone who trains large models, this is pretty much the only number I care about. Batching saves more time, and gets better results, than clock speed. For the A100, A6000s, and H100s you are paying for the memory and the upcharge for being a server card. AMD isn't really competing in this space, but they are moving in that direction if you look at the exascale machines[0,1] [0] https://www.top500.org/system/180047/ https://www.top500.org/system/180047/ [1] https://www.amd.com/en/products/server-accelerators/instinct-mi250x https://www.amd.com/en/products/server-accelerators/instinct...

erichocean 4y ago
> Batching saves more time, and gets better results, than clock speed. How much time is saved? If you've got a 12GB card at 5x speed vs. a 96GB card at 1x speed (so 5x slower), is the former still competitive? How much faster does the 12GB card need to be to get the same quality results with less memory (and smaller batches)? What if the faster card has 24GB of memory?
- atylerrice 4y ago
  Well the batch size also helps with making training more stable. So a larger batch size computes the gradient across more examples so it ends up being more stable than only using a batch size of day one. A batch size of one is only fixing the error on one example rather than many. a larger Batch size can help keep the gpu fed but usually that’s not a problem with these larger models.
- godelski 4y ago
  Hard to say tbh. Could be infinite, could be zero. In classification batch is going to help a ton (including stability) and you may not even get top results without a high batch, so infinite in that case. More realistically, consider that batch also helps with GPU I/O. I/O slows things down quite a lot, so if you're moving stuff in and out of the accelerator a lot then you can easily be I/O bound. For generation the conversation gets more complicated. For GANs the smaller card could do better if you can fit a proper batch size there (probably not tbh but often closer to 16 or 24 so that may be better here). Here distributed tends to help more. For VAEs, Diffusion, Autoregressive, etc are more stable and can also hugely benefit from large batches. So the truth is an unsatisfying "it depends." I know that may not be the answer you're looking for, but it is the real one. It takes a lot of work and tricks to scale (as anyone working in non-ML HPC will tell you very similar stories about scaling). But also note that the clock speeds aren't as big of a gap as you note though the memory gaps are.

TallGuyShort 4y ago

Could you educate me on what's really different about a card that is "server-grade" vs. one that is not?

jmalicki 4y ago
Two of the biggest differences are ECC memory and virtualization support.
- crabbone 4y ago
  Yeah, also while ECC is mostly transparent to the software layer, virtualization support is in itself a huge software layer that comes with this kind of equipment. MIG ( https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ https://docs.nvidia.com/datacenter/tesla/mig-user-guide/ ), there's also software made to expose GPU inside containers. This software is mostly supplied by vendor, and it mostly cares about vendor's tech, and will usually require special features exposed by the hardware, so, won't work with consumer-grade GPUs.
- m463 4y ago
  well, and it's a specialized die. A100 ~ 2x size of 3090
ericpauley 4y ago
For these GPUs, it partially comes down to form factor (being able to drop 8 in a chassis with unified cooling). I’d say that’s the primary difference here. This applies maybe a little more noticeably to the rest of the computer around the a100s, with ECC ram, hot swappable components, and redundant power supplies. The 8x a100 rigs are absolute beasts, difficult to move even by two people without removing some components. There is also price discrimination here where manufacturers aim for higher margins on server components (as nvidia clearly does). Between the two mentioned cards, though, it really does come down to performance. The cards are simply not interchangeable for large scale model training.