7 ms·

ChatGPT Hardware: A look at 8x NVIDIA A100 powering the tool

Valgrim 4y ago
Is this the hardware used to build the model, or to run the model? Does this single 100000$ computer run the entire chatGPT service concurrently for all users around the world? If so this seems incredibly cheap.
- paulluuk 4y ago
  In machine learning, generally the expensive part is training the model, not running the model.
  - bobleeswagger 4y ago
    At this point the costs of hosting the trained model probably surpass the training cost.
    - delijati 4y ago
      has anyone numbers on how many queries/second in inference time chatGPT can serve?
    - paulluuk 4y ago
      It might, I'm not sure. It depends on many factors: how much cloud discount are they getting, for example? And do you count the cost of training all previous models before chatGPT, or just the cost of training chatGPT itself? But you might be right, chatGPT is insanely popular right now.
- 0x008 4y ago
  The magnificent part is that the 8xA100 is only a blade and you can fill a whole rack with them. Yet one is already incredibly powerful. Also the successor H100 is already released and at least 2x as powerful.
  - kjs3 4y ago
    Also the successor H100 is already released and at least 2x as powerful. Checks eBay for used A100s...will not be getting one soon. :-)
- HPsquared 4y ago
  Supercomputer used for training, top 5 in the world apparently: https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/ https://news.microsoft.com/source/features/ai/openai-azure-s...
- anentropic 4y ago
  I would guess that's the minimum needed to run an 'instance' of the model for inference not sure how many concurrent users that could support, but I'd imagine the public service has a whole bunch of these racks from what I read... GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB) and then GPT-3 is 175B params or ~29x larger, so maybe needing 7-800GB RAM 8x 80GB A100s would give 640GB RAM, so its approx the right ballpark
  - erichocean 4y ago
    > GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB) You can load that on M1/M2 Macs, which have unified GPU memory up to 128GB.
pcdoodle 4y ago
5KW Peak Power per system pictured. I wonder, how many watt hours per query are used on average? Is this 100k investment enough to roll your own ChatGPT level bot using open source software?
- HPsquared 4y ago
  Not exactly... OpenAI apparently have the 5th most powerful supercomputer in the world. This is probably what they used for training: https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/ https://news.microsoft.com/source/features/ai/openai-azure-s... https://www.zdnet.com/article/microsoft-builds-a-supercomputer-for-openai-for-training-massive-ai-models/ https://www.zdnet.com/article/microsoft-builds-a-supercomput...
  - pcdoodle 4y ago
    Thanks for the links. I wonder what percentage of the energy expenditure is the initial training vs. operating the service?
    - HPsquared 4y ago
      That depends how much it's used, I suppose. Ever-increasing, it looks like. At some point it might be worth investing in more training to reduce operating expense. Pretty standard CAPEX vs OPEX calculation. On the other hand, we're not counting energy saved by people using it.. What if some scientist uses it and ends up solving fusion power? Or, say, making copywriters and programmers redundant (who I assume would go on to become subsistence farmers) - many energy savings to be had!
sxp 4y ago
Are there articles detailing the tradeoffs between an A100 system and a consumer system in terms of perf/$? E.g, if you're a startup trying to be scrappy and your choice is between a $100k+ A100 system and an off-the-shelf RTX 40x0 networked cluster, what would be the tradeoff?. For embarrassingly parallel use cases like ray-tracing or a compute backend that services many clients and doesn't need FP64, the consumer system would win. For other uses cases that have a hard requirement for the NVLinked access to 80GB of RAM per card, the A100 system would win. Has anyone done benchmarks to show what the tradeoff calculations are for various use cases?
- 0x008 4y ago
  Not for whole systems but the single GPUs (A100/A40) are significantly better in performance per watt (almost by a factor of 2) than RTX cards for Machine Learning workloads. You can watch this LTT video to get an idea: https://youtu.be/zBAxiQi2nPc https://youtu.be/zBAxiQi2nPc Probably if you run them full time in a enterprise environment the power consumption will be the bulk of your cost?
- synergy20 4y ago
  I was trying to build a ML training machine myself these days. I don't really need the RTX FP64 and its 4K graphic rendering with gaming stuff, all I need is ML training(FP16 mostly with lots of CUDA cores). There is no such thing, you either use multiple RTX GPU cards and 'bend' them for ML training, or you buy A100 40GB at about 125K or A100 80GB at about 250K, which is far beyond my budget. Google's TPU could be used to assist ML training, but then it does not have the CUDA ecosystem for software, plus, it kept its new TPU for inhouse use only as far as I can tell. AWS ML EC2 are very expensive to me which is why I want to build one for mid-sized ML trainings.
- Patrick-STH 4y ago
  There are a few big ones: - The CUDA license does not allow you to use GeForce in the data center. In the US it has become less popular, but if you look at our Inspur AIStation piece, that was a cluster located in China with GeForce cards. So it still happens, but less so. - The memory capacity is another big challenge. Newer models have 80GB which dwarfs the 24GB on a 4090. We just got the RTX 6000 Ada in, so that is an option for more memory. - For higher-end training, one of the big challenges is interconnect, so having NVLink and Infiniband or 100GbE+/ Infiniband NICs is important. The HGX A100 platform is designed for that with its NVSwitch and PCIe switch topology. With all of that said, you are 100% right that many startups have used consumer cards for years. For example, Andrej Karpathy talked about how our DeepLearning11 build (8x 1080 Ti's) had a ~3 month payback period versus AWS https://twitter.com/karpathy/status/924340245478256640 https://twitter.com/karpathy/status/924340245478256640
hexomancer 4y ago
Insanely misleading title. The author is basically showing off their A100 GPUs. The ChatGPT part is just clickbait, the only relevance ia that ChatGPT supposedly is also using A100s (definitely not only eight of them though).
- yieldcrv 4y ago
  the real question is whether ChatGPT needs that kind of hardware for a single user. a client side version.
  - speedgoose 4y ago
    Yes it does.
    - jpeter 4y ago
      When stable diffusion came out, my 8gb gpu could handle 1 image at 512x512. Now i can generate 1000x800 with batch size 4. I wonder if Chatgpt has the same unused optimisation potential
      - yieldcrv 4y ago
        what packages are you using? I think I have DiffusionBee installed but I haven't updated it for months
      - fpgaminer 4y ago
        > I wonder if Chatgpt has the same unused optimisation potential Doubtful. As an example, OpenAI released Triton, a programming language explicitly for optimizing models. For the scale they're at they'd be crazy not to throw an engineer or two at optimizing the models they're going to send to production.
      - speedgoose 4y ago
        Stable Diffusion is a lot smaller than GPT3. It has a bit less than a billion parameters while GPT3 has 175 billion parameters. I would bet that we can do better than GPT3 with less computing resources but that would need a different model architecture.
Const-me 4y ago
A100 has 1555 GB/sec of memory bandwidth. 50% more than Radeon 7900 XTX, but the AMD GPU costs 10x less. nVidia’s top management did amazing job with their strategy. They have spent decades, and millions of dollars, developing and promoting CUDA. Now this motivates people to spend that amount of money on nVidia’s hardware. I don’t believe the trend continues for ML inference, at least not for long. Unlike traditional HPC stuff like FEM or fluid dynamics, the GPU code of ML inference is not that complicated. The complexity is limited to the data being processed, these ML models. It’s not terribly hard to port the inference to other hardware architecture or software stack, reusing the models. I think it’s only a matter of time until AI companies like OpenAI will start porting their code from CUDA to VulkanCompute or some other vendor-agnostic stack, to avoid buying or renting such $100k computers. Don’t get me wrong, these computers are awesome, just very expensive.
- ericpauley 4y ago
  There's also a problem of dataset size. the 7900XTX maxes out at 24GB when a lot of new models require the higher-end 80GB A100 variants. The A100 is also a server-grade card, so even spec-for-spec it's not a fair comparison.
  - Const-me 4y ago
    > a lot of new models require the higher-end 80GB A100 variants For training, yeah. But at least in some cases, inference can be optimized to use less memory. For their large Whisper model, OpenAI says they need 10GB of VRAM: https://github.com/openai/whisper#available-models-and-languages https://github.com/openai/whisper#available-models-and-langu... My DirectCompute port of that uses slightly over 4GB of VRAM, see the last column in that table: https://github.com/Const-me/Whisper/blob/master/SampleClips/summary.tsv https://github.com/Const-me/Whisper/blob/master/SampleClips/... I haven’t actually optimized memory usage I only did the DirectCompute port, the memory savings were achieved by Georgi Gerganov in whisper.cpp project, which I ported to Windows.
    - anentropic 4y ago
hyperific 4y ago
"Over 80% of this assembly is a heatsink to dissipate massive power." Not what heatsinks do.
- Patrick-STH 4y ago
  Good catch. Fixed
mk_stjames 4y ago
I recently saw a rack server of 8x SXM2 P100's (so, the 2016 equivalent of this) for sale for $3600 on ebay. And you can buy individual P100 pci-e versions for about $300 now. They retailed for about $7600 each back in 2016. I wonder how long it'll be until A100's depreciate that amount. I see articles from "Serve The Home" pop up now often and I've yet to see one discussing hardware that I've ever come across in... someone's home.
- Patrick-STH 4y ago
  STH has been around for 14 years and so we have tested these big servers from the P100 days (see our DeepLearning12 build) and newer. Home was the /home/ directory in Linux. We do have some content for homes. Check out our TinyMiniMicro series as an example.
  - trvr 4y ago
    >> Home was the /home/ directory in Linux. I have never (until reading this) thought of it in that context. Was that the original meaning for the name, or did it shift over time?
    - Patrick-STH 4y ago
      It is in the "About" page :-) I always get confused by why folks ask why we have server content. The Wall Street Journal covers more than just one road in NYC.
      - trvr 4y ago
        Understood. But what about this? https://web.archive.org/web/20110201124803im_/http://www.servethehome.com/wp-content/uploads/2010/08/STH-Logo102.jpg.pagespeed.ce.ADlm7Tggpk.jpg https://web.archive.org/web/20110201124803im_/http://www.ser...
        Patrick-STH 4y ago
        Now that is a throwback! That old logo was created with GIMP at a table outside NetApp. STH started at a time when the SMB market (and VERY high-end homes) were using Dell, HP, and IBM rackmount gear off-lease. The original STH idea came from me learning Linux as an alternative to Windows for that segment. STH was what I used to learn the rest of the market outside of work. The first product we were sent for a review was a rackmount case and that started us on the path of reviewing new gear ~2010-2011 when it was a part time blog instead of having a team of folks working on the site. I still try to keep 15-20% of our content in the SMB/ high-end home arena.
jhoelzel 4y ago
Now we just need to let the models get big enough until they can make themselves smaller again ;)
brianjking 4y ago
Soon nvidia & OpenAI will be the only ones with any money so that's fun :)

Is this the hardware used to build the model, or to run the model? Does this single 100000$ computer run the entire chatGPT service concurrently for all users around the world? If so this seems incredibly cheap.

paulluuk 4y ago
In machine learning, generally the expensive part is training the model, not running the model.
- bobleeswagger 4y ago
  At this point the costs of hosting the trained model probably surpass the training cost.
  - delijati 4y ago
    has anyone numbers on how many queries/second in inference time chatGPT can serve?
  - paulluuk 4y ago
    It might, I'm not sure. It depends on many factors: how much cloud discount are they getting, for example? And do you count the cost of training all previous models before chatGPT, or just the cost of training chatGPT itself? But you might be right, chatGPT is insanely popular right now.
0x008 4y ago
The magnificent part is that the 8xA100 is only a blade and you can fill a whole rack with them. Yet one is already incredibly powerful. Also the successor H100 is already released and at least 2x as powerful.
- kjs3 4y ago
  Also the successor H100 is already released and at least 2x as powerful. Checks eBay for used A100s...will not be getting one soon. :-)
HPsquared 4y ago
Supercomputer used for training, top 5 in the world apparently: https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/ https://news.microsoft.com/source/features/ai/openai-azure-s...
anentropic 4y ago
I would guess that's the minimum needed to run an 'instance' of the model for inference not sure how many concurrent users that could support, but I'd imagine the public service has a whole bunch of these racks from what I read... GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB) and then GPT-3 is 175B params or ~29x larger, so maybe needing 7-800GB RAM 8x 80GB A100s would give 640GB RAM, so its approx the right ballpark
- erichocean 4y ago
  > GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB) You can load that on M1/M2 Macs, which have unified GPU memory up to 128GB.

pcdoodle 4y ago

5KW Peak Power per system pictured. I wonder, how many watt hours per query are used on average? Is this 100k investment enough to roll your own ChatGPT level bot using open source software?

HPsquared 4y ago
Not exactly... OpenAI apparently have the 5th most powerful supercomputer in the world. This is probably what they used for training: https://news.microsoft.com/source/features/ai/openai-azure-supercomputer/ https://news.microsoft.com/source/features/ai/openai-azure-s... https://www.zdnet.com/article/microsoft-builds-a-supercomputer-for-openai-for-training-massive-ai-models/ https://www.zdnet.com/article/microsoft-builds-a-supercomput...
- pcdoodle 4y ago
  Thanks for the links. I wonder what percentage of the energy expenditure is the initial training vs. operating the service?
  - HPsquared 4y ago
    That depends how much it's used, I suppose. Ever-increasing, it looks like. At some point it might be worth investing in more training to reduce operating expense. Pretty standard CAPEX vs OPEX calculation. On the other hand, we're not counting energy saved by people using it.. What if some scientist uses it and ends up solving fusion power? Or, say, making copywriters and programmers redundant (who I assume would go on to become subsistence farmers) - many energy savings to be had!

sxp 4y ago

Are there articles detailing the tradeoffs between an A100 system and a consumer system in terms of perf/$? E.g, if you're a startup trying to be scrappy and your choice is between a $100k+ A100 system and an off-the-shelf RTX 40x0 networked cluster, what would be the tradeoff?. For embarrassingly parallel use cases like ray-tracing or a compute backend that services many clients and doesn't need FP64, the consumer system would win. For other uses cases that have a hard requirement for the NVLinked access to 80GB of RAM per card, the A100 system would win. Has anyone done benchmarks to show what the tradeoff calculations are for various use cases?

0x008 4y ago
Not for whole systems but the single GPUs (A100/A40) are significantly better in performance per watt (almost by a factor of 2) than RTX cards for Machine Learning workloads. You can watch this LTT video to get an idea: https://youtu.be/zBAxiQi2nPc https://youtu.be/zBAxiQi2nPc Probably if you run them full time in a enterprise environment the power consumption will be the bulk of your cost?
synergy20 4y ago
I was trying to build a ML training machine myself these days. I don't really need the RTX FP64 and its 4K graphic rendering with gaming stuff, all I need is ML training(FP16 mostly with lots of CUDA cores). There is no such thing, you either use multiple RTX GPU cards and 'bend' them for ML training, or you buy A100 40GB at about 125K or A100 80GB at about 250K, which is far beyond my budget. Google's TPU could be used to assist ML training, but then it does not have the CUDA ecosystem for software, plus, it kept its new TPU for inhouse use only as far as I can tell. AWS ML EC2 are very expensive to me which is why I want to build one for mid-sized ML trainings.
Patrick-STH 4y ago
There are a few big ones: - The CUDA license does not allow you to use GeForce in the data center. In the US it has become less popular, but if you look at our Inspur AIStation piece, that was a cluster located in China with GeForce cards. So it still happens, but less so. - The memory capacity is another big challenge. Newer models have 80GB which dwarfs the 24GB on a 4090. We just got the RTX 6000 Ada in, so that is an option for more memory. - For higher-end training, one of the big challenges is interconnect, so having NVLink and Infiniband or 100GbE+/ Infiniband NICs is important. The HGX A100 platform is designed for that with its NVSwitch and PCIe switch topology. With all of that said, you are 100% right that many startups have used consumer cards for years. For example, Andrej Karpathy talked about how our DeepLearning11 build (8x 1080 Ti's) had a ~3 month payback period versus AWS https://twitter.com/karpathy/status/924340245478256640 https://twitter.com/karpathy/status/924340245478256640

hexomancer 4y ago

Insanely misleading title. The author is basically showing off their A100 GPUs. The ChatGPT part is just clickbait, the only relevance ia that ChatGPT supposedly is also using A100s (definitely not only eight of them though).

yieldcrv 4y ago
the real question is whether ChatGPT needs that kind of hardware for a single user. a client side version.
- speedgoose 4y ago
  Yes it does.
  - jpeter 4y ago
    When stable diffusion came out, my 8gb gpu could handle 1 image at 512x512. Now i can generate 1000x800 with batch size 4. I wonder if Chatgpt has the same unused optimisation potential
    - yieldcrv 4y ago
      what packages are you using? I think I have DiffusionBee installed but I haven't updated it for months
    - fpgaminer 4y ago
      > I wonder if Chatgpt has the same unused optimisation potential Doubtful. As an example, OpenAI released Triton, a programming language explicitly for optimizing models. For the scale they're at they'd be crazy not to throw an engineer or two at optimizing the models they're going to send to production.
    - speedgoose 4y ago
      Stable Diffusion is a lot smaller than GPT3. It has a bit less than a billion parameters while GPT3 has 175 billion parameters. I would bet that we can do better than GPT3 with less computing resources but that would need a different model architecture.

Const-me 4y ago

A100 has 1555 GB/sec of memory bandwidth. 50% more than Radeon 7900 XTX, but the AMD GPU costs 10x less. nVidia’s top management did amazing job with their strategy. They have spent decades, and millions of dollars, developing and promoting CUDA. Now this motivates people to spend that amount of money on nVidia’s hardware. I don’t believe the trend continues for ML inference, at least not for long. Unlike traditional HPC stuff like FEM or fluid dynamics, the GPU code of ML inference is not that complicated. The complexity is limited to the data being processed, these ML models. It’s not terribly hard to port the inference to other hardware architecture or software stack, reusing the models. I think it’s only a matter of time until AI companies like OpenAI will start porting their code from CUDA to VulkanCompute or some other vendor-agnostic stack, to avoid buying or renting such $100k computers. Don’t get me wrong, these computers are awesome, just very expensive.

ericpauley 4y ago
There's also a problem of dataset size. the 7900XTX maxes out at 24GB when a lot of new models require the higher-end 80GB A100 variants. The A100 is also a server-grade card, so even spec-for-spec it's not a fair comparison.
- Const-me 4y ago
  > a lot of new models require the higher-end 80GB A100 variants For training, yeah. But at least in some cases, inference can be optimized to use less memory. For their large Whisper model, OpenAI says they need 10GB of VRAM: https://github.com/openai/whisper#available-models-and-languages https://github.com/openai/whisper#available-models-and-langu... My DirectCompute port of that uses slightly over 4GB of VRAM, see the last column in that table: https://github.com/Const-me/Whisper/blob/master/SampleClips/summary.tsv https://github.com/Const-me/Whisper/blob/master/SampleClips/... I haven’t actually optimized memory usage I only did the DirectCompute port, the memory savings were achieved by Georgi Gerganov in whisper.cpp project, which I ported to Windows.
  - anentropic 4y ago

hyperific 4y ago

"Over 80% of this assembly is a heatsink to dissipate massive power." Not what heatsinks do.

Patrick-STH 4y ago
Good catch. Fixed

mk_stjames 4y ago

I recently saw a rack server of 8x SXM2 P100's (so, the 2016 equivalent of this) for sale for $3600 on ebay. And you can buy individual P100 pci-e versions for about $300 now. They retailed for about $7600 each back in 2016. I wonder how long it'll be until A100's depreciate that amount. I see articles from "Serve The Home" pop up now often and I've yet to see one discussing hardware that I've ever come across in... someone's home.

Patrick-STH 4y ago
STH has been around for 14 years and so we have tested these big servers from the P100 days (see our DeepLearning12 build) and newer. Home was the /home/ directory in Linux. We do have some content for homes. Check out our TinyMiniMicro series as an example.
- trvr 4y ago
  >> Home was the /home/ directory in Linux. I have never (until reading this) thought of it in that context. Was that the original meaning for the name, or did it shift over time?
  - Patrick-STH 4y ago
    It is in the "About" page :-) I always get confused by why folks ask why we have server content. The Wall Street Journal covers more than just one road in NYC.
    - trvr 4y ago
      Understood. But what about this? https://web.archive.org/web/20110201124803im_/http://www.servethehome.com/wp-content/uploads/2010/08/STH-Logo102.jpg.pagespeed.ce.ADlm7Tggpk.jpg https://web.archive.org/web/20110201124803im_/http://www.ser...
      - Patrick-STH 4y ago
        Now that is a throwback! That old logo was created with GIMP at a table outside NetApp. STH started at a time when the SMB market (and VERY high-end homes) were using Dell, HP, and IBM rackmount gear off-lease. The original STH idea came from me learning Linux as an alternative to Windows for that segment. STH was what I used to learn the rest of the market outside of work. The first product we were sent for a review was a rackmount case and that started us on the path of reviewing new gear ~2010-2011 when it was a part time blog instead of having a team of folks working on the site. I still try to keep 15-20% of our content in the SMB/ high-end home arena.

jhoelzel 4y ago

Now we just need to let the models get big enough until they can make themselves smaller again ;)

brianjking 4y ago

Soon nvidia & OpenAI will be the only ones with any money so that's fun :)