6 ms·

What lengths will Chinese companies go to get an Nvidia A100 chip?

60 points by yorwba 3y ago

wtcactus 3y ago
Interesting, I didn't know this was so sought after. I actually have one for sale (the 40GB PCIe one), but I haven't got to list it on eBay due to lack of time yet (and because I didn't think there was so much interest in it). To be sincere, maybe for DL this is really much better than the alternatives, but for some simulations and parallelizing come radiative transfer code, it was not that much better than a RTX 4090 with the extra hassle that it's more difficult to cool it.
- moondev 3y ago
  How did you end up cooling it? I have an a40 and it's been interesting testing all kinds of methods, from two 40mm fans to a 3A 9030 centrifugal blower with 3d printed duct
- siver_john 3y ago
  As someone who used various types of GPUs in graduate school. For most simulations, and even machine learning (unless you need the VRAM) you are generally better off going with a consumer card. There is generally about the same number of CUDA cores and the higher clock speeds will generally net you better performance overall. Simulations where this isn't true is any that need double floating point (which you previously were able to do in the Titan series of consumer-ish cards). And where it is super important for DL is the VRAM it allows you to use much larger models. Plus the added features of being able to string them together and share memory which is an important feature that has been left off consumer cards (honestly in a way that makes sense because SLI has been dumb for some time).
- rhdunn 3y ago
  The A100 is comparable to the 3090 but with more memory. The H100 is the one comparable to the 4090. The advantage of these is the access to the larger memory. And they are able to be linked together such that they all share the same memory via NVLink. This makes them scalable for processing the large data and holding the models for the larger scale LLMs and other NN/ML based models.
  - tmaly 3y ago
    Have you seen an actual A100? They are massive, I can imagine them being comparable to a 3090 at all.
    - Feorn 3y ago
      A reference 3090 is longer by 69mm, wider by 29mm, and thicker by a slot than a PCIe A100. Though I think the comment you're replying to was talking about them both using the same Nvidia GPU architecture, Ampere.
  - adfbkandfionio 3y ago
    >And they are able to be linked together such that they all share the same memory via NVLink. This makes them scalable for processing the large data and holding the models for the larger scale LLMs and other NN/ML based models. GPUs connected with NVLink do not exactly share memory. They don't look like a single logical GPU. One GPU can issue loads or stores to a different GPU's memory using "GPUDirect Peer-To-Peer", but you cannot have a single buffer or a single kernel that spans multiple GPUs. This is easier to use and more powerful than the previous system of explicit copies from device to device, perhaps, but a far cry from the way multiple CPU sockets "just work". Even if you could treat the system as one big GPU you wouldn't want to. The performance takes a serious hit if you constantly access off-device memory. NVLink doesn't open up any functionality that isn't available over PCIe, as far as I know. It's "merely" a performance improvement. The peer-to-peer technology still works without NVLink. NVidia's docs are, as always, confusing at best. There are several similarly-named technologies. The main documentation page just says "email us for more info". The best online documentation I've found is in some random slides. https://developer.nvidia.com/gpudirect https://developer.nvidia.com/gpudirect https://developer.download.nvidia.com/CUDA/training/cuda_webinars_GPUDirect_uva.pdf https://developer.download.nvidia.com/CUDA/training/cuda_web...
    - rhdunn 3y ago
      Interesting. So that would mean that you would still need a 40 or 80 GB card to run the larger models (30B LLM, 70B LLM, 8x7B LLM) and perform training of them. Or would it be possible to split the model layers between the cards like you can between RAM and VRAM? I suppose in that case each card would be able to evaluate the results of the layers in its own memory and then pass those results to the other card(s) as necessary.
dachworker 3y ago
Here's an interesting question: why are there lots of startups claiming they can create better bespoke DL hardware accelerators than Nvidia's offerings? I understand how a bespoke architecture could in theory accommodate larger models and offer better throughput despite being based on older generations of MOFSET nodes. But if that was the case, wouldn't China simply create their own bespoke hardware accelerators? So what's stopping them?
- russli1993 3y ago
  hint: they are
- alephnerd 3y ago
  > wouldn't China simply create their own bespoke hardware accelerators? So what's stopping them? Fabrication. SMEE is supposedly able to mass produce 28nm lithography machines, but most modern (2016/Pascal onwards) GPUs are fabricated using 16nm lithography or lower (eg. Ampere uses a 7nm process, and there are multiple newer architectures in the pipeline that leverage 3nm fabrication processes at Samsung, TSMC, and Intel). Chinese companies like SMEE are trying, but it will take 3-5 years to reach 16nm lithography at scale. Also, GPUs are being limited for simulation (aka Nuclear weapons testing) reasons - not "AI" - as just about every country except North Korea honors the Comprehensive Nuclear Test Ban Treaty, forcing countries to test using HPC (edit: also used for Jet Turbine simulation a la Autodesk Federal). A 28nm process is more than enough to make EWS, Avionics, and Precision weapons, which is what Russia uses to manufacture the Elbus-8s chipset domestically. This is partially related to why companies like Nvidia has begun moving fabrication to Samsung fabs over TSMC in the short term, as SK has a formal defense agreement unlike Taiwan.
  - mistrial9 3y ago
    > GPUs are being limited for simulation (aka Nuclear weapons testing) reasons - not "AI" that information seems dated in 2024
    - alephnerd 3y ago
elzbardico 3y ago
If money is not a problem for them, I don't see how can we deny Chinese companies access to those cards. The soviets were able to smuggle whole mainframes and mini-computers during the 70s. Giant fucking machines that you could only buy trough very specific sales channels. It was not like you could ebay yourself a S/360, and yet, they did it.
- The_Colonel 3y ago
  It's not a question of "if" , but "how many". Abd that's largely what matters, same as in the case of USSR.
  - BriggyDwiggs42 3y ago
    Wait why? Do you really think the number could be reduced below the requisite amount to do good reverse engineering? That number can’t be more than a couple dozen, and good luck preventing that.
    - The_Colonel 3y ago
      Sure, they can reverse engineer it, at a great cost. The production costs will be then much higher than what TSMC charges as well. In the end everything can be reduce to question of "how much will it cost?". It was the same in USSR. They also managed to reverse engineer many western chips and produce clones. But the production wasn't economical and they were a couple of years behind anyway.
- __loam 3y ago
  Asianometry on YouTube has a ton of videos on Soviet computing including one's discussing things like the failed semiconductor industry in East Germany. The Soviets certainly smuggled and tried to clone a lot of machines but they were always behind the United States.
  - Tostino 3y ago
    I love that channel. It's a mix of topics that I find very, very interesting.
tibbydudeza 3y ago
I have seen some reports of modded GPU's in Brazil and China where they convert earlier gen consumer cards (3000 series) into 12GB or 16GB variants using GDDR6 from Samsung. Seems the GPU boot code is not hard limited but reads capacity from what is installed.

60 points by yorwba 3y ago

Interesting, I didn't know this was so sought after. I actually have one for sale (the 40GB PCIe one), but I haven't got to list it on eBay due to lack of time yet (and because I didn't think there was so much interest in it). To be sincere, maybe for DL this is really much better than the alternatives, but for some simulations and parallelizing come radiative transfer code, it was not that much better than a RTX 4090 with the extra hassle that it's more difficult to cool it.

moondev 3y ago
How did you end up cooling it? I have an a40 and it's been interesting testing all kinds of methods, from two 40mm fans to a 3A 9030 centrifugal blower with 3d printed duct
siver_john 3y ago
As someone who used various types of GPUs in graduate school. For most simulations, and even machine learning (unless you need the VRAM) you are generally better off going with a consumer card. There is generally about the same number of CUDA cores and the higher clock speeds will generally net you better performance overall. Simulations where this isn't true is any that need double floating point (which you previously were able to do in the Titan series of consumer-ish cards). And where it is super important for DL is the VRAM it allows you to use much larger models. Plus the added features of being able to string them together and share memory which is an important feature that has been left off consumer cards (honestly in a way that makes sense because SLI has been dumb for some time).
rhdunn 3y ago
The A100 is comparable to the 3090 but with more memory. The H100 is the one comparable to the 4090. The advantage of these is the access to the larger memory. And they are able to be linked together such that they all share the same memory via NVLink. This makes them scalable for processing the large data and holding the models for the larger scale LLMs and other NN/ML based models.
- tmaly 3y ago
  Have you seen an actual A100? They are massive, I can imagine them being comparable to a 3090 at all.
  - Feorn 3y ago
    A reference 3090 is longer by 69mm, wider by 29mm, and thicker by a slot than a PCIe A100. Though I think the comment you're replying to was talking about them both using the same Nvidia GPU architecture, Ampere.
- adfbkandfionio 3y ago
  >And they are able to be linked together such that they all share the same memory via NVLink. This makes them scalable for processing the large data and holding the models for the larger scale LLMs and other NN/ML based models. GPUs connected with NVLink do not exactly share memory. They don't look like a single logical GPU. One GPU can issue loads or stores to a different GPU's memory using "GPUDirect Peer-To-Peer", but you cannot have a single buffer or a single kernel that spans multiple GPUs. This is easier to use and more powerful than the previous system of explicit copies from device to device, perhaps, but a far cry from the way multiple CPU sockets "just work". Even if you could treat the system as one big GPU you wouldn't want to. The performance takes a serious hit if you constantly access off-device memory. NVLink doesn't open up any functionality that isn't available over PCIe, as far as I know. It's "merely" a performance improvement. The peer-to-peer technology still works without NVLink. NVidia's docs are, as always, confusing at best. There are several similarly-named technologies. The main documentation page just says "email us for more info". The best online documentation I've found is in some random slides. https://developer.nvidia.com/gpudirect https://developer.nvidia.com/gpudirect https://developer.download.nvidia.com/CUDA/training/cuda_webinars_GPUDirect_uva.pdf https://developer.download.nvidia.com/CUDA/training/cuda_web...
  - rhdunn 3y ago
    Interesting. So that would mean that you would still need a 40 or 80 GB card to run the larger models (30B LLM, 70B LLM, 8x7B LLM) and perform training of them. Or would it be possible to split the model layers between the cards like you can between RAM and VRAM? I suppose in that case each card would be able to evaluate the results of the layers in its own memory and then pass those results to the other card(s) as necessary.

dachworker 3y ago

Here's an interesting question: why are there lots of startups claiming they can create better bespoke DL hardware accelerators than Nvidia's offerings? I understand how a bespoke architecture could in theory accommodate larger models and offer better throughput despite being based on older generations of MOFSET nodes. But if that was the case, wouldn't China simply create their own bespoke hardware accelerators? So what's stopping them?

russli1993 3y ago
hint: they are
alephnerd 3y ago
> wouldn't China simply create their own bespoke hardware accelerators? So what's stopping them? Fabrication. SMEE is supposedly able to mass produce 28nm lithography machines, but most modern (2016/Pascal onwards) GPUs are fabricated using 16nm lithography or lower (eg. Ampere uses a 7nm process, and there are multiple newer architectures in the pipeline that leverage 3nm fabrication processes at Samsung, TSMC, and Intel). Chinese companies like SMEE are trying, but it will take 3-5 years to reach 16nm lithography at scale. Also, GPUs are being limited for simulation (aka Nuclear weapons testing) reasons - not "AI" - as just about every country except North Korea honors the Comprehensive Nuclear Test Ban Treaty, forcing countries to test using HPC (edit: also used for Jet Turbine simulation a la Autodesk Federal). A 28nm process is more than enough to make EWS, Avionics, and Precision weapons, which is what Russia uses to manufacture the Elbus-8s chipset domestically. This is partially related to why companies like Nvidia has begun moving fabrication to Samsung fabs over TSMC in the short term, as SK has a formal defense agreement unlike Taiwan.
- mistrial9 3y ago
  > GPUs are being limited for simulation (aka Nuclear weapons testing) reasons - not "AI" that information seems dated in 2024
  - alephnerd 3y ago

elzbardico 3y ago

If money is not a problem for them, I don't see how can we deny Chinese companies access to those cards. The soviets were able to smuggle whole mainframes and mini-computers during the 70s. Giant fucking machines that you could only buy trough very specific sales channels. It was not like you could ebay yourself a S/360, and yet, they did it.

The_Colonel 3y ago
It's not a question of "if" , but "how many". Abd that's largely what matters, same as in the case of USSR.
- BriggyDwiggs42 3y ago
  Wait why? Do you really think the number could be reduced below the requisite amount to do good reverse engineering? That number can’t be more than a couple dozen, and good luck preventing that.
  - The_Colonel 3y ago
    Sure, they can reverse engineer it, at a great cost. The production costs will be then much higher than what TSMC charges as well. In the end everything can be reduce to question of "how much will it cost?". It was the same in USSR. They also managed to reverse engineer many western chips and produce clones. But the production wasn't economical and they were a couple of years behind anyway.
__loam 3y ago
Asianometry on YouTube has a ton of videos on Soviet computing including one's discussing things like the failed semiconductor industry in East Germany. The Soviets certainly smuggled and tried to clone a lot of machines but they were always behind the United States.
- Tostino 3y ago
  I love that channel. It's a mix of topics that I find very, very interesting.

tibbydudeza 3y ago

I have seen some reports of modded GPU's in Brazil and China where they convert earlier gen consumer cards (3000 series) into 12GB or 16GB variants using GDDR6 from Samsung. Seems the GPU boot code is not hard limited but reads capacity from what is installed.