5 ms·

Nvidia reveals new A.I. chip, says costs of running LLMs will drop significantly

73 points by TalktoCrystal 3y ago

jerojero 3y ago
Although this is obviously great news, it is increasingly troubling how AI innovation (from the hardware sector) is pretty much limited to Nvidia. I haven't really gotten too much into it, so I'm not sure how has Nvidia come to absolutely dominate this market; AMD GPUs are quite good in the gaming sector... though I guess... with just two real players in the GPU market it's difficult to really get anywhere.
- sesteel 3y ago
  The article talks about AMD announcing a chip supporting 192GB memory.
  - pbronez 3y ago
    AMD’s hardware is essentially irrelevant for non-gaming applications because the software stack isn’t there. Unfortunately same goes for just about everyone BUT NVIDIA. It’s a situation that could be fixed, but it would require cooperative investment from, well, the rest of the semiconductor industry.
    - nullc 3y ago
      The fact that a large number of people care about a small number of ML models right now means that AMD has an opportunity to port just a few things and take market share.
- pjmlp 3y ago
  By having modern tooling to program GPGPU, allowing several languages to target them instead of plain C dialect, IDE and graphical debuggers to track down issues on GPGPU just like on the CPU, and a myriad of good quality libraries. Meanwhile Intel and AMD, never delivered something at the same level as CUDA for OpenCL, and when they finally decided to react with SPIR and C++, everyone was already too busy to care, and it isn't as if the tooling has improved that much. Hence why OpenCL 3.0 is basically OpenCL 1.0 rebranded.
- faeriechangling 3y ago
  Nvidia in a very real way built the market by providing not just the hardware but the APIs and the tooling and so on. The rest of the market has been reacting to them for a long time. Can't say they don't deserve their success in a capitalist sense. Can't say their success doesn't alarm me.
  - PartiallyTyped 3y ago
    They literally built their success and shaped the future of humanity… Our architectures are what they are because they run well on nVidia GPUs, same for our optimisers. What run well set a direction, the hardware got optimised and so did the software in a beautiful feedback loop.
  - shapefrog 3y ago
    > The rest of the market has been reacting to them for a long time The rest of the market has not been reacting to them for a long time.
- phero_cnstrcts 3y ago
  Competition will have to come from somewhere else than amd since the CEOs are literally family.
- Havoc 3y ago
  >I'm not sure how has Nvidia come to absolutely dominate this market They had the foresight to see the coming wave and ensure they have both best hardware and best software stack. What puzzles me more is why AMD isn't throwing money at their software side to sort that out. Rocm was launched in 2016. 7 years later running pytorch on their flagship GPU still requires a janky laundry list of crowdsourced instructions: https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/9591 https://github.com/AUTOMATIC1111/stable-diffusion-webui/disc...
DarthNebo 3y ago
JM2C lood_in_4bit=True will let you run Llama2-7B variants at 6.3GB VRAM.
skavi 3y ago
The GH200 was announced in May. What’s new is a variant with HBM3e memory which increases capacity (96GB -> 144GB) and bandwidth (4TB/s -> 5TB/s). The posted article butchers this. Here’s a much better source: https://www.anandtech.com/show/20001/nvidia-unveils-gh200-grace-hopper-gpu-with-hbm3e-memory https://www.anandtech.com/show/20001/nvidia-unveils-gh200-gr...
iFire 3y ago
Regarding NVIDIA domination, I want to promote that the Google project https://github.com/openxla/iree https://github.com/openxla/iree exists and IREE acts as a way to turn Tensorflow, Pytorch, and MLIR workflows to compute on cpu, vulkan compute, cuda, rocm, metal and others. https://github.com/RechieKho/IREE.gd https://github.com/RechieKho/IREE.gd -- RechieKho and I collaborate on making this work for Godot Engine, but IREE.gd is at a proof of concept stage.

73 points by TalktoCrystal 3y ago

Although this is obviously great news, it is increasingly troubling how AI innovation (from the hardware sector) is pretty much limited to Nvidia. I haven't really gotten too much into it, so I'm not sure how has Nvidia come to absolutely dominate this market; AMD GPUs are quite good in the gaming sector... though I guess... with just two real players in the GPU market it's difficult to really get anywhere.

sesteel 3y ago
The article talks about AMD announcing a chip supporting 192GB memory.
- pbronez 3y ago
  AMD’s hardware is essentially irrelevant for non-gaming applications because the software stack isn’t there. Unfortunately same goes for just about everyone BUT NVIDIA. It’s a situation that could be fixed, but it would require cooperative investment from, well, the rest of the semiconductor industry.
  - nullc 3y ago
    The fact that a large number of people care about a small number of ML models right now means that AMD has an opportunity to port just a few things and take market share.
pjmlp 3y ago
By having modern tooling to program GPGPU, allowing several languages to target them instead of plain C dialect, IDE and graphical debuggers to track down issues on GPGPU just like on the CPU, and a myriad of good quality libraries. Meanwhile Intel and AMD, never delivered something at the same level as CUDA for OpenCL, and when they finally decided to react with SPIR and C++, everyone was already too busy to care, and it isn't as if the tooling has improved that much. Hence why OpenCL 3.0 is basically OpenCL 1.0 rebranded.
faeriechangling 3y ago
Nvidia in a very real way built the market by providing not just the hardware but the APIs and the tooling and so on. The rest of the market has been reacting to them for a long time. Can't say they don't deserve their success in a capitalist sense. Can't say their success doesn't alarm me.
- PartiallyTyped 3y ago
  They literally built their success and shaped the future of humanity… Our architectures are what they are because they run well on nVidia GPUs, same for our optimisers. What run well set a direction, the hardware got optimised and so did the software in a beautiful feedback loop.
- shapefrog 3y ago
  > The rest of the market has been reacting to them for a long time The rest of the market has not been reacting to them for a long time.
phero_cnstrcts 3y ago
Competition will have to come from somewhere else than amd since the CEOs are literally family.
Havoc 3y ago
>I'm not sure how has Nvidia come to absolutely dominate this market They had the foresight to see the coming wave and ensure they have both best hardware and best software stack. What puzzles me more is why AMD isn't throwing money at their software side to sort that out. Rocm was launched in 2016. 7 years later running pytorch on their flagship GPU still requires a janky laundry list of crowdsourced instructions: https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/9591 https://github.com/AUTOMATIC1111/stable-diffusion-webui/disc...

DarthNebo 3y ago

JM2C lood_in_4bit=True will let you run Llama2-7B variants at 6.3GB VRAM.

skavi 3y ago

The GH200 was announced in May. What’s new is a variant with HBM3e memory which increases capacity (96GB -> 144GB) and bandwidth (4TB/s -> 5TB/s). The posted article butchers this. Here’s a much better source: https://www.anandtech.com/show/20001/nvidia-unveils-gh200-grace-hopper-gpu-with-hbm3e-memory https://www.anandtech.com/show/20001/nvidia-unveils-gh200-gr...

iFire 3y ago

Regarding NVIDIA domination, I want to promote that the Google project https://github.com/openxla/iree https://github.com/openxla/iree exists and IREE acts as a way to turn Tensorflow, Pytorch, and MLIR workflows to compute on cpu, vulkan compute, cuda, rocm, metal and others. https://github.com/RechieKho/IREE.gd https://github.com/RechieKho/IREE.gd -- RechieKho and I collaborate on making this work for Godot Engine, but IREE.gd is at a proof of concept stage.