5 ms·

I wonder, is the dual-use of GPUs a significant engineering bottleneck for Nvidia et al? My understanding is that for libraries like Tensorflow and PyTorch, muc

I wonder, is the dual-use of GPUs a significant engineering bottleneck for Nvidia et al? My understanding is that for libraries like Tensorflow and PyTorch, much of the complexity is supporting the many-to-many mapping of model structures to GPU silicon in a way that reduces “compile time”, with all the tradeoff landmines you might expect there. I would imagine that abstracting this at the hardware layer is really valuable. Are these non-GPU architectures the future of machine learning, and can we expect the big vendors to start competing here or is this a business risk for them? (robbing Peter to pay Paul, like Google investing in AI that cannibalizes their search revenue)

dannyw 3y ago
I doubt it. Toolkits like TensorRT are specifically optimised, and NVIDIA already segregated their HPC vs gaming dies (and corresponding, die space). The H100s feature a minuscule amount of ROPs and TMUs (mainly for graphics); and doesn’t come with NVENC/DEC, etc. DirectX, Vulkan is not supported. You’ll just see more and more tensor cores, optimised CUDA cores for bfloat, etc.
- paulmd 3y ago
  H100 does have NVDEC and jpeg ingest accelerators https://www.servethehome.com/wp-content/uploads/2023/10/NVIDIA-H100-SXM-PCIe-and-NVL-Spec-Table.jpg https://www.servethehome.com/wp-content/uploads/2023/10/NVID... Tbh it’s mildly surprising they even removed NVENC considering the overall size of the chip (in the absolute we are only talking about low-single-digit mm2 savings) and then H100 is still advertised and has features targeting VM graphics/visualization still… remember they also still put a full graphics pipeline with ROPs/TMUs on the chip, just no actual display hardware.
  - lobochrome 3y ago
    It's a tradeoff with verification and validation, not so much die-space. If you include the IP, you need to run an expensive (time-consuming) simulation and emulation for the IP. Then, you need to validate it on first silicon and pray that it doesn't impact yield in HVM. These days, simulation runs basically nonstop on all the FPGAs the company is able to throw at it until tape out. Then, in post-silicon validation, you have engineering teams working 24/7 to validate the device on the bench. Finally, test engineering tries to achieve yield in HVM. Anything that you don't need on the spec sheet or the designer insists on is cut to save time to market.
  - eoerl 3y ago
    These can also be used for machine learning actually (see Dali for data loading for instance)
smoldesu 3y ago
The dual-use of GPUs is what's driving Nvidia's demand, for the most part. They bet big on more outlandish designs (eg. CUDA/Tensor cores) and mixed-precision computation, and did a lot of the hardware/software work to get it shipped out years ago. There wasn't much interest in CUDA outside niche research/industry applications for a while, it's a small miracle they kept it around long enough to see the crypto and AI booms. Now their bet is paying big dividends, and other companies are trying to figure out fast ways to leverage raster compute for general-purpose acceleration. Apple initially developed OpenCL with Khronos as a similar-ish analog for GPU acceleration a while ago. There were a few partners that invested in it, but it suffered the same lack of demand as CUDA and languished for a while. Now Apple doesn't support OpenCL, so the purpose of a multi-platform acceleration library has kinda been scuttled. Nvidia played their cards well, and their competitors are going to feel the pain for a while unless they work together again.
weebull 3y ago
> I wonder, is the dual-use of GPUs a significant engineering bottleneck for Nvidia et al? Not much. GPUs became array processing engines in the early 2000s and the amount of residual fixed function graphics hardware is tiny now. The biggest thing is that a lot of network evaluation is being done at lower precision (BF16/ FP8 / ternary) now, but my understanding is that training the networks still requires FP32. As these companies are really targeting the training market the high precision modes needed for graphics can't be thrown away. If a company was to target the end user, then you could optimise for lower precision, but end users find dedicated neural net acceleration a luxury they can get for free by buying a GPU.

dannyw 3y ago

I doubt it. Toolkits like TensorRT are specifically optimised, and NVIDIA already segregated their HPC vs gaming dies (and corresponding, die space). The H100s feature a minuscule amount of ROPs and TMUs (mainly for graphics); and doesn’t come with NVENC/DEC, etc. DirectX, Vulkan is not supported. You’ll just see more and more tensor cores, optimised CUDA cores for bfloat, etc.

paulmd 3y ago
H100 does have NVDEC and jpeg ingest accelerators https://www.servethehome.com/wp-content/uploads/2023/10/NVIDIA-H100-SXM-PCIe-and-NVL-Spec-Table.jpg https://www.servethehome.com/wp-content/uploads/2023/10/NVID... Tbh it’s mildly surprising they even removed NVENC considering the overall size of the chip (in the absolute we are only talking about low-single-digit mm2 savings) and then H100 is still advertised and has features targeting VM graphics/visualization still… remember they also still put a full graphics pipeline with ROPs/TMUs on the chip, just no actual display hardware.
- lobochrome 3y ago
  It's a tradeoff with verification and validation, not so much die-space. If you include the IP, you need to run an expensive (time-consuming) simulation and emulation for the IP. Then, you need to validate it on first silicon and pray that it doesn't impact yield in HVM. These days, simulation runs basically nonstop on all the FPGAs the company is able to throw at it until tape out. Then, in post-silicon validation, you have engineering teams working 24/7 to validate the device on the bench. Finally, test engineering tries to achieve yield in HVM. Anything that you don't need on the spec sheet or the designer insists on is cut to save time to market.
- eoerl 3y ago
  These can also be used for machine learning actually (see Dali for data loading for instance)

smoldesu 3y ago

The dual-use of GPUs is what's driving Nvidia's demand, for the most part. They bet big on more outlandish designs (eg. CUDA/Tensor cores) and mixed-precision computation, and did a lot of the hardware/software work to get it shipped out years ago. There wasn't much interest in CUDA outside niche research/industry applications for a while, it's a small miracle they kept it around long enough to see the crypto and AI booms. Now their bet is paying big dividends, and other companies are trying to figure out fast ways to leverage raster compute for general-purpose acceleration. Apple initially developed OpenCL with Khronos as a similar-ish analog for GPU acceleration a while ago. There were a few partners that invested in it, but it suffered the same lack of demand as CUDA and languished for a while. Now Apple doesn't support OpenCL, so the purpose of a multi-platform acceleration library has kinda been scuttled. Nvidia played their cards well, and their competitors are going to feel the pain for a while unless they work together again.

weebull 3y ago

> I wonder, is the dual-use of GPUs a significant engineering bottleneck for Nvidia et al? Not much. GPUs became array processing engines in the early 2000s and the amount of residual fixed function graphics hardware is tiny now. The biggest thing is that a lot of network evaluation is being done at lower precision (BF16/ FP8 / ternary) now, but my understanding is that training the networks still requires FP32. As these companies are really targeting the training market the high precision modes needed for graphics can't be thrown away. If a company was to target the end user, then you could optimise for lower precision, but end users find dedicated neural net acceleration a luxury they can get for free by buying a GPU.