4 ms·

People have complained Nvidia dominates the market. Many were looking at their older GPU's for cheap experimentation. One idea I had was just using OpenCL to us

People have complained Nvidia dominates the market. Many were looking at their older GPU's for cheap experimentation. One idea I had was just using OpenCL to use all the cross-platform support it has. Even some FPGA's support it. Good to see GPT-2 done with shader programming. A port of the techniques used in smaller models, like TinyLlama or Gemma-2B, might lead to more experimentation with older or cheaper hardware [on-site].

bigyabai 1y ago
Nvidia dominates training moreso than inference, and mostly because of their hardware efficiency and not entirely because of CUDA. To dominate Nvidia, you have to beat their TSMC investments, beat their design chops and match their software support. The shortlist consisting of companies that can do that reads "Apple" and nobody else, which is exactly the sort of captive market Nvidia wants to conquer. OpenCL has been pretty handy for inference on older cards, but I'd argue it's relevance is waning. llama.cpp has Vulkan compute now which requires a smaller featureset for hardware to support. Many consumer devices skip OpenCL/CUDA altogether and delegate inference to an NPU library.
- tsurba 1y ago
  1.5-2 years ago I did some training for a ML paper on 4 AMD MI250x (each is essentially 2 gpus so 8 in total really, each with 64GB VRAM) on LUMI. My Jax models and the baseline PyTorch models were quite easy to set up there, and there was not a noticeable perf difference to 8x A100s (which I used for prototyping on our university cluster) in practice. Of course it’s just a random anecdote, but I don’t think nvidia is actually that much ahead.
  - pama 1y ago
    I have nothing against random anecdotes per se, but a lot of academic code does not correctly optimize computation on GPU hardware. If you can estimate by pen and paper how many FLOPs/s your code was using because of the main operations it had to do and how that number compared to the theoretical limit of bfloat16 performance on the NVIDIA GPU (about 2.6 * 10^15 for the 8 A100 IIRC) then you can see a bit better how close your code was to optimality. I have seen low effort performance scaling reach less than 1% of these theoretical numbers and people were super happy because it was sufficiently fast anyways (which is fine) and it showed the GPU utilized all the time (but with only 1% of these possible ALU doing anything useful at all times).
    - nickpsecurity 1y ago
      That is totally true. I'll add one thing to remember for those discussions is to compare the cost to what they'd use without a GPU. They'd probably have to spend $5k-20k on a multicore or NUMA-style box to get huge gains on multithreaded code. They also loose the cool factor of saying they're using a RTX. Maybe grant money if it's tied to GPU use. Between the three, it might make sense, even financial sense, to get a sub-$2000 GPU to accelerate academic code that barely uses the GPU. I'm just brainstorming here, though.
- nickpsecurity 1y ago
  My goal isnt to top Nvidia. It is to allow people with little money or export restrictions to conduct realistic experiments (or usage) on local GPU's. Especially those on older process nodes that are significantly cheaper than $200 on eBay. Maybe $50. How many sub-$100 cards could train and inference with OpenCL vs Vulkan? Which is the bigger opportunity for poor people right now if doing GPGPU projects on local hardware?
pjmlp 1y ago
CUDA dominates and almost no one cares about OpenCL, exactly because it is stuck on C99 model, never cared for Fortran, they only adopted a bytecode format when the war was already lost, SYCL/DPC++ is mostly an Intel thing and after they acquired CodePlay, Intel/AMD kind of always fail on delivering good OpenCL tooling, now they decided to reboot the whole oneAPI effort as UXL Foundation,.... NVidia has a lot to thank the competition why they happen to dominate the market.
- nickpsecurity 1y ago
  The "no one cares" market is large enough to support the existence of many competitors. Many have OpenCL support as their only thing in common. That might let us target older or cheap hardware in a cross-platform manner. I agree the competition mostly did it to themselves. I'll add Nvidia throwing GPU's at academics who built a lot of stuff on them. Classic strategy that goes back to IBM vs Burroughs.

bigyabai 1y ago

Nvidia dominates training moreso than inference, and mostly because of their hardware efficiency and not entirely because of CUDA. To dominate Nvidia, you have to beat their TSMC investments, beat their design chops and match their software support. The shortlist consisting of companies that can do that reads "Apple" and nobody else, which is exactly the sort of captive market Nvidia wants to conquer. OpenCL has been pretty handy for inference on older cards, but I'd argue it's relevance is waning. llama.cpp has Vulkan compute now which requires a smaller featureset for hardware to support. Many consumer devices skip OpenCL/CUDA altogether and delegate inference to an NPU library.

tsurba 1y ago
1.5-2 years ago I did some training for a ML paper on 4 AMD MI250x (each is essentially 2 gpus so 8 in total really, each with 64GB VRAM) on LUMI. My Jax models and the baseline PyTorch models were quite easy to set up there, and there was not a noticeable perf difference to 8x A100s (which I used for prototyping on our university cluster) in practice. Of course it’s just a random anecdote, but I don’t think nvidia is actually that much ahead.
- pama 1y ago
  I have nothing against random anecdotes per se, but a lot of academic code does not correctly optimize computation on GPU hardware. If you can estimate by pen and paper how many FLOPs/s your code was using because of the main operations it had to do and how that number compared to the theoretical limit of bfloat16 performance on the NVIDIA GPU (about 2.6 * 10^15 for the 8 A100 IIRC) then you can see a bit better how close your code was to optimality. I have seen low effort performance scaling reach less than 1% of these theoretical numbers and people were super happy because it was sufficiently fast anyways (which is fine) and it showed the GPU utilized all the time (but with only 1% of these possible ALU doing anything useful at all times).
  - nickpsecurity 1y ago
    That is totally true. I'll add one thing to remember for those discussions is to compare the cost to what they'd use without a GPU. They'd probably have to spend $5k-20k on a multicore or NUMA-style box to get huge gains on multithreaded code. They also loose the cool factor of saying they're using a RTX. Maybe grant money if it's tied to GPU use. Between the three, it might make sense, even financial sense, to get a sub-$2000 GPU to accelerate academic code that barely uses the GPU. I'm just brainstorming here, though.
nickpsecurity 1y ago
My goal isnt to top Nvidia. It is to allow people with little money or export restrictions to conduct realistic experiments (or usage) on local GPU's. Especially those on older process nodes that are significantly cheaper than $200 on eBay. Maybe $50. How many sub-$100 cards could train and inference with OpenCL vs Vulkan? Which is the bigger opportunity for poor people right now if doing GPGPU projects on local hardware?

pjmlp 1y ago

CUDA dominates and almost no one cares about OpenCL, exactly because it is stuck on C99 model, never cared for Fortran, they only adopted a bytecode format when the war was already lost, SYCL/DPC++ is mostly an Intel thing and after they acquired CodePlay, Intel/AMD kind of always fail on delivering good OpenCL tooling, now they decided to reboot the whole oneAPI effort as UXL Foundation,.... NVidia has a lot to thank the competition why they happen to dominate the market.

nickpsecurity 1y ago
The "no one cares" market is large enough to support the existence of many competitors. Many have OpenCL support as their only thing in common. That might let us target older or cheap hardware in a cross-platform manner. I agree the competition mostly did it to themselves. I'll add Nvidia throwing GPU's at academics who built a lot of stuff on them. Classic strategy that goes back to IBM vs Burroughs.