3 ms·

This seems to have cause and effect backwards. LLMs emerged because of Nvidia.

dagmx 2y ago
LLMs absolutely did not emerge because of NVIDIA. you’re the one imho who is mistaking correlation with causality. The first transformer models were developed at Google. NVIDIA were the card du jour for accelerating it in the years since and have contributed research too, but your statement goes way too far
- BobbyJo 2y ago
  I think their point is that easily accessible hardware capable of supporting the research is the reason the research has come as far as it has, and I would tend to agree. At the very least, GPUs keeping the PCI ecosystem going has played a major role in allowing the specialized accelerator market to flourish.
  - dagmx 2y ago
    But that could apply to any of the GPU manufacturers. CUDA made for an easier ecosystem but if it didn’t exist it would have been any of the other APIs. The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I’m not trying to downplay NVIDIA but they specifically mentioned cause and effect, and then said it was because of NVIDIA.
    - KaoruAoiShiho 2y ago
      First transformer != LLM. Imagine a world where you had to use AMD or CPUs, there would be no AlexNet, there would be no LLMs. Nvidia seeded universities with gifted hardware accelerators for a over a decade. Nvidia built the foundations for modern ML on which transformer lies, it's just one step on a long road to LLMs.
      - dagmx 2y ago
        AMD had GPU compute APIs at the time as well. They also used to contribute GPUs (albeit in much smaller quantities). They just ended up killing them in favor of OpenCL which then withered on the vine. NVIDIA absolutely contributed to the foundation but they are not the foundation alone. Alexnet was great research but they could have done the same on other vendors at the time too. The hardware didn’t exist in a vacuum.
        physicsguy 2y ago
        OpenCL itself is kinda fine, but the libraries never existed because every academic from postdoc up could get a free NVidia card. You literally filled a form out and they sent you it.
        dagmx 2y ago
        I agree the library situation for OpenCL never stood up. Khronos tried at the start but then lost steam. It’s one of the projects I think the Khronos group mishandled the most unfortunately.
        DeathArrow 2y ago
        Do they have a project that achieved success?
        echoangle 2y ago
        OpenGL, Vulkan, WebGL?
        dagmx 2y ago
        Here’s a list of projects https://www.khronos.org/ https://www.khronos.org/ If I were to categorize the successful ones: glTF, KTX, SPIR-V, OpenGL and its variants, WebGL People will say Vulkan but it has the same level of adoption as OpenCL, and has the same issue that it competes against vendor specific APIs (DX and Metal) that are just better to use. It’s still used though of course as a translation target but imho that doesn’t qualify it as a success. OpenCL was and is a failure of grand magnitude. As was colada.
    - BobbyJo 2y ago
      > The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I graduated college in 2010 and I took a class taught in CUDA before graduating. CUDA was a primary driver of NN research at the time. Sure, other tools were available, but CUDA allowed people to build and distribute actually useful software which further encouraged the space. Could things have happened without it? Yeah, for sure, but it would have taken a good deal longer.
- llm_trw 2y ago
  I was around the space before Alexnet came out. Without NVidia the last 15 years of AI would not have happened. Deep learning was only possible because you could do it on NVidia cards with Cuda without having to use the big machines in the basement. Trying to convince anyone that neural networks could be useful in 2009 was impossible - I got a grant declined and my PhD supervisor told me to drop the useless tech and focus on something better like support vector machines.
  - dagmx 2y ago
    I was around then too and people forget that AMD also had GPU compute APIs. The difference is AMD killed theirs to use OpenCL and NVIDIA kept CUDA around as well.
    - llm_trw 2y ago
      Just like how ROCm is supposed to be competitive today and it isn't unless you have an army of grad students to look after your data center cards. I tried using AMD Stream and it lacked documentation, debugging information and most of the tools needed to get anything done without a large team of experts. NVidia by comparison could - and was - used by single grad students on their franken-stations which we build out of gaming GPUs. The less we talk about the disaster that the move to opencl was the better.
      - dagmx 2y ago
        I agree ROCm was a mess and only recently is even really usable. That said ROCm is quite a recent thing borne out of acknowledging that OpenCL 2 was a disaster. OpenCL 1 had a reasonable shot and was gaining adoption but 2 scuppered it.
        llm_trw 2y ago
        Did you ever train anything past mnist on AMD? I try every five years or so and promise myself never again each and every time.

LLMs absolutely did not emerge because of NVIDIA. you’re the one imho who is mistaking correlation with causality. The first transformer models were developed at Google. NVIDIA were the card du jour for accelerating it in the years since and have contributed research too, but your statement goes way too far

BobbyJo 2y ago
I think their point is that easily accessible hardware capable of supporting the research is the reason the research has come as far as it has, and I would tend to agree. At the very least, GPUs keeping the PCI ecosystem going has played a major role in allowing the specialized accelerator market to flourish.
- dagmx 2y ago
  But that could apply to any of the GPU manufacturers. CUDA made for an easier ecosystem but if it didn’t exist it would have been any of the other APIs. The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I’m not trying to downplay NVIDIA but they specifically mentioned cause and effect, and then said it was because of NVIDIA.
  - KaoruAoiShiho 2y ago
    First transformer != LLM. Imagine a world where you had to use AMD or CPUs, there would be no AlexNet, there would be no LLMs. Nvidia seeded universities with gifted hardware accelerators for a over a decade. Nvidia built the foundations for modern ML on which transformer lies, it's just one step on a long road to LLMs.
    - dagmx 2y ago
      AMD had GPU compute APIs at the time as well. They also used to contribute GPUs (albeit in much smaller quantities). They just ended up killing them in favor of OpenCL which then withered on the vine. NVIDIA absolutely contributed to the foundation but they are not the foundation alone. Alexnet was great research but they could have done the same on other vendors at the time too. The hardware didn’t exist in a vacuum.
      - physicsguy 2y ago
        OpenCL itself is kinda fine, but the libraries never existed because every academic from postdoc up could get a free NVidia card. You literally filled a form out and they sent you it.
        dagmx 2y ago
        I agree the library situation for OpenCL never stood up. Khronos tried at the start but then lost steam. It’s one of the projects I think the Khronos group mishandled the most unfortunately.
        DeathArrow 2y ago
        Do they have a project that achieved success?
        echoangle 2y ago
        OpenGL, Vulkan, WebGL?
        dagmx 2y ago
        Here’s a list of projects https://www.khronos.org/ https://www.khronos.org/ If I were to categorize the successful ones: glTF, KTX, SPIR-V, OpenGL and its variants, WebGL People will say Vulkan but it has the same level of adoption as OpenCL, and has the same issue that it competes against vendor specific APIs (DX and Metal) that are just better to use. It’s still used though of course as a translation target but imho that doesn’t qualify it as a success. OpenCL was and is a failure of grand magnitude. As was colada.
  - BobbyJo 2y ago
    > The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I graduated college in 2010 and I took a class taught in CUDA before graduating. CUDA was a primary driver of NN research at the time. Sure, other tools were available, but CUDA allowed people to build and distribute actually useful software which further encouraged the space. Could things have happened without it? Yeah, for sure, but it would have taken a good deal longer.
llm_trw 2y ago
I was around the space before Alexnet came out. Without NVidia the last 15 years of AI would not have happened. Deep learning was only possible because you could do it on NVidia cards with Cuda without having to use the big machines in the basement. Trying to convince anyone that neural networks could be useful in 2009 was impossible - I got a grant declined and my PhD supervisor told me to drop the useless tech and focus on something better like support vector machines.
- dagmx 2y ago
  I was around then too and people forget that AMD also had GPU compute APIs. The difference is AMD killed theirs to use OpenCL and NVIDIA kept CUDA around as well.
  - llm_trw 2y ago
    Just like how ROCm is supposed to be competitive today and it isn't unless you have an army of grad students to look after your data center cards. I tried using AMD Stream and it lacked documentation, debugging information and most of the tools needed to get anything done without a large team of experts. NVidia by comparison could - and was - used by single grad students on their franken-stations which we build out of gaming GPUs. The less we talk about the disaster that the move to opencl was the better.
    - dagmx 2y ago
      I agree ROCm was a mess and only recently is even really usable. That said ROCm is quite a recent thing borne out of acknowledging that OpenCL 2 was a disaster. OpenCL 1 had a reasonable shot and was gaining adoption but 2 scuppered it.
      - llm_trw 2y ago
        Did you ever train anything past mnist on AMD? I try every five years or so and promise myself never again each and every time.