3 ms·

I think their point is that easily accessible hardware capable of supporting the research is the reason the research has come as far as it has, and I would tend

I think their point is that easily accessible hardware capable of supporting the research is the reason the research has come as far as it has, and I would tend to agree. At the very least, GPUs keeping the PCI ecosystem going has played a major role in allowing the specialized accelerator market to flourish.

dagmx 2y ago
But that could apply to any of the GPU manufacturers. CUDA made for an easier ecosystem but if it didn’t exist it would have been any of the other APIs. The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I’m not trying to downplay NVIDIA but they specifically mentioned cause and effect, and then said it was because of NVIDIA.
- KaoruAoiShiho 2y ago
  First transformer != LLM. Imagine a world where you had to use AMD or CPUs, there would be no AlexNet, there would be no LLMs. Nvidia seeded universities with gifted hardware accelerators for a over a decade. Nvidia built the foundations for modern ML on which transformer lies, it's just one step on a long road to LLMs.
  - dagmx 2y ago
    AMD had GPU compute APIs at the time as well. They also used to contribute GPUs (albeit in much smaller quantities). They just ended up killing them in favor of OpenCL which then withered on the vine. NVIDIA absolutely contributed to the foundation but they are not the foundation alone. Alexnet was great research but they could have done the same on other vendors at the time too. The hardware didn’t exist in a vacuum.
    - physicsguy 2y ago
      OpenCL itself is kinda fine, but the libraries never existed because every academic from postdoc up could get a free NVidia card. You literally filled a form out and they sent you it.
      - dagmx 2y ago
        I agree the library situation for OpenCL never stood up. Khronos tried at the start but then lost steam. It’s one of the projects I think the Khronos group mishandled the most unfortunately.
        DeathArrow 2y ago
        Do they have a project that achieved success?
        echoangle 2y ago
        OpenGL, Vulkan, WebGL?
        dagmx 2y ago
        Here’s a list of projects https://www.khronos.org/ https://www.khronos.org/ If I were to categorize the successful ones: glTF, KTX, SPIR-V, OpenGL and its variants, WebGL People will say Vulkan but it has the same level of adoption as OpenCL, and has the same issue that it competes against vendor specific APIs (DX and Metal) that are just better to use. It’s still used though of course as a translation target but imho that doesn’t qualify it as a success. OpenCL was and is a failure of grand magnitude. As was colada.
    - KaoruAoiShiho 2y ago
      Alexnet could not have been done on AMD, nope.
      - dagmx 2y ago
        Please, this is just revisionism. There’s nothing inherent to AlexNet that relied on NVIDIA hardware or even software. It was just what happened to be available and most straightforward at the time. To say it wouldn’t have been possible on AMD is ludicrous and there is a pattern to your comments where you dismiss any other companies efforts or capabilities, but are quite happy to lay all the laurels on NVIDIA. The reality is that multiple companies and individuals got us to where we are, and multiple products could have done the same. That's not to take away from NVIDIA's success, it's well earned, but if you took them out of the equation, there's nothing that would have prevented the tech existing.
        hnlmorg 2y ago
        The crux of the issue is this: > It was just what happened to be available and most straightforward at the time AMD made better hardware for a while and people wanted OpenCL to succeed. The reason why nvidia became dominant was because their competitors simply weren’t good enough for general purpose parallel compute. Would AI still have happened without CUDA? Almost certainly. However nvidia still had a massive role in shaping what it looks like today.
        EVa5I7bHFq9mnYK 2y ago
        Many people here are heavily invested in NVDA stock, which certainly clouds the judgement.
        dagmx 2y ago
        That’s my take away too, even though I’m invested myself. The discussion around them has transcended into mythology. Imho part of it is because GPUs themselves are so foreign to many people as a development target and so they’re not really as aware of the landscape
    - deleted 2y ago
      [deleted]
- BobbyJo 2y ago
  > The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I graduated college in 2010 and I took a class taught in CUDA before graduating. CUDA was a primary driver of NN research at the time. Sure, other tools were available, but CUDA allowed people to build and distribute actually useful software which further encouraged the space. Could things have happened without it? Yeah, for sure, but it would have taken a good deal longer.

dagmx 2y ago

But that could apply to any of the GPU manufacturers. CUDA made for an easier ecosystem but if it didn’t exist it would have been any of the other APIs. The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I’m not trying to downplay NVIDIA but they specifically mentioned cause and effect, and then said it was because of NVIDIA.

KaoruAoiShiho 2y ago
First transformer != LLM. Imagine a world where you had to use AMD or CPUs, there would be no AlexNet, there would be no LLMs. Nvidia seeded universities with gifted hardware accelerators for a over a decade. Nvidia built the foundations for modern ML on which transformer lies, it's just one step on a long road to LLMs.
- dagmx 2y ago
  AMD had GPU compute APIs at the time as well. They also used to contribute GPUs (albeit in much smaller quantities). They just ended up killing them in favor of OpenCL which then withered on the vine. NVIDIA absolutely contributed to the foundation but they are not the foundation alone. Alexnet was great research but they could have done the same on other vendors at the time too. The hardware didn’t exist in a vacuum.
  - physicsguy 2y ago
    OpenCL itself is kinda fine, but the libraries never existed because every academic from postdoc up could get a free NVidia card. You literally filled a form out and they sent you it.
    - dagmx 2y ago
      I agree the library situation for OpenCL never stood up. Khronos tried at the start but then lost steam. It’s one of the projects I think the Khronos group mishandled the most unfortunately.
      - DeathArrow 2y ago
        Do they have a project that achieved success?
        echoangle 2y ago
        OpenGL, Vulkan, WebGL?
        dagmx 2y ago
        Here’s a list of projects https://www.khronos.org/ https://www.khronos.org/ If I were to categorize the successful ones: glTF, KTX, SPIR-V, OpenGL and its variants, WebGL People will say Vulkan but it has the same level of adoption as OpenCL, and has the same issue that it competes against vendor specific APIs (DX and Metal) that are just better to use. It’s still used though of course as a translation target but imho that doesn’t qualify it as a success. OpenCL was and is a failure of grand magnitude. As was colada.
  - KaoruAoiShiho 2y ago
    Alexnet could not have been done on AMD, nope.
    - dagmx 2y ago
      Please, this is just revisionism. There’s nothing inherent to AlexNet that relied on NVIDIA hardware or even software. It was just what happened to be available and most straightforward at the time. To say it wouldn’t have been possible on AMD is ludicrous and there is a pattern to your comments where you dismiss any other companies efforts or capabilities, but are quite happy to lay all the laurels on NVIDIA. The reality is that multiple companies and individuals got us to where we are, and multiple products could have done the same. That's not to take away from NVIDIA's success, it's well earned, but if you took them out of the equation, there's nothing that would have prevented the tech existing.
      - hnlmorg 2y ago
        The crux of the issue is this: > It was just what happened to be available and most straightforward at the time AMD made better hardware for a while and people wanted OpenCL to succeed. The reason why nvidia became dominant was because their competitors simply weren’t good enough for general purpose parallel compute. Would AI still have happened without CUDA? Almost certainly. However nvidia still had a massive role in shaping what it looks like today.
      - EVa5I7bHFq9mnYK 2y ago
        Many people here are heavily invested in NVDA stock, which certainly clouds the judgement.
        dagmx 2y ago
        That’s my take away too, even though I’m invested myself. The discussion around them has transcended into mythology. Imho part of it is because GPUs themselves are so foreign to many people as a development target and so they’re not really as aware of the landscape
  - deleted 2y ago
    [deleted]
BobbyJo 2y ago
> The first transformer models didn’t even use CUDA and CUDA didn’t have mass ecosystem inroads till years later. I graduated college in 2010 and I took a class taught in CUDA before graduating. CUDA was a primary driver of NN research at the time. Sure, other tools were available, but CUDA allowed people to build and distribute actually useful software which further encouraged the space. Could things have happened without it? Yeah, for sure, but it would have taken a good deal longer.