5 ms·

Exactly! Plus, even before CUDA existed, people in numerical computing had already noticed that GPUs were ideal hardware for certain classes of problems and we

by ms013 9y ago

Exactly! Plus, even before CUDA existed, people in numerical computing had already noticed that GPUs were ideal hardware for certain classes of problems and were doing awkward things like embedding numerical calculations as shaders and other tricks to use the graphics APIs as a way to encode computations. I recall colleagues who worked on those pre-CUDA systems meeting some resistance from NVIDIA in terms of opening up the low-level APIs that would make the hardware more directly accessible, versus the encodings that they needed to go through to use graphics primitives for the work. NVIDIA got lucky by having the right kind of hardware, and created CUDA not due to some insight into the AI world, but based on the needs of scientific computing. Just look at their early papers on CUDA (E.g., Luebke's 2008 paper "CUDA: SCALABLE PARALLEL PROGRAMMING FOR HIGH-PERFORMANCE SCIENTIFIC COMPUTING") - AI was nowhere to be seen. They were focused on traditional high performance computing problems.

dragandj 9y ago
HPC techniques are exactly those that ML and deep learning use in implementation. I don't see how DL is much different from this point of view. It relies on fast linear algebra. HPC relies on fast linear algebra. Nvidia perfected the processor that does this job orders of magnitude faster and provided convenient libraries and development environment to make it accessible.
- ms013 9y ago
  Nobody is arguing that not to be true. The issue is arguing that NVIDIA foresaw their systems being relevant to AI/DL/ML: they simply were lucky. They opened their architecture to HPC people, concerned largely with fluid dynamics, finite element methods, nbody methods, and so on. To claim that they had any insight into the emergence of ML as a core application of their silicon is simply a laughable revision of history: as we were talking about in an ancestor of this thread.
  - dragandj 9y ago
    Well, I argue that they did foresaw its business viability. They invested heavily in DL before any one of their competitors. Of course there were researchers everywhere working on that, but there are researchers working on everything. What is the most difficult part is recognizing that it can be taken to mainstream and betting huge resources on that. Of course no one argues that Nvidia CEO is the first man in the world that saw GPU/AI applicability. But of those in positions of power in the tech business community...
    - Eridrus 9y ago
      But they didn't really bet anything huge on it. In 2014 they had a few researchers work on cuDNN; a few years after deep learning had already taken off in academia. The first chip they made which did anything specific for DL was the P100 last year; but no-one is actually using that chip, everyone is using high end gaming chips. They really haven't made much forward looking investment in this area, they just got lucky since they had built CUDA beforehand for other reasons, and that was what people standardized on. AMD's chips would work just fine for DL if everything wasn't already written in CUDA, needing them to invest in a cross-platform CUDA alternative in the form of ROCm/HIP.
      - dragandj 9y ago
        I've been following AMD's efforts, and I think what you say obscures the practical points. In fact, I prefer OpenCL to CUDA. However, that it is not only a matter of "a few people"'s effort is that AMD had a few people on BLAS libraries since forever, and their stuff is almost unusable for DL, let alone a match for CUDA. Add to that that Nvidia provided cuDNN that everyone uses. That it is not a matter of just a few people can be seen from the fact that AMD still does not provide an alternative, even a toy one, that works. Everythinig in AMD/OpenCL world relies on a few 3-rd party open-source efforts. Some of those work OK, some are cool, some are great, some are garbage, but there is no ecosystem anywhere near Nvidia's. They have HIP (and they have OpenCL) but these are only general purpose compilers. They have nothing when it comes to the libraries.
        9y ago
- JustFinishedBSG 9y ago
  When Nvidia was targeting HPC the push was for better DP power in GPUs. Now that Deep Learning is big it's a push toward Half-Precision actually. Most of the "incredible power" of a Tesla V100 is 100% unusable by a Physicist for example Also I love your MCMC sampler
  - dragandj 9y ago
    Thanks :)

by ms013 9y ago

dragandj 9y ago

HPC techniques are exactly those that ML and deep learning use in implementation. I don't see how DL is much different from this point of view. It relies on fast linear algebra. HPC relies on fast linear algebra. Nvidia perfected the processor that does this job orders of magnitude faster and provided convenient libraries and development environment to make it accessible.

ms013 9y ago
Nobody is arguing that not to be true. The issue is arguing that NVIDIA foresaw their systems being relevant to AI/DL/ML: they simply were lucky. They opened their architecture to HPC people, concerned largely with fluid dynamics, finite element methods, nbody methods, and so on. To claim that they had any insight into the emergence of ML as a core application of their silicon is simply a laughable revision of history: as we were talking about in an ancestor of this thread.
- dragandj 9y ago
  Well, I argue that they did foresaw its business viability. They invested heavily in DL before any one of their competitors. Of course there were researchers everywhere working on that, but there are researchers working on everything. What is the most difficult part is recognizing that it can be taken to mainstream and betting huge resources on that. Of course no one argues that Nvidia CEO is the first man in the world that saw GPU/AI applicability. But of those in positions of power in the tech business community...
  - Eridrus 9y ago
    But they didn't really bet anything huge on it. In 2014 they had a few researchers work on cuDNN; a few years after deep learning had already taken off in academia. The first chip they made which did anything specific for DL was the P100 last year; but no-one is actually using that chip, everyone is using high end gaming chips. They really haven't made much forward looking investment in this area, they just got lucky since they had built CUDA beforehand for other reasons, and that was what people standardized on. AMD's chips would work just fine for DL if everything wasn't already written in CUDA, needing them to invest in a cross-platform CUDA alternative in the form of ROCm/HIP.
    - dragandj 9y ago
      I've been following AMD's efforts, and I think what you say obscures the practical points. In fact, I prefer OpenCL to CUDA. However, that it is not only a matter of "a few people"'s effort is that AMD had a few people on BLAS libraries since forever, and their stuff is almost unusable for DL, let alone a match for CUDA. Add to that that Nvidia provided cuDNN that everyone uses. That it is not a matter of just a few people can be seen from the fact that AMD still does not provide an alternative, even a toy one, that works. Everythinig in AMD/OpenCL world relies on a few 3-rd party open-source efforts. Some of those work OK, some are cool, some are great, some are garbage, but there is no ecosystem anywhere near Nvidia's. They have HIP (and they have OpenCL) but these are only general purpose compilers. They have nothing when it comes to the libraries.
      - 9y ago
JustFinishedBSG 9y ago
When Nvidia was targeting HPC the push was for better DP power in GPUs. Now that Deep Learning is big it's a push toward Half-Precision actually. Most of the "incredible power" of a Tesla V100 is 100% unusable by a Physicist for example Also I love your MCMC sampler
- dragandj 9y ago
  Thanks :)