6 ms·

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familia

It may help your digging and search if you have in mind what those chips really try to do: Accelerate numerical linear algebra calculations. If you are familiar with linear algebra these specialized chips literally etch silicon so as to perform vector (and more general multi-array or tensor) computations faster than a general purpose CPU. They do that by loading and operating a whole set of numbers (a chunk of a vector or a matrix) simultaneously (whereas the CPU would operate mostly serially - one at a time). The advantage is (in a nutshell) that you can get a significant speedup. How much depends on the problem and how big a chunk you can process simultaneously but it can be a significant factor. There are disadvantages that people ignore in the current AI hype: * The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's * It is extremely specialized and fine-tuned software you need to develop and run and it only applies to the above linear algebra problems. * In the past such specialized numerical algebra hardware was the domain of HPC (high performance computing). Many a supercomputer vendor went bankrupt in the past because the cost versus market size was not there.

deleted 3y ago
[deleted]
benreesman 3y ago
While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades. I remember when NVIDIA didn’t have hardware for ReLU. The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”. This is an old story, things migrate in and out of the “CPU”, but the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future.
- microtonal 3y ago
  I remember when NVIDIA didn’t have hardware for ReLU. Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).
  - vgatherps 3y ago
    Not sure if or what the chance was, but fmax(x, 0) only requires checking the sign bit instead of doing a full floating point comparison (putting aside nan handling). A hypothetical relu instruction could probably get away with much less power and die soace?
  - benreesman 3y ago
    I could easily be wrong, but IIUC the software/hardware stack was, as of 2014 or so, fusing down to a specific set of circuits for broadcast activation functions under the broader banner of “Tensor Cores” (along with a bunch of devilish FMA hot-pathing under the sheets). This is a fairly random press piece off Google but there are a ton of them. Hard to tell whether it’s the hardware, the blob, or the process node unless you work there. https://developer.nvidia.com/blog/accelerating-relu-and-gelu-activation-functions-and-batched-sparse-gemm-in-cusparselt-v0-2-0/ https://developer.nvidia.com/blog/accelerating-relu-and-gelu...
- nologic01 3y ago
  > the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future. I think there are some important question marks still unresolved that bear on how things will play out. E.g. how the training versus inference balance will land in terms of usage and economics. Inference is inherently more "mass market". You need it locally without lags from moving data around. But inference is just numerical linear algebra. Ultimately augmenting the CPU to provide inference natively might be the optimal arrangement.
- JoeAltmaier 3y ago
  There have been some really strange instruction sets conceived. As a student at Stanford we had a time-share system that was home-grown (as I remember). It had opcodes to reverse bits in a bitstring! And odder things. Somebody needed that for some research project I guess. And then repurposed the damn thing for timeshare. It was pretty sad timeshare as I recall. The only machine(s?) available to a population of what? 20,000? And about 40 terminals total. You had to sign up for 15-minute measured timeslots. I came from a state school that had over 1000 terminals on a dozen machines, all unlimited time to students. It was a big shock to find the star of Silicon Valley had such crappy student services.
  - anthomtb 3y ago
    > It had opcodes to reverse bits in a bitstring Bit reversal is used in Fast Fourier Transforms. Its not entirely surprising to me that you'd have specialized hardware for that operation. Ref: https://en.wikipedia.org/wiki/Bit-reversal_permutation#Applications https://en.wikipedia.org/wiki/Bit-reversal_permutation#Appli...
    - JoeAltmaier 3y ago
      Also to count ones in a bitstring. And so on.
juujian 3y ago
Good explanation. That also gives you an idea why GPUs are decent at acceleration computations for neural networks (think cuda) -- they are already optimized for doing many small computations in parallel rather with slower processors and have a lot of dedicated ram (VRAM).
VeninVidiaVicii 3y ago
Is this why my M1 MacBook Air can run the same R code at least 10x faster than my giant Linux tower?
- microtonal 3y ago
  Apple Silicon Macs have special matrix multiplication units (AMX) that can do matrix multiplication fast and with low energy requirements [1]. These AMX units can often beat matrix multiplication on AMD/Intel CPUs (especially those without a very large number of cores). Since a lot of linear algebra code uses matrix multiplication and using the AMX units is only a matter of linking against Accelerate (for its BLAS interface), a lot of software that uses BLAS is faster o Apple Silicon Macs. That said, the GPUs in your M1 Mac are faster than the AMX units and any reasonably modern NVIDIA GPU will wipe the floor with the AMX units or Apple Silicon GPUs in raw compute. However, a lot of software does not use CUDA by default and for small problem sets AMX units or CPUs with just AVX can be faster because they don't incur the cost of data transfers from main memory to GPU memory and vice versa. [1] Benchmarks: https://github.com/danieldk/gemm-benchmark#example-results https://github.com/danieldk/gemm-benchmark#example-results https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders (scroll down a bit for AMX and MPS numbers)
  - stephencanon 3y ago
    > That said, the GPUs in your M1 Mac are faster than the AMX units Not for double, which is what R mostly uses IIRC.
    - microtonal 3y ago
      Ah, thanks for the correction! I never use R, so I assumed that it uses/supports single-precision floating point.
amelius 3y ago
> Accelerate numerical linear algebra calculations. Like compute eigenvalues/eigenvectors of large matrices, compute SVDs, solve large sparse systems of equations, etc?
- Q6T46nT668w6i3m 3y ago
  Nothing that fancy. Usually matrix-matrix and matrix-scalar multiplication.
  - nologic01 3y ago
    This is indeed the bread-and-butter, but there is use of all sorts of standard linear algebra algorithms. You can check various xla-related (accelerated linear algebra) folders in tensorflow or torch folders in pytorch to see the list of what is used [1],[2] [1] https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b383e3170439fe1c4ed354a163f/tensorflow/compiler/xla/client/lib https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b... [2] https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee890ff6c6e5dd54aca679/torch/csrc/api/include/torch/linalg.h https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee...
    - amelius 3y ago
      But is this used for AI or is this "what can we do with an AI chip since we happen to have one anyway?"
bhu1st 3y ago
> If you are familiar with linear algebra Could you please write what are some common day to day life applications of linear algebra in computing?
- wongarsu 3y ago
  The obvious fields are computer graphics (all kinds: 2d, 3d rasterized and 3d raytraced are all heavy on linear algebra, though 3d rasterized is the easiest to speed up with the lockstep SIMD architectures we call GPUs) and neural networks. Computer graphics mostly because you can view the real world as 3d space and the screen as 2d space, and linear algebra gives you all the tools to manipulate something in 3d space and project it into 2d space. Neural networks because you can treat them as matrix multiplications.
- juujian 3y ago
  I have really lost touch with common day to day life at this point, but Excel would be an excellent example. If you have a column with a million data points and do some basic calculation -- subtract or multiply another column -- whether or not the software you use can translate that into a vectorized operation under the hood using linear algebra can significantly speed up your operation.
- dreamcompiler 3y ago
  Nvidia exists because linear algebra is essential for 3D graphics. They got a second boost from crypto-currency, which isn't strictly about linear algebra but it can put those computation units to good use. Now Nvidia is riding high again because neural nets are all about linear algebra, and LLMs are big neural nets.
  - visarga 3y ago
    I would also add scientific simulations to the list of tasks that GPUs are used for. They parallelise well on many cores.
- JamesLeonis 3y ago
  Linear Algebra is like a lot of math in CS; you don't necessarily see it initially, but once you get some familiarity you start seeing it everywhere. Others have commented on computer graphics, but (as it turns out) the exact same algorithms apply for Collision Detection in 3D space. And since games already are manipulating graphics, you add on another set of Linear Algebra transforms that change the position/rotation/shear of those vertices. In a similar way, science (especially physics) use linear algebra to build simulations of all kinds of systems. One surprising use is in Advertising (and other user preference aggregators). Turns out a preference acts like a magnitude of a one-dimensional vector. String N preference vectors together and you get an N-Dimension Vector that you can perform Linear Algebra operations upon. One common application is the Dot Product, which is a fancy way of taking two N-Dimension vectors and measuring how close those vectors point in the same direction in a [1, -1] range. Yet another common place to find Linear Algebra is in computer science papers. Most of the time this is simply notation; a lot of common programming forms can be represented by MxN matrices. However some of those algorithms will use LA as a way of parsing and breaking down a problem. You will see this in compiler papers often, but its transferable to many other domains. As a final and personal observation, I found that Linear Algebra helped me grasp Functional Programming. In both cases I am applying a transform to some input, and often stringing together many such transforms. Also in both cases, the transformations are sensitive to their order, and a bad ordering produces nonsense just like garbage data.
sebzim4500 3y ago
>* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's This does not seem to be true in reality. The H100 is about 2.5x 'better' for AI than the A100 (obviously depends exactly what you are doing), and they released about 2 years apart. That is roughly in line with Moore's law.
- _nalply 3y ago
  The difference seems to be: more units for parallel calculation, but the speed of calculation in itself doesn't double anymore. In other words: Moore's law has stopped for raw speed and perhaps other areas, but is still alive in other areas. This has some weird consequences: Some models can't be processed in smaller chips (because swapping in parts is too slow to be useful), but after a threshold is crossed, suddenly the large models run efficiently. Probably we will see usage of large AI models in smaller devices the next years because there's another way to optimize: use more efficient representations of the model weights. I think about posits, a different floating point system where even 6 bits are perhaps usable. When models can switch to 6 bit posits from f16 (half floats), hardware can load more than three times larger models. We will see whether hardware for this will be mass-produced.
  - daveguy 3y ago
    Moore's law was never about raw speed. It is about transistors per unit area and it is still very much active. That transistor count is going into cache and core count, but it can just as easily go into specialized linear algebra units, which is what AI chips do.
  - raincole 3y ago
    Why from 16 to 6 is "more than three times larger"? Why not 16/6=2.667 times?
rerdavies 3y ago
> Accelerate numerical linear algebra calculations. Technically, for AI, you need to accelerate numerical non-linear algebra calculations, which take the general form of matrix multiplication, but interpose non-linear functions at key points in the calculation.
- ra1231963 3y ago
  the calculations within a neural network involve both linear and non-linear operations, but the fundamental mathematical framework underlying AI remains rooted in linear algebra

deleted 3y ago

[deleted]

benreesman 3y ago

While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades. I remember when NVIDIA didn’t have hardware for ReLU. The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”. This is an old story, things migrate in and out of the “CPU”, but the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future.

microtonal 3y ago
I remember when NVIDIA didn’t have hardware for ReLU. Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).
- vgatherps 3y ago
  Not sure if or what the chance was, but fmax(x, 0) only requires checking the sign bit instead of doing a full floating point comparison (putting aside nan handling). A hypothetical relu instruction could probably get away with much less power and die soace?
- benreesman 3y ago
  I could easily be wrong, but IIUC the software/hardware stack was, as of 2014 or so, fusing down to a specific set of circuits for broadcast activation functions under the broader banner of “Tensor Cores” (along with a bunch of devilish FMA hot-pathing under the sheets). This is a fairly random press piece off Google but there are a ton of them. Hard to tell whether it’s the hardware, the blob, or the process node unless you work there. https://developer.nvidia.com/blog/accelerating-relu-and-gelu-activation-functions-and-batched-sparse-gemm-in-cusparselt-v0-2-0/ https://developer.nvidia.com/blog/accelerating-relu-and-gelu...
nologic01 3y ago
> the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future. I think there are some important question marks still unresolved that bear on how things will play out. E.g. how the training versus inference balance will land in terms of usage and economics. Inference is inherently more "mass market". You need it locally without lags from moving data around. But inference is just numerical linear algebra. Ultimately augmenting the CPU to provide inference natively might be the optimal arrangement.
JoeAltmaier 3y ago
There have been some really strange instruction sets conceived. As a student at Stanford we had a time-share system that was home-grown (as I remember). It had opcodes to reverse bits in a bitstring! And odder things. Somebody needed that for some research project I guess. And then repurposed the damn thing for timeshare. It was pretty sad timeshare as I recall. The only machine(s?) available to a population of what? 20,000? And about 40 terminals total. You had to sign up for 15-minute measured timeslots. I came from a state school that had over 1000 terminals on a dozen machines, all unlimited time to students. It was a big shock to find the star of Silicon Valley had such crappy student services.
- anthomtb 3y ago
  > It had opcodes to reverse bits in a bitstring Bit reversal is used in Fast Fourier Transforms. Its not entirely surprising to me that you'd have specialized hardware for that operation. Ref: https://en.wikipedia.org/wiki/Bit-reversal_permutation#Applications https://en.wikipedia.org/wiki/Bit-reversal_permutation#Appli...
  - JoeAltmaier 3y ago
    Also to count ones in a bitstring. And so on.

juujian 3y ago

Good explanation. That also gives you an idea why GPUs are decent at acceleration computations for neural networks (think cuda) -- they are already optimized for doing many small computations in parallel rather with slower processors and have a lot of dedicated ram (VRAM).

VeninVidiaVicii 3y ago

Is this why my M1 MacBook Air can run the same R code at least 10x faster than my giant Linux tower?

microtonal 3y ago
Apple Silicon Macs have special matrix multiplication units (AMX) that can do matrix multiplication fast and with low energy requirements [1]. These AMX units can often beat matrix multiplication on AMD/Intel CPUs (especially those without a very large number of cores). Since a lot of linear algebra code uses matrix multiplication and using the AMX units is only a matter of linking against Accelerate (for its BLAS interface), a lot of software that uses BLAS is faster o Apple Silicon Macs. That said, the GPUs in your M1 Mac are faster than the AMX units and any reasonably modern NVIDIA GPU will wipe the floor with the AMX units or Apple Silicon GPUs in raw compute. However, a lot of software does not use CUDA by default and for small problem sets AMX units or CPUs with just AVX can be faster because they don't incur the cost of data transfers from main memory to GPU memory and vice versa. [1] Benchmarks: https://github.com/danieldk/gemm-benchmark#example-results https://github.com/danieldk/gemm-benchmark#example-results https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders (scroll down a bit for AMX and MPS numbers)
- stephencanon 3y ago
  > That said, the GPUs in your M1 Mac are faster than the AMX units Not for double, which is what R mostly uses IIRC.
  - microtonal 3y ago
    Ah, thanks for the correction! I never use R, so I assumed that it uses/supports single-precision floating point.

amelius 3y ago

> Accelerate numerical linear algebra calculations. Like compute eigenvalues/eigenvectors of large matrices, compute SVDs, solve large sparse systems of equations, etc?

Q6T46nT668w6i3m 3y ago
Nothing that fancy. Usually matrix-matrix and matrix-scalar multiplication.
- nologic01 3y ago
  This is indeed the bread-and-butter, but there is use of all sorts of standard linear algebra algorithms. You can check various xla-related (accelerated linear algebra) folders in tensorflow or torch folders in pytorch to see the list of what is used [1],[2] [1] https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b383e3170439fe1c4ed354a163f/tensorflow/compiler/xla/client/lib https://github.com/tensorflow/tensorflow/tree/8d9b35f442045b... [2] https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee890ff6c6e5dd54aca679/torch/csrc/api/include/torch/linalg.h https://github.com/pytorch/pytorch/blob/6e3e3dd477e0fb9768ee...
  - amelius 3y ago
    But is this used for AI or is this "what can we do with an AI chip since we happen to have one anyway?"

bhu1st 3y ago

> If you are familiar with linear algebra Could you please write what are some common day to day life applications of linear algebra in computing?

wongarsu 3y ago
The obvious fields are computer graphics (all kinds: 2d, 3d rasterized and 3d raytraced are all heavy on linear algebra, though 3d rasterized is the easiest to speed up with the lockstep SIMD architectures we call GPUs) and neural networks. Computer graphics mostly because you can view the real world as 3d space and the screen as 2d space, and linear algebra gives you all the tools to manipulate something in 3d space and project it into 2d space. Neural networks because you can treat them as matrix multiplications.
juujian 3y ago
I have really lost touch with common day to day life at this point, but Excel would be an excellent example. If you have a column with a million data points and do some basic calculation -- subtract or multiply another column -- whether or not the software you use can translate that into a vectorized operation under the hood using linear algebra can significantly speed up your operation.
dreamcompiler 3y ago
Nvidia exists because linear algebra is essential for 3D graphics. They got a second boost from crypto-currency, which isn't strictly about linear algebra but it can put those computation units to good use. Now Nvidia is riding high again because neural nets are all about linear algebra, and LLMs are big neural nets.
- visarga 3y ago
  I would also add scientific simulations to the list of tasks that GPUs are used for. They parallelise well on many cores.
JamesLeonis 3y ago
Linear Algebra is like a lot of math in CS; you don't necessarily see it initially, but once you get some familiarity you start seeing it everywhere. Others have commented on computer graphics, but (as it turns out) the exact same algorithms apply for Collision Detection in 3D space. And since games already are manipulating graphics, you add on another set of Linear Algebra transforms that change the position/rotation/shear of those vertices. In a similar way, science (especially physics) use linear algebra to build simulations of all kinds of systems. One surprising use is in Advertising (and other user preference aggregators). Turns out a preference acts like a magnitude of a one-dimensional vector. String N preference vectors together and you get an N-Dimension Vector that you can perform Linear Algebra operations upon. One common application is the Dot Product, which is a fancy way of taking two N-Dimension vectors and measuring how close those vectors point in the same direction in a [1, -1] range. Yet another common place to find Linear Algebra is in computer science papers. Most of the time this is simply notation; a lot of common programming forms can be represented by MxN matrices. However some of those algorithms will use LA as a way of parsing and breaking down a problem. You will see this in compiler papers often, but its transferable to many other domains. As a final and personal observation, I found that Linear Algebra helped me grasp Functional Programming. In both cases I am applying a transform to some input, and often stringing together many such transforms. Also in both cases, the transformations are sensitive to their order, and a bad ordering produces nonsense just like garbage data.

sebzim4500 3y ago

>* The speedup in a one-off gain, the death of Moore's law is equally dead for "AI chips" and CPU's This does not seem to be true in reality. The H100 is about 2.5x 'better' for AI than the A100 (obviously depends exactly what you are doing), and they released about 2 years apart. That is roughly in line with Moore's law.

_nalply 3y ago
The difference seems to be: more units for parallel calculation, but the speed of calculation in itself doesn't double anymore. In other words: Moore's law has stopped for raw speed and perhaps other areas, but is still alive in other areas. This has some weird consequences: Some models can't be processed in smaller chips (because swapping in parts is too slow to be useful), but after a threshold is crossed, suddenly the large models run efficiently. Probably we will see usage of large AI models in smaller devices the next years because there's another way to optimize: use more efficient representations of the model weights. I think about posits, a different floating point system where even 6 bits are perhaps usable. When models can switch to 6 bit posits from f16 (half floats), hardware can load more than three times larger models. We will see whether hardware for this will be mass-produced.
- daveguy 3y ago
  Moore's law was never about raw speed. It is about transistors per unit area and it is still very much active. That transistor count is going into cache and core count, but it can just as easily go into specialized linear algebra units, which is what AI chips do.
- raincole 3y ago
  Why from 16 to 6 is "more than three times larger"? Why not 16/6=2.667 times?

rerdavies 3y ago

> Accelerate numerical linear algebra calculations. Technically, for AI, you need to accelerate numerical non-linear algebra calculations, which take the general form of matrix multiplication, but interpose non-linear functions at key points in the calculation.

ra1231963 3y ago
the calculations within a neural network involve both linear and non-linear operations, but the fundamental mathematical framework underlying AI remains rooted in linear algebra