3 ms·

How future proof is all this stuff against changes in AI? What if someone discovers a viable alternative to ANN's or a way to implement them better that doesn't

How future proof is all this stuff against changes in AI? What if someone discovers a viable alternative to ANN's or a way to implement them better that doesn't benefit from GPU's?

kcb 3y ago
That's one of the benefits of GPGPUs vs custom training silicon. As long as the solution requires highly parallelizable calculations a GPU will be able to accelerate it.
danielmarkbruce 3y ago
Chances that some algorithm won't involved a truck load of matmuls are close to zero.
- 29athrowaway 3y ago
  We at least know the mammalian brain doesn't do it. Quick maths.
  - danielmarkbruce 3y ago
    Maybe. It might just be matmuls all the way down.
  - jjoonathan 3y ago
    On the contrary, dendrites/synapses/axons implement something that looks an awful lot like matrix multiplication followed by an activation function. The biggest differences between bio and artificial neural networks -- spiking, backpropagation, etc -- seem to be in the non-matmul parts of the story.
    - 29athrowaway 3y ago
      But it is not matrix multiplication. We want it to be matrix multiplication, we want to use a minimalistic, elegant, reductionist approach, we want to claim we understand it, but in practice it's none of that otherwise we would have a working artificial brain. As we discover things we find those things at work in the brain, but the brain is too advanced for us to make proper sense of it.
      - jjoonathan 3y ago
        We only understand a small part of a big story, but we do understand that small part and it does look like matrix multiplication. Most of what goes on in the brain is up for debate. But not that part. Incidentally, this is the whole reason why reductionist approaches are so important: without reductionism, you can never get a foothold. You are adrift in a sea of woo and can never make any progress towards understanding because one nonfalsifiable holistic theory is as good as the next. However, if you break the system down into pieces until you do understand a piece, you can start to put the puzzle together. Dendrites, synapses, action potentials, thresholds, potentiation, plasticity -- these things are real, we do understand them under an ever growing set of circumstances, and while the picture on the puzzle is still anyone's guess the corner that looks like matrix multiplication is fully assembled and not going anywhere.
        29athrowaway 3y ago
        It is a straw man.
dkislyuk 3y ago
In the current world, deep learning with homogeneous computation graphs, tuned with backprop, has won the Hardware Lottery [1]. This is unfortunate for research outside of that area, but just looking at the momentum of development it seems like a sure bet to keep investing in GPU-based training and inference for the next decade. There's just too much lock-in already to this paradigm. If a new algorithm appears from with a novel approach (analog compute, heterogeneous computation graphs from genetic algorithms, quantum, much more...), there will be a whole generation of R&D + tool + framework building, which gives the major players enough time to adapt. [1] https://arxiv.org/abs/2009.06489 https://arxiv.org/abs/2009.06489

How future proof is all this stuff against changes in AI? What if someone discovers a viable alternative to ANN's or a way to implement them better that doesn't benefit from GPU's?

kcb 3y ago

That's one of the benefits of GPGPUs vs custom training silicon. As long as the solution requires highly parallelizable calculations a GPU will be able to accelerate it.

danielmarkbruce 3y ago

Chances that some algorithm won't involved a truck load of matmuls are close to zero.

29athrowaway 3y ago
We at least know the mammalian brain doesn't do it. Quick maths.
- danielmarkbruce 3y ago
  Maybe. It might just be matmuls all the way down.
- jjoonathan 3y ago
  On the contrary, dendrites/synapses/axons implement something that looks an awful lot like matrix multiplication followed by an activation function. The biggest differences between bio and artificial neural networks -- spiking, backpropagation, etc -- seem to be in the non-matmul parts of the story.
  - 29athrowaway 3y ago
    But it is not matrix multiplication. We want it to be matrix multiplication, we want to use a minimalistic, elegant, reductionist approach, we want to claim we understand it, but in practice it's none of that otherwise we would have a working artificial brain. As we discover things we find those things at work in the brain, but the brain is too advanced for us to make proper sense of it.
    - jjoonathan 3y ago
      We only understand a small part of a big story, but we do understand that small part and it does look like matrix multiplication. Most of what goes on in the brain is up for debate. But not that part. Incidentally, this is the whole reason why reductionist approaches are so important: without reductionism, you can never get a foothold. You are adrift in a sea of woo and can never make any progress towards understanding because one nonfalsifiable holistic theory is as good as the next. However, if you break the system down into pieces until you do understand a piece, you can start to put the puzzle together. Dendrites, synapses, action potentials, thresholds, potentiation, plasticity -- these things are real, we do understand them under an ever growing set of circumstances, and while the picture on the puzzle is still anyone's guess the corner that looks like matrix multiplication is fully assembled and not going anywhere.
      - 29athrowaway 3y ago
        It is a straw man.

dkislyuk 3y ago

In the current world, deep learning with homogeneous computation graphs, tuned with backprop, has won the Hardware Lottery [1]. This is unfortunate for research outside of that area, but just looking at the momentum of development it seems like a sure bet to keep investing in GPU-based training and inference for the next decade. There's just too much lock-in already to this paradigm. If a new algorithm appears from with a novel approach (analog compute, heterogeneous computation graphs from genetic algorithms, quantum, much more...), there will be a whole generation of R&D + tool + framework building, which gives the major players enough time to adapt. [1] https://arxiv.org/abs/2009.06489 https://arxiv.org/abs/2009.06489