3 ms·

Given how much the "brains" of AI is built upon fairly old (and very old) mathematics, I find this sentiment to be somewhat myopic. Yes, deep learning has its r

Given how much the "brains" of AI is built upon fairly old (and very old) mathematics, I find this sentiment to be somewhat myopic. Yes, deep learning has its roots in something 50+ years old. Yes, we still use Bayesian concepts unearthed a century or more ago. Probability theory took foot 80 years ago. So what? Do we throw out the old models and approaches simply because they're old? But to say that AI "hasn't really come very far" because its technological achievements outweigh its computational side doesn't really make much sense to me. Progress is slow. Seeing a story like this and saying "wow, and we're only now seeing this in use in practice today" makes an implicit assumption that once something is proved it should become widespread. The original self-driving cars were not practical; it took advancements in computational power and speed to make it so. That's "coming far." We don't need some novel ML approach to advance AI.

carlmr 8y ago
Another nice example are Fourier transforms. It took quite a while from 1822 until we had great applications for it in microcontrollers and DSPs for sound processing. However I can also kind of see the argument here that the deep learning ideas don't seem to have such a fundamental idea behind them. We modeled how biology works at a very simple and abstract level. Now we have more computing power (and especially more data) so we could find a few applications. But to me at least it doesn't look like a solid theory to build upon. It doesn't deliver insights into how things work like Fourier transforms did with signals and Bayes did with probabilities.
- bunderbunder 8y ago
  > We modeled how biology works That's the popular story behind neural networks. Like any good popular explanation of a STEM concept, it's as useful as it is inaccurate.
  - carlmr 8y ago
    >at a simple and abstract level It would be nice if you quoted honestly.
    - bunderbunder 8y ago
      Sorry, I wasn't meaning to quote mine. I honestly didn't think that part was relevant to the point I was making. Neural networks have taken some inspiration from biology, but that doesn't account for the vast bulk of the work. Biology analogies play a much bigger part in popular explanations than they did in the development of the technology. (edit) Here, for example, take a look at the original paper on perceptrons. Whole lotta math. This having been released during one of the AI flaps, there was also a fair bit of psychology talk. But not much indication that he was just trying to copy a brain and otherwise didn't understand how it works. https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-... Similar happens with deep learning. They are grounded in theory, most notably a mathematical proof that deep MLPs are, in principle, capable of learning any mathematical function. You just won't see much of that stuff if you aren't actually reading the papers, because most people aren't that interested in vector calculus. Much easier to explain things by analogy.
      - deleted 8y ago
        [deleted]
  - nkozyra 8y ago
    > That's the popular story behind neural networks. Like any good popular explanation of a STEM concept, it's as useful as it is inaccurate. To be fair, our own understanding of "how it works" in practice is not necessarily set in stone. Approximating our belief in how it works is the best case scenario, and expecting it to perfectly mirror human reasoning is unrealistic and frankly not all that appealing from a problem-solving stance. Human reasoning has a lot of built-in buffers and fault tolerance, and it is, itself, a fairly narrow AI.
- twtw 8y ago
  Fun fact: the FFT (the algorithm that is actually used in applications today) was originally developed around 1805 by Gauss, but he didn't publish the work. Sorta crazy since it precedes Fourier's work.
  - dekhn 8y ago
    This is a bit of a stretch. The algorithm used today is due to Tukey. Gauss's work was great, but it's not the algorithm used today. Also, another interesting fact about fourier transforms- in the old days, computing FTs was too expensive, so physicists used lenses- which perform approximate FTs- to decompose signals into spectra.
    - twtw 8y ago
      From https://www.cis.rit.edu/class/simg716/Gauss_History_FFT.pdf https://www.cis.rit.edu/class/simg716/Gauss_History_FFT.pdf > Thus, Gauss' algorithm is as general and powerful as the Cooley-Tukey common factor algorithm and is, in fact, equivalent to a decimation in frequency algorithm adapted to a real data sequence.
bunderbunder 8y ago
> Progress is slow. And incremental. Sometimes you'll hear people say that deep learning is nothing new, because the first deep neural networks were developed many decades ago, or because CNNs and LSTMs were developed in the late 90s, etc. That mindset glosses over a lot of the incremental developments (e.g., getting a decent handle on transfer learning) that were necessary to get from the initial academic demonstration of the idea to its successful commercial application.
squirrelicus 8y ago
I don't know what your perception of my original sentiment is, so let me elaborate. People generally assume, likely because of the astronomical rate of quality of life expansion due to technology and engineering in the 1900s, that things keep "going forward" at some rate. Often they think this rate is exponential. I see people citing AI implementation engineering as their primary examples of this. It's frustrating because it's not true. AI is old. That's fine and good and wholesome. But there is no real progress in AI theory, it's all practical engineering. Unfortunately, theory and engineering are often conflated with "technological progress" without realizing the former limits the latter with a very hard ceiling.

carlmr 8y ago

Another nice example are Fourier transforms. It took quite a while from 1822 until we had great applications for it in microcontrollers and DSPs for sound processing. However I can also kind of see the argument here that the deep learning ideas don't seem to have such a fundamental idea behind them. We modeled how biology works at a very simple and abstract level. Now we have more computing power (and especially more data) so we could find a few applications. But to me at least it doesn't look like a solid theory to build upon. It doesn't deliver insights into how things work like Fourier transforms did with signals and Bayes did with probabilities.

bunderbunder 8y ago
> We modeled how biology works That's the popular story behind neural networks. Like any good popular explanation of a STEM concept, it's as useful as it is inaccurate.
- carlmr 8y ago
  >at a simple and abstract level It would be nice if you quoted honestly.
  - bunderbunder 8y ago
    Sorry, I wasn't meaning to quote mine. I honestly didn't think that part was relevant to the point I was making. Neural networks have taken some inspiration from biology, but that doesn't account for the vast bulk of the work. Biology analogies play a much bigger part in popular explanations than they did in the development of the technology. (edit) Here, for example, take a look at the original paper on perceptrons. Whole lotta math. This having been released during one of the AI flaps, there was also a fair bit of psychology talk. But not much indication that he was just trying to copy a brain and otherwise didn't understand how it works. https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-1957.pdf https://blogs.umass.edu/brain-wars/files/2016/03/rosenblatt-... Similar happens with deep learning. They are grounded in theory, most notably a mathematical proof that deep MLPs are, in principle, capable of learning any mathematical function. You just won't see much of that stuff if you aren't actually reading the papers, because most people aren't that interested in vector calculus. Much easier to explain things by analogy.
    - deleted 8y ago
      [deleted]
- nkozyra 8y ago
  > That's the popular story behind neural networks. Like any good popular explanation of a STEM concept, it's as useful as it is inaccurate. To be fair, our own understanding of "how it works" in practice is not necessarily set in stone. Approximating our belief in how it works is the best case scenario, and expecting it to perfectly mirror human reasoning is unrealistic and frankly not all that appealing from a problem-solving stance. Human reasoning has a lot of built-in buffers and fault tolerance, and it is, itself, a fairly narrow AI.
twtw 8y ago
Fun fact: the FFT (the algorithm that is actually used in applications today) was originally developed around 1805 by Gauss, but he didn't publish the work. Sorta crazy since it precedes Fourier's work.
- dekhn 8y ago
  This is a bit of a stretch. The algorithm used today is due to Tukey. Gauss's work was great, but it's not the algorithm used today. Also, another interesting fact about fourier transforms- in the old days, computing FTs was too expensive, so physicists used lenses- which perform approximate FTs- to decompose signals into spectra.
  - twtw 8y ago
    From https://www.cis.rit.edu/class/simg716/Gauss_History_FFT.pdf https://www.cis.rit.edu/class/simg716/Gauss_History_FFT.pdf > Thus, Gauss' algorithm is as general and powerful as the Cooley-Tukey common factor algorithm and is, in fact, equivalent to a decimation in frequency algorithm adapted to a real data sequence.

bunderbunder 8y ago

> Progress is slow. And incremental. Sometimes you'll hear people say that deep learning is nothing new, because the first deep neural networks were developed many decades ago, or because CNNs and LSTMs were developed in the late 90s, etc. That mindset glosses over a lot of the incremental developments (e.g., getting a decent handle on transfer learning) that were necessary to get from the initial academic demonstration of the idea to its successful commercial application.

squirrelicus 8y ago

I don't know what your perception of my original sentiment is, so let me elaborate. People generally assume, likely because of the astronomical rate of quality of life expansion due to technology and engineering in the 1900s, that things keep "going forward" at some rate. Often they think this rate is exponential. I see people citing AI implementation engineering as their primary examples of this. It's frustrating because it's not true. AI is old. That's fine and good and wholesome. But there is no real progress in AI theory, it's all practical engineering. Unfortunately, theory and engineering are often conflated with "technological progress" without realizing the former limits the latter with a very hard ceiling.