6 ms·

I believe it was Russell and Norvig who said about current AI techniques that they are like a man trying to reach the moon by climbing a tree: “One can report s

by maest 6y ago

I believe it was Russell and Norvig who said about current AI techniques that they are like a man trying to reach the moon by climbing a tree: “One can report steady progress, all the way to the top of the tree.”

bagels 6y ago
And then we eventually build the space elevator.
sn41 6y ago
Jitendra Malik, Computer Vision fame, calls it "the fallacy of the successful first step". Look ma, I can leap! It's only a matter of time till we get to the moon.
- beambot 6y ago
  To be fair: if you measure the cosmological time from the first human step to man walking on the moon, it was barely the blink of an eye...
  - coldtea 6y ago
    Sure. But why should we compare it to the cosmological time, and not to e.g. civilization time? Or homo sapiens time? In which case it took 100s to 1000s of lifetimes between the step and the moon...
earthboundkid 6y ago
The other criticism is that the sheer amount of data in use shows we’re doing something wrong. A child can internalize the grammar of a language with only a few years of exposure. Modern AI corpuses consists of terabytes of data. Why are the results so lacking despite using vastly more input?
- mpoteat 6y ago
  Why would you assume that years of exposure isn't terabytes of data? At least we can train language models in hours or days rather than years. If you view it that way we've made quite a bit of progress!
  - dlkf 6y ago
    Your first point is spot-on. Neural nets aren't unrealistic in virtue of having too much data, the problem is they have too little. Your second point doesn't make sense though. Language models don't understand language in the way that humans do. They just maximize the probability of the next word given some corpus. This is completely different than what humans do.
    - andrepd 6y ago
      GPT-3 certainly was fed far more data than any human needs to start talking.
      - dlkf 6y ago
        So your claim is that if a human child was given a subset of the text GPT-3 was fed — with no audio, video, or reinforcement feedback from its environment — the child could learn English? The important point to observe here is that what a kid does is fundamentally different than what GPT-3 does: - GPT-3 learns the word that minimizes perplexity given some context - A kid learns the word that helps them accomplish some task in some environment. In the learning process, children get feedback from their environment - including the responses of other agents in the environment (eg Mom, Dad, friends, etc). This is going to be far more memory-intensive than some text files. Also important to observe is that the child's experience can't be reduced to the audio, video and haptic streams they get, because the audio and video stream depend on their own actions. So you need the conditional statements that were embedded in the environment the kid was learning in. All of which is to say this is a little heavier than ascii. Edit: From your other comments, it seems we agree that bigger neural networks with more training data are not going to yield some quantum leap where GPT-3 starts talking. I'm not at all suggesting that throwing more data at the existing architectures is a fruitful direction for understanding cognition. (In general I think NLP is utterly pointless as regards understanding cognition, and if that's your goal you should focus on vision and reinforcement learning.) But my point is that we can't expect that a smarter architecture on a smaller corpus will get us anywhere. If we ever develop machines that are intelligent in some robust sense of the word, their "training data" will most likely be a physical environment with other agents in it.
nbardy 6y ago
If you think that still holds you have seen where the branches on this tree are starting to lead. Feels like we found a magic bean
lacker 6y ago
I believe that quote actually predates the modern era of AI! Here is an article from 2013 referring to it, as being in the then-current version of their textbook: https://www.theatlantic.com/magazine/archive/2013/11/the-man-who-would-teach-machines-to-think/309529/ https://www.theatlantic.com/magazine/archive/2013/11/the-man... Look at the other claims in this article from 2013 for comparison: Consider that computers today still have trouble recognizing a handwritten A.... In Hofstadter’s mind, there is nothing to be surprised about. To know what all A’s have in common would be, he argued in a 1982 essay, to “understand the fluid nature of mental categories.” And that, he says, is the core of human intelligence. I would say the modern era of AI was kicked off with AlexNet in 2012 and hit its stride a few years later. So, I believe this quote and this article are really referring to pre-GPU AI techniques. Basically, the predictions from this era just underestimated the value of scaling the data. Modern AI has several impressive achievements. It can certainly recognize an A. Image recognition and voice recognition are now critical parts of real products that help people in their everyday lives. At the same time, it's true that current AI techniques might not get us all the way to AGI. We'll just have to wait and see. But I think it's important to recognize that we have had real progress in the modern AI era that has seriously outperformed the pessimistic expectations from ~2010.
- StrictDabbler 6y ago
  Modern AI cannot recognize the kind of A that Hofstadter wrote about in that essay. Hofstadter identifies novelty typefaces where A has none of the characteristics that you might associate with the letter. Often A is not a triangle, sure, but sometimes it has no upper bar, no left stroke, no right stroke. Sometimes the bars are defined by a transition from negative to positive space. Sometimes A is made of 20 strokes, or a pattern of dots, or an open eye where the half-circular upper lid defines the triangle and the pupil is the bar. A neural network trained on conventional As would not conclude that any of these are an A. These shapes wouldn't necessarily read as A in isolation. Sometimes they only read correctly when you're looking at text using a whole typeface defined on the same principles. Modern AI still cannot recognize that kind of A even though a human easily can. There are ways to make an A that require a human to understand an idea of "letterness" that's different than anything they've seen before. I'd recommend reading the original essay. The illustrations make this clear in a way my description can't.

by maest 6y ago

bagels 6y ago

And then we eventually build the space elevator.

sn41 6y ago

Jitendra Malik, Computer Vision fame, calls it "the fallacy of the successful first step". Look ma, I can leap! It's only a matter of time till we get to the moon.

beambot 6y ago
To be fair: if you measure the cosmological time from the first human step to man walking on the moon, it was barely the blink of an eye...
- coldtea 6y ago
  Sure. But why should we compare it to the cosmological time, and not to e.g. civilization time? Or homo sapiens time? In which case it took 100s to 1000s of lifetimes between the step and the moon...

earthboundkid 6y ago

The other criticism is that the sheer amount of data in use shows we’re doing something wrong. A child can internalize the grammar of a language with only a few years of exposure. Modern AI corpuses consists of terabytes of data. Why are the results so lacking despite using vastly more input?

mpoteat 6y ago
Why would you assume that years of exposure isn't terabytes of data? At least we can train language models in hours or days rather than years. If you view it that way we've made quite a bit of progress!
- dlkf 6y ago
  Your first point is spot-on. Neural nets aren't unrealistic in virtue of having too much data, the problem is they have too little. Your second point doesn't make sense though. Language models don't understand language in the way that humans do. They just maximize the probability of the next word given some corpus. This is completely different than what humans do.
  - andrepd 6y ago
    GPT-3 certainly was fed far more data than any human needs to start talking.
    - dlkf 6y ago
      So your claim is that if a human child was given a subset of the text GPT-3 was fed — with no audio, video, or reinforcement feedback from its environment — the child could learn English? The important point to observe here is that what a kid does is fundamentally different than what GPT-3 does: - GPT-3 learns the word that minimizes perplexity given some context - A kid learns the word that helps them accomplish some task in some environment. In the learning process, children get feedback from their environment - including the responses of other agents in the environment (eg Mom, Dad, friends, etc). This is going to be far more memory-intensive than some text files. Also important to observe is that the child's experience can't be reduced to the audio, video and haptic streams they get, because the audio and video stream depend on their own actions. So you need the conditional statements that were embedded in the environment the kid was learning in. All of which is to say this is a little heavier than ascii. Edit: From your other comments, it seems we agree that bigger neural networks with more training data are not going to yield some quantum leap where GPT-3 starts talking. I'm not at all suggesting that throwing more data at the existing architectures is a fruitful direction for understanding cognition. (In general I think NLP is utterly pointless as regards understanding cognition, and if that's your goal you should focus on vision and reinforcement learning.) But my point is that we can't expect that a smarter architecture on a smaller corpus will get us anywhere. If we ever develop machines that are intelligent in some robust sense of the word, their "training data" will most likely be a physical environment with other agents in it.

nbardy 6y ago

If you think that still holds you have seen where the branches on this tree are starting to lead. Feels like we found a magic bean

lacker 6y ago

I believe that quote actually predates the modern era of AI! Here is an article from 2013 referring to it, as being in the then-current version of their textbook: https://www.theatlantic.com/magazine/archive/2013/11/the-man-who-would-teach-machines-to-think/309529/ https://www.theatlantic.com/magazine/archive/2013/11/the-man... Look at the other claims in this article from 2013 for comparison: Consider that computers today still have trouble recognizing a handwritten A.... In Hofstadter’s mind, there is nothing to be surprised about. To know what all A’s have in common would be, he argued in a 1982 essay, to “understand the fluid nature of mental categories.” And that, he says, is the core of human intelligence. I would say the modern era of AI was kicked off with AlexNet in 2012 and hit its stride a few years later. So, I believe this quote and this article are really referring to pre-GPU AI techniques. Basically, the predictions from this era just underestimated the value of scaling the data. Modern AI has several impressive achievements. It can certainly recognize an A. Image recognition and voice recognition are now critical parts of real products that help people in their everyday lives. At the same time, it's true that current AI techniques might not get us all the way to AGI. We'll just have to wait and see. But I think it's important to recognize that we have had real progress in the modern AI era that has seriously outperformed the pessimistic expectations from ~2010.

StrictDabbler 6y ago
Modern AI cannot recognize the kind of A that Hofstadter wrote about in that essay. Hofstadter identifies novelty typefaces where A has none of the characteristics that you might associate with the letter. Often A is not a triangle, sure, but sometimes it has no upper bar, no left stroke, no right stroke. Sometimes the bars are defined by a transition from negative to positive space. Sometimes A is made of 20 strokes, or a pattern of dots, or an open eye where the half-circular upper lid defines the triangle and the pupil is the bar. A neural network trained on conventional As would not conclude that any of these are an A. These shapes wouldn't necessarily read as A in isolation. Sometimes they only read correctly when you're looking at text using a whole typeface defined on the same principles. Modern AI still cannot recognize that kind of A even though a human easily can. There are ways to make an A that require a human to understand an idea of "letterness" that's different than anything they've seen before. I'd recommend reading the original essay. The illustrations make this clear in a way my description can't.