7 ms·

Wishful thinking, make large and dumb models even larger, throw in more data and it will somehow magically lead to artificial common sense. The only thing missi

Wishful thinking, make large and dumb models even larger, throw in more data and it will somehow magically lead to artificial common sense. The only thing missing is philosopher's stone and the right mix of hyperparameters.

R0b0t1 6y ago
>make large and dumb models even larger, throw in more data and it will somehow magically lead to artificial common sense. What exactly do you think a human brain is?
- heavyset_go 6y ago
  Not an organ that performs backpropagation or gradient descent.
  - AQXt 6y ago
    What if I tell you it is? Will that information back-propagate into your brain?
    - heavyset_go 6y ago
      I'd ask you to show me how and where backpropogation takes place in the brain. Then I'd ask how thin you're willing to stretch the "neural net" metaphor to re-apply it to biology.
  - R0b0t1 6y ago
    Admittedly backpropogation/gradient descent bother me a fair bit but in a human brain what is desireable still has some kind of encoding, the mechanism is just more complicated and more general.
- nradov 6y ago
  We don't know exactly what a human brain is. That's kind of the problem.
  - bra-ket 6y ago
    Well we do know something that was learned by brilliant scientists over more than 100 years of neuroscience and cognition research, but apparently it's almost completely ignored by people who aspire to create an 'intelligence'. The idea of neural network in its current form came from someone in 1943 going through actual neuroscience papers and translating them to mathematical model by abstracting away the noise. While brilliant, Mccullough and Pitts work was done in pre-historic times if you consider the amount of information we learned about the brain and cognition since. Saying "We don't know much about human brain" is just being lazy. We know too much about the brain! Brain research has collected so much empirical data and a good amount of good theories. For an example of a good theory at various levels of abstraction see Moser, O'Keefe and Nadel work on place cells and grid cells, Kandel on learning, Edmund Rolls on vision, Olshausen & Field on sparse coding, Kanerva' SDM, Plate' convolutions, Widdows on geometry and meaning, Wickelgren and J. R. Anderson on associative memory, Fukushima' neocognitron, Hofstadter on analogical reasoning, Quillian on semantic memory, Pribram holonomic theory, Valiant' neuroids, Pearl' causality. More of these "bridges" and a meta-bridge is needed if you're serious about AI.
    - jugg1es 6y ago
      You are mixing up our ability to understand certain components of brain function (definitely respectable!) with our understanding of how the brain confers what we consider human-level intelligence and cognition (virtually none!).
      - bra-ket 6y ago
        well, you've got to start somewhere, and arguably only by understanding something limited, like navigation in innate coordinate space takes us to a higher abstraction like concept spaces, which in turn can take us to higher cognitive abilities. And we do know a whole lot on how navigation works in animal and human brain, for example, and there is an active research on abstract representations in the brain as an extension of spatial navigation "toolbox". reduction is a basic principle of science.
fossuser 6y ago
Well, there is something kind of interesting here. 5 + 5 = _ Predict what's next. The most accurate way to do this is to understand how arithmetic actually works. Sure, if you've only seen a handful of examples (or this exact example) you may have memorized 10 comes next or you may know a number comes next, but not which number. If you've seen enormous amounts of this you may deduce the underlying rules in order to more accurately predict what comes next. There's evidence of this happening with gpt-3. The scaling hypothesis may be stronger than people suspect.
- typon 6y ago
  What's the evidence?
  - fossuser 6y ago
    https://www.gwern.net/newsletter/2020/05#meta-learning https://www.gwern.net/newsletter/2020/05#meta-learning
    - typon 6y ago
      The site you linked asserts "meta-learning" is happening in GPT3 without any evidence of it
      - fossuser 6y ago
        Isn't the capability to answer arithmetic prompts without the exact equation being present in the training data evidence? I'm not an AI researcher so I only have a lay-person's opinion from reading about this kind of thing via things like that blog post (I also don't have OpenAI access to play with the API myself), but if that is an accurate statement it seems like pretty good evidence of figuring out the underlying mathematical rules. I think this could be a path towards scaling being more effective than people think. I sense though that you're just arguing from a position of motivated reasoning based on a pre-existing conclusion rather than actually trying to look at new things that may contradict what you already believe to be true. Repeated one sentence responses arguing that none of this is evidence just isn't worth time to respond to. In the end we'll see what ends up happening.

R0b0t1 6y ago

>make large and dumb models even larger, throw in more data and it will somehow magically lead to artificial common sense. What exactly do you think a human brain is?

heavyset_go 6y ago
Not an organ that performs backpropagation or gradient descent.
- AQXt 6y ago
  What if I tell you it is? Will that information back-propagate into your brain?
  - heavyset_go 6y ago
    I'd ask you to show me how and where backpropogation takes place in the brain. Then I'd ask how thin you're willing to stretch the "neural net" metaphor to re-apply it to biology.
- R0b0t1 6y ago
  Admittedly backpropogation/gradient descent bother me a fair bit but in a human brain what is desireable still has some kind of encoding, the mechanism is just more complicated and more general.
nradov 6y ago
We don't know exactly what a human brain is. That's kind of the problem.
- bra-ket 6y ago
  Well we do know something that was learned by brilliant scientists over more than 100 years of neuroscience and cognition research, but apparently it's almost completely ignored by people who aspire to create an 'intelligence'. The idea of neural network in its current form came from someone in 1943 going through actual neuroscience papers and translating them to mathematical model by abstracting away the noise. While brilliant, Mccullough and Pitts work was done in pre-historic times if you consider the amount of information we learned about the brain and cognition since. Saying "We don't know much about human brain" is just being lazy. We know too much about the brain! Brain research has collected so much empirical data and a good amount of good theories. For an example of a good theory at various levels of abstraction see Moser, O'Keefe and Nadel work on place cells and grid cells, Kandel on learning, Edmund Rolls on vision, Olshausen & Field on sparse coding, Kanerva' SDM, Plate' convolutions, Widdows on geometry and meaning, Wickelgren and J. R. Anderson on associative memory, Fukushima' neocognitron, Hofstadter on analogical reasoning, Quillian on semantic memory, Pribram holonomic theory, Valiant' neuroids, Pearl' causality. More of these "bridges" and a meta-bridge is needed if you're serious about AI.
  - jugg1es 6y ago
    You are mixing up our ability to understand certain components of brain function (definitely respectable!) with our understanding of how the brain confers what we consider human-level intelligence and cognition (virtually none!).
    - bra-ket 6y ago
      well, you've got to start somewhere, and arguably only by understanding something limited, like navigation in innate coordinate space takes us to a higher abstraction like concept spaces, which in turn can take us to higher cognitive abilities. And we do know a whole lot on how navigation works in animal and human brain, for example, and there is an active research on abstract representations in the brain as an extension of spatial navigation "toolbox". reduction is a basic principle of science.

fossuser 6y ago

Well, there is something kind of interesting here. 5 + 5 = _ Predict what's next. The most accurate way to do this is to understand how arithmetic actually works. Sure, if you've only seen a handful of examples (or this exact example) you may have memorized 10 comes next or you may know a number comes next, but not which number. If you've seen enormous amounts of this you may deduce the underlying rules in order to more accurately predict what comes next. There's evidence of this happening with gpt-3. The scaling hypothesis may be stronger than people suspect.

typon 6y ago
What's the evidence?
- fossuser 6y ago
  https://www.gwern.net/newsletter/2020/05#meta-learning https://www.gwern.net/newsletter/2020/05#meta-learning
  - typon 6y ago
    The site you linked asserts "meta-learning" is happening in GPT3 without any evidence of it
    - fossuser 6y ago
      Isn't the capability to answer arithmetic prompts without the exact equation being present in the training data evidence? I'm not an AI researcher so I only have a lay-person's opinion from reading about this kind of thing via things like that blog post (I also don't have OpenAI access to play with the API myself), but if that is an accurate statement it seems like pretty good evidence of figuring out the underlying mathematical rules. I think this could be a path towards scaling being more effective than people think. I sense though that you're just arguing from a position of motivated reasoning based on a pre-existing conclusion rather than actually trying to look at new things that may contradict what you already believe to be true. Repeated one sentence responses arguing that none of this is evidence just isn't worth time to respond to. In the end we'll see what ends up happening.