4 ms·

> Hassabis and Silver and their fellow researchers have built a machine capable of something super-human. But at the same time, it's flawed. It can't do everyth

> Hassabis and Silver and their fellow researchers have built a machine capable of something super-human. But at the same time, it's flawed. It can't do everything we humans can do. In fact, it can't even come close. It can't carry on a conversation. It can't play charades. It can't pass an eighth grade science test. It can't account for God's Touch. That didn't age well

wslh 3y ago
I understand that the next test is about passing an elementary school science test? It is good to put cincrete goals. A few time ago computers couldn't beat amateur Go players. Not saying that computers will think or not, just saying we have new challenges before the Turing test.
- codedokode 3y ago
  But humans failed to find an algorithmic solution for Go. All they could do is to throw a lot of data and get a bunch of coefficients without discovering underlying rules. Same with drawing images and understanding language: this is not solved yet. This is like showing an answer on an exam but failing to explain how you got it. I doubt you can get away with this.
  - 317070 3y ago
    Wait, we know the underlying rules, we have an explanation. You can read all the coefficients. We don't understand the explanation, though it is correct. Not sure if the problem here is with the capabilities of the examinee or with the examiner.
    - codedokode 3y ago
      Imagine if you go to your work at the bank tomorrow and instead of a well documented, maintainable and formatted code see a gibberish. And your neural coworker tells you that it is just a problem with your capabilities if you cannot understand it. He just refactored it to improve performance. That's the situation with machine learning today.
      - Jack000 3y ago
        The thing is, it's not gibberish. A sufficiently small language model can be understood by humans: https://twitter.com/karpathy/status/1645115622517542913 https://twitter.com/karpathy/status/1645115622517542913 The explanation is perfectly sensical, just too complex for humans to understand as the model scales up. The thing you're looking for - a reductive explanation of the weights of a ANN that's easy to fit in your head, does not exist. If it were simple enough to satisfy your demands, it wouldn't work at all.
        zelphirkalt 3y ago
        Yet, when a master player makes a decision what move to play, they often have concrete reasons for it, that they discuss in after game analysis. They evaluate some advantage or chances higher than others or some risks greater than others and calculate specific sequences ahead to be sure to solve a subproblem correctly and base their decision on that.
      - pixl97 3y ago
        Banks don't typically attempt to solve P=NP problems. Meanwhile things like stock markets attempt to with things like partial future prediction, which means all possible outcomes are not calculable in finite time, hence they use things like ML/AI.
  - stefs 3y ago
    i'm not sure i understand you there. > But humans failed to find an algorithmic solution for Go. sure we have algorithmic solutions for go, they're just not very good. > All they could do is to throw a lot of data and get a bunch of coefficients without discovering underlying rules. that's not completely true either, the special thing about ~alphago~ alphazero* was that it learned by playing itself instead of learning from a pre-recorded catalog of human games (which is the reason for its - for humans - peculiar playstyle). now i'm not sure how you're arguing a neural network trained to play go doesn't understand the "underlying rules" of the game. to the contrary, it doesn't understand ANYTHING BUT the underlying rules. explaining why you did something isn't always easy for a human either. most times they couldn't say anything more concrete than "well it's obviously the best move according to my experience" without just making stuff up. *edit: mixed up alphago and alphazero
    - codedokode 3y ago
      By "underlying rules" I meant not rules of Go, but a detailed, commented algorithm that can win against human. Not a bunch of weights without any explanation.
      - wslh 3y ago
        It is possible that there is no algorithm that is understandable by normal humans or humans at all in the sense of the typical algorithmic approach of quick sort, etc. In other words, the algorithm is very long for a relatively reduced programming language.
  - wslh 3y ago
    Well, we humans also failed to find an algorithmic solution for our brain playing Go. I mean, the way AI and our brain work are mysterious. Surely at different level/layers but both share the mistery.
- reducesuffering 3y ago
  Not only has GPT4 passed an elementary school science test, it is outperforming on tests better than 95% of the world. https://twitter.com/emollick/status/1635700173946105856?lang=en https://twitter.com/emollick/status/1635700173946105856?lang...
__loam 3y ago
I think this still holds some water. The go bot was excellent at beating really good go players because it had a ton of data on high level go games. When it encountered someone intentionally doing a moronic strategy, it was trounced. It just didn't have enough data on bad players, so it lost to an obviously flawed strategy. I think that's the huge flaw in all of these ml systems. They don't build fundamental understanding. We're brute forcing it in a way, but perhaps we're losing something in the long tail. E: https://arstechnica.com/information-technology/2022/11/new-go-playing-trick-defeats-world-class-go-ai-but-loses-to-human-amateurs/ https://arstechnica.com/information-technology/2022/11/new-g...
- joshfee 3y ago
  This was a flaw in the original AlphaGo, but the subsequent AlphaZero (https://en.wikipedia.org/wiki/AlphaZero https://en.wikipedia.org/wiki/AlphaZero) trained entirely from self play with no prior information. So essentially it _does_ build fundamental understanding. I think the ability to learn by self play (essentially in a closed room without external training data) is where the line between "fundamental understanding" and "regurgitating information" from these AIs lie.
  - famouswaffles 3y ago
    There isn't really any difference between self play and no self play in terms of "fundamental understanding" and "regurgitation". It's the same training scheme just with different data.
  - white_beach 3y ago
    more recently https://www.science.org/doi/10.1126/sciadv.adg3256 https://www.science.org/doi/10.1126/sciadv.adg3256
taneq 3y ago
Will Smith's memorable character in I, Robot: Can you compose a symphony? Can you turn a blank canvas into a masterpiece? Sonny: Uh... yes?
- markwkw 3y ago
  Sonny: Yes! Rapidly and repeatedly. [then add the actual quote:] Can you? Human: ... Oh how the tables have turned.
  - taneq 3y ago
    "Cool. Create a 4x4 matrix of symphonies, with 'vivaciousness' on the X axis and 'dramatic themes' on the Y axis, then listen to them all and tell me which one's best." Sonny: "...oh god"
varelse 3y ago
[dead]
beepbooptheory 3y ago
2/4 (generously) is not bad for 7 years! And its not like we are ever going to get to 4/4... I give it a C- minus on the Evergreen Scale.
deleted 3y ago
[deleted]

wslh 3y ago

I understand that the next test is about passing an elementary school science test? It is good to put cincrete goals. A few time ago computers couldn't beat amateur Go players. Not saying that computers will think or not, just saying we have new challenges before the Turing test.

codedokode 3y ago
But humans failed to find an algorithmic solution for Go. All they could do is to throw a lot of data and get a bunch of coefficients without discovering underlying rules. Same with drawing images and understanding language: this is not solved yet. This is like showing an answer on an exam but failing to explain how you got it. I doubt you can get away with this.
- 317070 3y ago
  Wait, we know the underlying rules, we have an explanation. You can read all the coefficients. We don't understand the explanation, though it is correct. Not sure if the problem here is with the capabilities of the examinee or with the examiner.
  - codedokode 3y ago
    Imagine if you go to your work at the bank tomorrow and instead of a well documented, maintainable and formatted code see a gibberish. And your neural coworker tells you that it is just a problem with your capabilities if you cannot understand it. He just refactored it to improve performance. That's the situation with machine learning today.
    - Jack000 3y ago
      The thing is, it's not gibberish. A sufficiently small language model can be understood by humans: https://twitter.com/karpathy/status/1645115622517542913 https://twitter.com/karpathy/status/1645115622517542913 The explanation is perfectly sensical, just too complex for humans to understand as the model scales up. The thing you're looking for - a reductive explanation of the weights of a ANN that's easy to fit in your head, does not exist. If it were simple enough to satisfy your demands, it wouldn't work at all.
      - zelphirkalt 3y ago
        Yet, when a master player makes a decision what move to play, they often have concrete reasons for it, that they discuss in after game analysis. They evaluate some advantage or chances higher than others or some risks greater than others and calculate specific sequences ahead to be sure to solve a subproblem correctly and base their decision on that.
    - pixl97 3y ago
      Banks don't typically attempt to solve P=NP problems. Meanwhile things like stock markets attempt to with things like partial future prediction, which means all possible outcomes are not calculable in finite time, hence they use things like ML/AI.
- stefs 3y ago
  i'm not sure i understand you there. > But humans failed to find an algorithmic solution for Go. sure we have algorithmic solutions for go, they're just not very good. > All they could do is to throw a lot of data and get a bunch of coefficients without discovering underlying rules. that's not completely true either, the special thing about ~alphago~ alphazero* was that it learned by playing itself instead of learning from a pre-recorded catalog of human games (which is the reason for its - for humans - peculiar playstyle). now i'm not sure how you're arguing a neural network trained to play go doesn't understand the "underlying rules" of the game. to the contrary, it doesn't understand ANYTHING BUT the underlying rules. explaining why you did something isn't always easy for a human either. most times they couldn't say anything more concrete than "well it's obviously the best move according to my experience" without just making stuff up. *edit: mixed up alphago and alphazero
  - codedokode 3y ago
    By "underlying rules" I meant not rules of Go, but a detailed, commented algorithm that can win against human. Not a bunch of weights without any explanation.
    - wslh 3y ago
      It is possible that there is no algorithm that is understandable by normal humans or humans at all in the sense of the typical algorithmic approach of quick sort, etc. In other words, the algorithm is very long for a relatively reduced programming language.
- wslh 3y ago
  Well, we humans also failed to find an algorithmic solution for our brain playing Go. I mean, the way AI and our brain work are mysterious. Surely at different level/layers but both share the mistery.
reducesuffering 3y ago
Not only has GPT4 passed an elementary school science test, it is outperforming on tests better than 95% of the world. https://twitter.com/emollick/status/1635700173946105856?lang=en https://twitter.com/emollick/status/1635700173946105856?lang...

__loam 3y ago

I think this still holds some water. The go bot was excellent at beating really good go players because it had a ton of data on high level go games. When it encountered someone intentionally doing a moronic strategy, it was trounced. It just didn't have enough data on bad players, so it lost to an obviously flawed strategy. I think that's the huge flaw in all of these ml systems. They don't build fundamental understanding. We're brute forcing it in a way, but perhaps we're losing something in the long tail. E: https://arstechnica.com/information-technology/2022/11/new-go-playing-trick-defeats-world-class-go-ai-but-loses-to-human-amateurs/ https://arstechnica.com/information-technology/2022/11/new-g...

joshfee 3y ago
This was a flaw in the original AlphaGo, but the subsequent AlphaZero (https://en.wikipedia.org/wiki/AlphaZero https://en.wikipedia.org/wiki/AlphaZero) trained entirely from self play with no prior information. So essentially it _does_ build fundamental understanding. I think the ability to learn by self play (essentially in a closed room without external training data) is where the line between "fundamental understanding" and "regurgitating information" from these AIs lie.
- famouswaffles 3y ago
  There isn't really any difference between self play and no self play in terms of "fundamental understanding" and "regurgitation". It's the same training scheme just with different data.
- white_beach 3y ago
  more recently https://www.science.org/doi/10.1126/sciadv.adg3256 https://www.science.org/doi/10.1126/sciadv.adg3256

taneq 3y ago

Will Smith's memorable character in I, Robot: Can you compose a symphony? Can you turn a blank canvas into a masterpiece? Sonny: Uh... yes?

markwkw 3y ago
Sonny: Yes! Rapidly and repeatedly. [then add the actual quote:] Can you? Human: ... Oh how the tables have turned.
- taneq 3y ago
  "Cool. Create a 4x4 matrix of symphonies, with 'vivaciousness' on the X axis and 'dramatic themes' on the Y axis, then listen to them all and tell me which one's best." Sonny: "...oh god"

varelse 3y ago

[dead]

beepbooptheory 3y ago

2/4 (generously) is not bad for 7 years! And its not like we are ever going to get to 4/4... I give it a C- minus on the Evergreen Scale.

deleted 3y ago

[deleted]