4 ms·

He also said in an interview with Jensen soon after ChatGPT's launch that "before 2003, machines couldn't learn" ... LOL. I was stunned when I heard that nonsen

He also said in an interview with Jensen soon after ChatGPT's launch that "before 2003, machines couldn't learn" ... LOL. I was stunned when I heard that nonsensical assertion. I guess it depends on his definition of "learn" ...

bbor 2y ago
"learn" is usually used in opposition to "taught", which refers to "expert systems"-type engineering; in other words, providing data and a success heuristic and asking it to devise its own optimal strategies vs. providing strategies hand-designed by humans. Obviously Perceptrons came out well before 2003, but I don't think it's necessarily out of line to say that they had limited efficacy before then, both for theoretical and compute reasons. But maybe I'm misunderstanding your criticism?
- remexre 2y ago
  ILP goes back to the 80s, and was used to do drug discovery in the 90s. Bayes nets go back to the 80s as well.
  - YeGoblynQueenne 2y ago
    "ILP" (as in Inductive Logic Programming not Integer Linear Programming) was first named in 1991 in a paper by Stephen Muggleton ("Inductive Logic Programming and Progol). The paper properly launched the field and generated a great deal of excitement at the time. There were precursors. At least Ehud Shapiro's doctoral thesis ("Automated Debugging") in the 1980's and Gordon Plotkin's doctoral thesis in the 1970's ("Automated Methods of Inductive Inference"). Sorry for not giving the exact years off the top of my head but I think it was 1983 and 1976, respectively. The point you are making is very right however because modern machine learning as a field started in the 1980's with the fall of expert systems, in fact it basically started as an effort to overcome one of the major limitations of expert systems, the so-called "knowledge acquisition bottleneck", which is to say, the difficulty of creating and maintaining huge databases of expert knowledge (in the form of production rules). In any case the seminal textbook in the field for the first 20 years, Tom Mitchell's Machine Learning came out in 1997 (https://www.cse.iitb.ac.in/~cs725/notes/slides/tom_mitchell/mlbook.html https://www.cse.iitb.ac.in/~cs725/notes/slides/tom_mitchell/...) and includes probabilistic, neural-net based and symbolic, logic-based approaches. So not only machines could "learn" way before 2003 but they could also learn in many different ways than what Ilya Sutskever means. We can go further back, to Donald Michie's 1961 MENACE (the first Reinforcement Learning system, implemented on a computer made of matchboxes with coloured beads used to encode state) and Arthur Samuel's 1959 checkers player (a paper on which gave the name to the field of machine learning). Lots of learning all over the place long, long before 2003.
    - katamari-damacy 2y ago
      Yes, so what does it say about our rigged system for a guy who neither invented the attention mechanism nor the concept of GPTs to be given that much credit for the current wave of AI? One lucky choice (betting on scale) backed by $100Ms of other people's money does not entitle one to genius-hood.
      - bbor 2y ago
        I mean, tbf, he hasn't won a Turing yet, right? So the academy hasn't fully embraced him as a genius. VCs/SV are more fickle, but even they aren't necessarily in love -- his Superintelligence startup raised a modest but far from unusual amount of cash, AFAIR

bbor 2y ago

"learn" is usually used in opposition to "taught", which refers to "expert systems"-type engineering; in other words, providing data and a success heuristic and asking it to devise its own optimal strategies vs. providing strategies hand-designed by humans. Obviously Perceptrons came out well before 2003, but I don't think it's necessarily out of line to say that they had limited efficacy before then, both for theoretical and compute reasons. But maybe I'm misunderstanding your criticism?

remexre 2y ago
ILP goes back to the 80s, and was used to do drug discovery in the 90s. Bayes nets go back to the 80s as well.
- YeGoblynQueenne 2y ago
  "ILP" (as in Inductive Logic Programming not Integer Linear Programming) was first named in 1991 in a paper by Stephen Muggleton ("Inductive Logic Programming and Progol). The paper properly launched the field and generated a great deal of excitement at the time. There were precursors. At least Ehud Shapiro's doctoral thesis ("Automated Debugging") in the 1980's and Gordon Plotkin's doctoral thesis in the 1970's ("Automated Methods of Inductive Inference"). Sorry for not giving the exact years off the top of my head but I think it was 1983 and 1976, respectively. The point you are making is very right however because modern machine learning as a field started in the 1980's with the fall of expert systems, in fact it basically started as an effort to overcome one of the major limitations of expert systems, the so-called "knowledge acquisition bottleneck", which is to say, the difficulty of creating and maintaining huge databases of expert knowledge (in the form of production rules). In any case the seminal textbook in the field for the first 20 years, Tom Mitchell's Machine Learning came out in 1997 (https://www.cse.iitb.ac.in/~cs725/notes/slides/tom_mitchell/mlbook.html https://www.cse.iitb.ac.in/~cs725/notes/slides/tom_mitchell/...) and includes probabilistic, neural-net based and symbolic, logic-based approaches. So not only machines could "learn" way before 2003 but they could also learn in many different ways than what Ilya Sutskever means. We can go further back, to Donald Michie's 1961 MENACE (the first Reinforcement Learning system, implemented on a computer made of matchboxes with coloured beads used to encode state) and Arthur Samuel's 1959 checkers player (a paper on which gave the name to the field of machine learning). Lots of learning all over the place long, long before 2003.
  - katamari-damacy 2y ago
    Yes, so what does it say about our rigged system for a guy who neither invented the attention mechanism nor the concept of GPTs to be given that much credit for the current wave of AI? One lucky choice (betting on scale) backed by $100Ms of other people's money does not entitle one to genius-hood.
    - bbor 2y ago
      I mean, tbf, he hasn't won a Turing yet, right? So the academy hasn't fully embraced him as a genius. VCs/SV are more fickle, but even they aren't necessarily in love -- his Superintelligence startup raised a modest but far from unusual amount of cash, AFAIR