5 ms·

> So, about four years ago, I went on one of my week-long retreats, where I just take a computer and a stack of reference materials and I spend a week kind of

> So, about four years ago, I went on one of my week-long retreats, where I just take a computer and a stack of reference materials and I spend a week kind of reimplementing the fundamentals of the industry. And getting to the point where it’s like, ‘All right, I understand this well enough to have a serious conversation with a researcher about it.’ And I was pretty excited about getting to that level of understanding. As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded. Unless he can produce something that outdoes stable diffusion, chatgpt, alphago etc he should just hand over technical leadership of his start up to a leading AI researcher. Even Yann Le Cun at Meta is struggling to make any progress and is keeping himself busy by calling every other research labs output pedestrian. We cannot take any of Carmacks AGI predictions seriously, he simply lacks any expertise in the field.

dharma1 4y ago
He’s been working on machine learning long enough now to have some chance of success. It may go the way of his rocket ambitions (nothing comes out of it) but let the man try
carabiner 4y ago
Yep - he also failed with his lean, simple, first-principles approach with Armadillo Aerospace. The guy is proudly uncreative, and so he could have never come up with Scaled's design of a variable geometry rocket ship launched from a jet mothership.
WoodenChair 4y ago
> As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. Publishing papers is the way the academic/scientific world measures notability and/or competence. It's not the way the engineering world that Carmack comes from measures it. They measure it by building. But you're right, we kind of have to just trust that he has the expertise he says he does by his statements since he has not built any modern AI programs (that I know of at least). > If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded. This is not an accurate account of how he said he developed his knowledge base. Just how he got started so he could have conversations. He said that he spent a retreat learning the basics and then later in the interview he said he took the time to understand the 40 most essential papers in the field as related to him by a well known researcher. He has since largely put the last 4 years of his professional life into this. While we have no proof of his knowledge, given his intelligence and high competence in computer programming and math, I have no doubt that if he did put in the work he could achieve an understanding equivalent to that of your average AI researcher. That said, of course it makes sense to be skeptical.
- tgv 4y ago
  AI researcher perhaps, but almost none of them understand cognition. They're focussed on getting something that vaguely resembles a part of the brain to predict the next most likely token. Their idea of cognition apparently stops at Skinner.
  - Al-Khwarizmi 4y ago
    Any AI researcher worth their salt knows that those models aren't representative of how the human brain works... but they're just the kind of models that work best out of what we know how to implement right now. There are models with stronger cognitive inspiration, but their performance is worse.
- random314 4y ago
  > This is not an accurate account of how he said he developed his knowledge base. I quoted him directly, because I was expecting this kind of response. He took a week off and implemented some stuff from the ground up and was ready to have serious conversations with AI researchers. The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints. That's quite a low bar, especially if you are prone to skip the math and simply read the abstract and conclusions. His whole approach gives me a ML for hackers vibe and his thoughts on AGI, if it had come from anyone else, would have been described as word salad.
  - WoodenChair 4y ago
    > The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints. I would say it’s more likely John Carmack is capable of learning the state-of-the-art of AI from 40 papers than a random (pun intended based on username) from 100.
    - random314 4y ago
      Sure, he must be faster than Geoff Hinton too and it took Hinton a life time. Funnily enough, I am able to publish ML papers - but John Carmack isn't. I wonder why. I would also like to learn more about all the computer graphics algorithms Carmack has invented before I trust him to invent AGI. Here is one example of a person I am familiar with - Math Olympiad bronze medalist. Princeton PhD in ML Theory. AI researcher in Google. https://scholar.google.com/citations?user=gZgQLkgAAAAJ&hl=en https://scholar.google.com/citations?user=gZgQLkgAAAAJ&hl=en Sadly enough, nobody seems to care about his opinion on AGI but we have 1000s of people hanging off Carmacks words because he built Wolf3D and Doom.
jillesvangurp 4y ago
Anything you study for a few months you can become the world's leading expert on. It's a lesson I learned while doing my Ph. D. That's all it takes. After a few weeks, you get to the point where there are only a few others in the world that have read and are able to understand what you've read. A few months on, you are generating new ideas and insights. They might be wrong. But they won't be uninformed. John Carmack did not start from zero. He already has a firm grasp on algorithms related to linear algebra. Basically machine learning is a whole bunch of matrix manipulation. He's been doing that for 3 decades. The rest is just absorbing concepts about how to apply linear algebra to ML. I'd say he's probably uniquely qualified to really absorb a lot of knowledge quickly on this. It's not about publishing papers, it's about reading and understanding the right papers. I have no doubt he can chew his way through lots of research material in a week or so.
- random314 4y ago
  If it is simply about linear algebra, can you please read this ML Paper[1], go through all the proofs and lemmas over a week? You already have a PhD in ML, should be easy. Every kid graduating in STEM understands/should understand linear algebra. Knowing linear algebra is such a low bar. [1] https://arxiv.org/pdf/1904.09237 https://arxiv.org/pdf/1904.09237
  - lostmsu 4y ago
    Frankly, one does not need this paper to get towards the AI. Adam the optimization algo you might need (and even there I am not sure). And it is very readable. The fact that this particular proof of Adam's convergence is complicated is largely irrelevant. https://arxiv.org/pdf/1412.6980.pdf https://arxiv.org/pdf/1412.6980.pdf
    - random314 4y ago
      Yes, if you set the bar low enough - everything is easy and can be learnt in a month.
      - 4y ago

dharma1 4y ago

He’s been working on machine learning long enough now to have some chance of success. It may go the way of his rocket ambitions (nothing comes out of it) but let the man try

carabiner 4y ago

Yep - he also failed with his lean, simple, first-principles approach with Armadillo Aerospace. The guy is proudly uncreative, and so he could have never come up with Scaled's design of a variable geometry rocket ship launched from a jet mothership.

WoodenChair 4y ago

> As much as I respect Carmack as a computer graphics expert, I really doubt his competence in machine learning. He doesn't have a single notable paper published. Publishing papers is the way the academic/scientific world measures notability and/or competence. It's not the way the engineering world that Carmack comes from measures it. They measure it by building. But you're right, we kind of have to just trust that he has the expertise he says he does by his statements since he has not built any modern AI programs (that I know of at least). > If he really thought that implementing gradient descent and basic stuff in a week long retreat gave him the chops to have serious conversations with AI researchers, he is really deluded. This is not an accurate account of how he said he developed his knowledge base. Just how he got started so he could have conversations. He said that he spent a retreat learning the basics and then later in the interview he said he took the time to understand the 40 most essential papers in the field as related to him by a well known researcher. He has since largely put the last 4 years of his professional life into this. While we have no proof of his knowledge, given his intelligence and high competence in computer programming and math, I have no doubt that if he did put in the work he could achieve an understanding equivalent to that of your average AI researcher. That said, of course it makes sense to be skeptical.

tgv 4y ago
AI researcher perhaps, but almost none of them understand cognition. They're focussed on getting something that vaguely resembles a part of the brain to predict the next most likely token. Their idea of cognition apparently stops at Skinner.
- Al-Khwarizmi 4y ago
  Any AI researcher worth their salt knows that those models aren't representative of how the human brain works... but they're just the kind of models that work best out of what we know how to implement right now. There are models with stronger cognitive inspiration, but their performance is worse.
random314 4y ago
> This is not an accurate account of how he said he developed his knowledge base. I quoted him directly, because I was expecting this kind of response. He took a week off and implemented some stuff from the ground up and was ready to have serious conversations with AI researchers. The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints. That's quite a low bar, especially if you are prone to skip the math and simply read the abstract and conclusions. His whole approach gives me a ML for hackers vibe and his thoughts on AGI, if it had come from anyone else, would have been described as word salad.
- WoodenChair 4y ago
  > The 40 papers by Ilya came later. I have read a 100 ML papers and reviewed preprints. I would say it’s more likely John Carmack is capable of learning the state-of-the-art of AI from 40 papers than a random (pun intended based on username) from 100.
  - random314 4y ago
    Sure, he must be faster than Geoff Hinton too and it took Hinton a life time. Funnily enough, I am able to publish ML papers - but John Carmack isn't. I wonder why. I would also like to learn more about all the computer graphics algorithms Carmack has invented before I trust him to invent AGI. Here is one example of a person I am familiar with - Math Olympiad bronze medalist. Princeton PhD in ML Theory. AI researcher in Google. https://scholar.google.com/citations?user=gZgQLkgAAAAJ&hl=en https://scholar.google.com/citations?user=gZgQLkgAAAAJ&hl=en Sadly enough, nobody seems to care about his opinion on AGI but we have 1000s of people hanging off Carmacks words because he built Wolf3D and Doom.

jillesvangurp 4y ago

Anything you study for a few months you can become the world's leading expert on. It's a lesson I learned while doing my Ph. D. That's all it takes. After a few weeks, you get to the point where there are only a few others in the world that have read and are able to understand what you've read. A few months on, you are generating new ideas and insights. They might be wrong. But they won't be uninformed. John Carmack did not start from zero. He already has a firm grasp on algorithms related to linear algebra. Basically machine learning is a whole bunch of matrix manipulation. He's been doing that for 3 decades. The rest is just absorbing concepts about how to apply linear algebra to ML. I'd say he's probably uniquely qualified to really absorb a lot of knowledge quickly on this. It's not about publishing papers, it's about reading and understanding the right papers. I have no doubt he can chew his way through lots of research material in a week or so.

random314 4y ago
If it is simply about linear algebra, can you please read this ML Paper[1], go through all the proofs and lemmas over a week? You already have a PhD in ML, should be easy. Every kid graduating in STEM understands/should understand linear algebra. Knowing linear algebra is such a low bar. [1] https://arxiv.org/pdf/1904.09237 https://arxiv.org/pdf/1904.09237
- lostmsu 4y ago
  Frankly, one does not need this paper to get towards the AI. Adam the optimization algo you might need (and even there I am not sure). And it is very readable. The fact that this particular proof of Adam's convergence is complicated is largely irrelevant. https://arxiv.org/pdf/1412.6980.pdf https://arxiv.org/pdf/1412.6980.pdf
  - random314 4y ago
    Yes, if you set the bar low enough - everything is easy and can be learnt in a month.
    - 4y ago