11 ms·

It's always a treat to watch a Carmack lecture or read anything he writes, and his notes here are no exception. He writes as an engineer, for engineers and docu

It's always a treat to watch a Carmack lecture or read anything he writes, and his notes here are no exception. He writes as an engineer, for engineers and documents all his thought processes and misteps in the exact detailed yet concise way you'd want a colleague to who was handing off some work. One question I would have about the research direction is the emphasis on realtime. If I understand correctly he's doing online learning in realtime. Obviously makes for a cool demo and pulls on his optimisation background, and no doubt some great innovations will be required to make this work. But I guess the bitter lesson and recent history also tell us that some solutions may only emerge at compute levels beyond what is currently possible for realtime inference let alone learning. And the only example we have of entities solving Atari games is the human brain, of which we don't have a clear understanding of the compute capacity. In which case, why wouldn't it be better to focus purely on learning efficiency and relax the realtime requirement for now? That's a genuine question by the way, definitely not an expert here and I'm sure there's a bunch of value to working within these constraints. I mean, jumping spiders solve reasonably complex problems with 100k neurons, so who knows.

suddenlybananas 1y ago
It's because humans (and other animals) have enormous innate capacities and knowledge which makes learning new things much much simpler than if you start from scratch. It's not really because of human's computational capacity.
- MrScruff 1y ago
  By innate do you mean evolved/instinctive? Surely even evolved behaviour must be expressed as brain function, and therefore would need a brain capable of handling that level of processing. I don't think it's clear how much of a human brains function exists at birth though, I know it's theorised than even much of the sensory processing has to be learned.
  - suddenlybananas 1y ago
    I'm not arguing against computational theory of mind, I'm just saying that innate behaviours don't require the same level of scale as learnt ones. Existing at birth is not the same thing as innate. Puberty is innate but it is not present at birth.
    - MrScruff 1y ago
      That's an interesting point. I can see that, as you say puberty and hormones impact brain function and hence behaviour, and those are inate and not learned. But at least superfically that would appear to be primarily broad behavioural effects, similar to what might be induced by medication. Rather than something that impacts pure abstract problem solving, which I guess is what the Atari games are supposed to represent?
      - rafaelmn 1y ago
        This is obviously wrong from genetic defects that cause predictable development problems in specialized areas. They are innate but not present at birth.
- xnx 1y ago
  > enormous innate capacities and knowledge Hundreds of millions of years of trial-and-error biological pre-training where survival/propagation is the reward function
  - suddenlybananas 1y ago
    Yes but it remains innate to the individual.
- Nopoint2 1y ago
  There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it? It just isn't needed. Just like you can find let's say kangaroos in the latent space of an image generator, so we learn abstract concepts and principles of how things work as a bonus of learning to process the senses. Maybe a way to AGI could be figuring out how to combine a video generator with a LLM or something similar in a way that allows it to understand things intuitively, instead of doing just lots and lots of some statistical bullsit.
  - Jensson 1y ago
    > There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it? We do have that, ever felt fear of heights? That isn't learned, we are born with it. Same with fear of small moving objects like spiders or snakes. Such things are learned/stored very different from memories, but its certainly there and we can see animals also have those. Like cats gets very scared of objects that are long and appear suddenly, like a cucumber, since their genetic instincts thinks its a snake.
    - Nopoint2 1y ago
      Of course it is learned, and fear is triggered by anything unfamiliar, that causes a high reconstruction error. Because it means you don't understand it, and it could be dangerous. We are just not used to encoding anything so deep below the eye level, and it freaks us out.
      - Jensson 1y ago
        Do you really think every single ant is learning all that on its own? And if ants can store that in their DNA, why don't you think other animals can? DNA works just fine as generic information storage, there are obviously a ton of behaviors and information encoded there from hundreds of millions of years of survival of the fittest.
      - 1y ago
johnb231 1y ago
From the notes: "A reality check for people that think full embodied AGI is right around the corner is to ask your dancing humanoid robot to pick up a joystick and learn how to play an obscure video game."
- throw_nbvc1234 1y ago
  This sounds like a problem that could be solved around the corner with a caveat. Games generally are solvable for AI because they have feedback loops and a clear success or failure criteria. If the "picking up a Joystick" part is the limiting factor, sure. But why would we want robots to use an interface (especially a modern controller) heavily optimized for human hands; that seems like the definition of a horseless carriage. I'm sure if you compared a monkey and a dolphins performance using a joystick you'd get results that aren't really correlated with their intelligence. I would guess that if you gave robots an R2D2 like port to jack into and play a game, that problem could be solved relatively quickly.
  - mellosouls 1y ago
    The point isn't about learning video games its about learning tasks unrelated to its specific competency generally.
  - johnb231 1y ago
    No, the joystick part is really not the limiting factor. They’ve already done this with a direct software interface. Physical interface is a new challenge. But overall you are missing the point.
  - xnickb 1y ago
    Just like OpenAI early on promised us an AGI and showed us how it "solved" Dota 2. They also claimed it "learned" to play by playing itself only however it was clear that most of the advanced techniques were borrowed from existing AI and by observing humans. No surprise they gave up on that project completely and I doubt they'll ever engage in anything like that again. Money better spent on different marketing platforms.
nlitened 1y ago
> the human brain, of which we don't have a clear understanding of the compute capacity Neurons have finite (very low) speed of signal transfer, so just by measuring cognitive reaction time we can deduce upper bounds on how many _consecutive_ neuron connections are involved in reception, cognitive processing, and resulting reaction via muscles, even for very complex cognitive processes. And the number is just around 100 consecutive neurons involved one after another. So “the algorithm” could not be _that_ complex in the end (100x matmul+tanh?) Granted, a lot of parallelism and feedback loops are involved, but overall it gives me (and many others) an impression that when the AGI algorithm is ever found, it’s “mini” version should be able to run on modest 2025 hardware in real time.
- johnb231 1y ago
  > (100x matmul+tanh?) Biological neurons are way more complex than that. A single neuron has dentritic trees with subunits doing their own local computations. There are temporal dynamics in the firing sequences. There is so much more complexity in the biological networks. It's not comparable.
  - woolion 1y ago
    You could implement a Turing-machine with humans acting physically operating as logic gates. Then, every human is just a boolean function.
    - Jensson 1y ago
      Neurons are stateful though, it is core to their function and how they learn.
  - neffy 1y ago
    This is exactly it. Biology is making massive use of hacked real time local network communication in ways we haven´t begun to explore.
- scajanus 1y ago
  The granted is doing a lot of work there. In fact, if you imagine a computer being able to do similar tasks as human brain can in around 100 steps, it becomes clear that considering parallelism is absolutely critical.
kilpikaarna 1y ago
I'm sure there were offline rendering and 3D graphics workstation people saying the same about the comparatively crude work he was doing in the early 90s... Obviously both Carmack and the rest of the world has changed since then, but it seems to me his main strength has always been in doing more with less (early id/Oculus, AA). When he's working in bigger orgs and/or with more established tech his output seems to suffer, at least in my view (possibly in his as well since he quit both Bethesda-id and Meta). I don't know Carmack and can't claim to be anywhere close to his level, but as someone also mainly interested in realtime stuff I can imagine he also feels a slight disdain for the throw-more-compute-at-it approach of the current AI boom. I'm certainly glad he's not running around asking for investor money to train an LLM. Best case scenario he teams up with some people who complement his skillset (akin to the game designers and artists at id back in the day) and comes up with a way to help bring some of the cutting edge to the masses, like with 3D graphics.
- LarsDu88 1y ago
  The thing about Carmack in the 90s... There was a lot of research going on around 3d graphics. Companies like SGI and Pixar were building specialized workstations for doing vector operations for 3d rendering. 3d was a thing. Game consoles with specialized 3d hardware would launch in 1994 with the Sega Saturn and the Sony Playstation (in Japan only for one year) What Carmack did was basically get a 3d game running on existing COMMODITY hardware. The 386 chip that most people used for their excel spreadsheets did not do floating point operations well, so Carmack figured out how to do everything using integers. May 1992 -> Wolfenstein 3d releases December 1993 -> Doom releases December 1994 -> Sony Playstation launches in Japan June 1996 -> Quake releases So Wolfenstein and Doom were actually not really 3d games, but rather 2.5 games (you can't have rooms below other rooms). The first 3d game here is actually Quake which also eventually also got hardware acceleration support. Carmack was the master of doing the seeminly impossible on super constrained hardware on virtually impossible timelines. If DOOM released in 1994 or 1995, would we still remember it in the same way?
  - hx8 1y ago
    > If DOOM released in 1994 or 1995, would we still remember it in the same way? Maybe. One aspect of Wolfenstein and Doom's popularity is that it was years ahead of everyone else technically on PC hardware. The other aspect is that they were genre defining titles that set the standards for gameplay design. I think Doom Deathmatch would have caught on in 1995, as there really were very few (just Command and Conquer?) standout PC network multiplayer games released between 1993 and 1995.

suddenlybananas 1y ago

It's because humans (and other animals) have enormous innate capacities and knowledge which makes learning new things much much simpler than if you start from scratch. It's not really because of human's computational capacity.

MrScruff 1y ago
By innate do you mean evolved/instinctive? Surely even evolved behaviour must be expressed as brain function, and therefore would need a brain capable of handling that level of processing. I don't think it's clear how much of a human brains function exists at birth though, I know it's theorised than even much of the sensory processing has to be learned.
- suddenlybananas 1y ago
  I'm not arguing against computational theory of mind, I'm just saying that innate behaviours don't require the same level of scale as learnt ones. Existing at birth is not the same thing as innate. Puberty is innate but it is not present at birth.
  - MrScruff 1y ago
    That's an interesting point. I can see that, as you say puberty and hormones impact brain function and hence behaviour, and those are inate and not learned. But at least superfically that would appear to be primarily broad behavioural effects, similar to what might be induced by medication. Rather than something that impacts pure abstract problem solving, which I guess is what the Atari games are supposed to represent?
    - rafaelmn 1y ago
      This is obviously wrong from genetic defects that cause predictable development problems in specialized areas. They are innate but not present at birth.
xnx 1y ago
> enormous innate capacities and knowledge Hundreds of millions of years of trial-and-error biological pre-training where survival/propagation is the reward function
- suddenlybananas 1y ago
  Yes but it remains innate to the individual.
Nopoint2 1y ago
There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it? It just isn't needed. Just like you can find let's say kangaroos in the latent space of an image generator, so we learn abstract concepts and principles of how things work as a bonus of learning to process the senses. Maybe a way to AGI could be figuring out how to combine a video generator with a LLM or something similar in a way that allows it to understand things intuitively, instead of doing just lots and lots of some statistical bullsit.
- Jensson 1y ago
  > There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it? We do have that, ever felt fear of heights? That isn't learned, we are born with it. Same with fear of small moving objects like spiders or snakes. Such things are learned/stored very different from memories, but its certainly there and we can see animals also have those. Like cats gets very scared of objects that are long and appear suddenly, like a cucumber, since their genetic instincts thinks its a snake.
  - Nopoint2 1y ago
    Of course it is learned, and fear is triggered by anything unfamiliar, that causes a high reconstruction error. Because it means you don't understand it, and it could be dangerous. We are just not used to encoding anything so deep below the eye level, and it freaks us out.
    - Jensson 1y ago
      Do you really think every single ant is learning all that on its own? And if ants can store that in their DNA, why don't you think other animals can? DNA works just fine as generic information storage, there are obviously a ton of behaviors and information encoded there from hundreds of millions of years of survival of the fittest.
    - 1y ago

johnb231 1y ago

From the notes: "A reality check for people that think full embodied AGI is right around the corner is to ask your dancing humanoid robot to pick up a joystick and learn how to play an obscure video game."

throw_nbvc1234 1y ago
This sounds like a problem that could be solved around the corner with a caveat. Games generally are solvable for AI because they have feedback loops and a clear success or failure criteria. If the "picking up a Joystick" part is the limiting factor, sure. But why would we want robots to use an interface (especially a modern controller) heavily optimized for human hands; that seems like the definition of a horseless carriage. I'm sure if you compared a monkey and a dolphins performance using a joystick you'd get results that aren't really correlated with their intelligence. I would guess that if you gave robots an R2D2 like port to jack into and play a game, that problem could be solved relatively quickly.
- mellosouls 1y ago
  The point isn't about learning video games its about learning tasks unrelated to its specific competency generally.
- johnb231 1y ago
  No, the joystick part is really not the limiting factor. They’ve already done this with a direct software interface. Physical interface is a new challenge. But overall you are missing the point.
- xnickb 1y ago
  Just like OpenAI early on promised us an AGI and showed us how it "solved" Dota 2. They also claimed it "learned" to play by playing itself only however it was clear that most of the advanced techniques were borrowed from existing AI and by observing humans. No surprise they gave up on that project completely and I doubt they'll ever engage in anything like that again. Money better spent on different marketing platforms.

nlitened 1y ago

> the human brain, of which we don't have a clear understanding of the compute capacity Neurons have finite (very low) speed of signal transfer, so just by measuring cognitive reaction time we can deduce upper bounds on how many _consecutive_ neuron connections are involved in reception, cognitive processing, and resulting reaction via muscles, even for very complex cognitive processes. And the number is just around 100 consecutive neurons involved one after another. So “the algorithm” could not be _that_ complex in the end (100x matmul+tanh?) Granted, a lot of parallelism and feedback loops are involved, but overall it gives me (and many others) an impression that when the AGI algorithm is ever found, it’s “mini” version should be able to run on modest 2025 hardware in real time.

johnb231 1y ago
> (100x matmul+tanh?) Biological neurons are way more complex than that. A single neuron has dentritic trees with subunits doing their own local computations. There are temporal dynamics in the firing sequences. There is so much more complexity in the biological networks. It's not comparable.
- woolion 1y ago
  You could implement a Turing-machine with humans acting physically operating as logic gates. Then, every human is just a boolean function.
  - Jensson 1y ago
    Neurons are stateful though, it is core to their function and how they learn.
- neffy 1y ago
  This is exactly it. Biology is making massive use of hacked real time local network communication in ways we haven´t begun to explore.
scajanus 1y ago
The granted is doing a lot of work there. In fact, if you imagine a computer being able to do similar tasks as human brain can in around 100 steps, it becomes clear that considering parallelism is absolutely critical.

kilpikaarna 1y ago

I'm sure there were offline rendering and 3D graphics workstation people saying the same about the comparatively crude work he was doing in the early 90s... Obviously both Carmack and the rest of the world has changed since then, but it seems to me his main strength has always been in doing more with less (early id/Oculus, AA). When he's working in bigger orgs and/or with more established tech his output seems to suffer, at least in my view (possibly in his as well since he quit both Bethesda-id and Meta). I don't know Carmack and can't claim to be anywhere close to his level, but as someone also mainly interested in realtime stuff I can imagine he also feels a slight disdain for the throw-more-compute-at-it approach of the current AI boom. I'm certainly glad he's not running around asking for investor money to train an LLM. Best case scenario he teams up with some people who complement his skillset (akin to the game designers and artists at id back in the day) and comes up with a way to help bring some of the cutting edge to the masses, like with 3D graphics.

LarsDu88 1y ago
The thing about Carmack in the 90s... There was a lot of research going on around 3d graphics. Companies like SGI and Pixar were building specialized workstations for doing vector operations for 3d rendering. 3d was a thing. Game consoles with specialized 3d hardware would launch in 1994 with the Sega Saturn and the Sony Playstation (in Japan only for one year) What Carmack did was basically get a 3d game running on existing COMMODITY hardware. The 386 chip that most people used for their excel spreadsheets did not do floating point operations well, so Carmack figured out how to do everything using integers. May 1992 -> Wolfenstein 3d releases December 1993 -> Doom releases December 1994 -> Sony Playstation launches in Japan June 1996 -> Quake releases So Wolfenstein and Doom were actually not really 3d games, but rather 2.5 games (you can't have rooms below other rooms). The first 3d game here is actually Quake which also eventually also got hardware acceleration support. Carmack was the master of doing the seeminly impossible on super constrained hardware on virtually impossible timelines. If DOOM released in 1994 or 1995, would we still remember it in the same way?
- hx8 1y ago
  > If DOOM released in 1994 or 1995, would we still remember it in the same way? Maybe. One aspect of Wolfenstein and Doom's popularity is that it was years ahead of everyone else technically on PC hardware. The other aspect is that they were genre defining titles that set the standards for gameplay design. I think Doom Deathmatch would have caught on in 1995, as there really were very few (just Command and Conquer?) standout PC network multiplayer games released between 1993 and 1995.