5 ms·

You can’t just say that a human comes from a gigabyte of information. That info is fed into a massively complex physical system, which he is ignoring.

marviel 4y ago
I don't disagree that our culture gives us a lot. To use ML terms -- Humans have "Foundation Models" which are composed of: - Their Biological makeup - The culture into which they are raised
- serverholic 4y ago
  I'm not even talking about culture. DNA is only useful because the laws of physical reality are built a certain way. Without that context the information within DNA is meaningless.
  - marviel 4y ago
    Well sure but couldn't the same thing be said about a computer program?
    - serverholic 4y ago
      A good analogy is perhaps a config file for a computer program. My interpretation is that Carmack is essentially confusing a config file for the computer program itself, then saying "look how small it is, this shouldn't be that hard to reverse engineer".
kmoser 4y ago
Yes, this is where Carmack's optimism goes awry. He says: "...as soon as you’re at the point where you have the equivalent of a toddler—something that is a being, it’s conscious, it’s not Einstein, it can’t even do multiplication—if you’ve got a creature that can learn, you can interact with and teach it things on some level." He's wrong. There is currently no practical way to produce a software system that possesses the ability for human thought, reasoning, and motivation without that system possessing uniquely human (let alone organic) properties: the biological and chemical makeup, plus the physical characteristics, of a human, and the ability to process using human senses. (Hint: a neural net processing video images is a mere shadow of how a human processes things with their senses of sight, sound, and touch.) Carmack thinks humans can be reliably reduced to spherical cows in a vacuum, but that only holds true on paper. A real human being is not merely a meat machine: we are driven largely by emotions and physical desires, none of which exist in a computer except through elaborate simulation thereof. Now, I'm sure over the next couple of decades we will make huge strides in mimicking a human being's ability to learn, i.e. creating ever more complex LLMs and AI models that act increasingly more humanlike, but they will be nothing but more and more elaborate parlor tricks which, when prodded just the right way, will fail completely, revealing that they were never human at all. They will be like Avatars (from the movie): sophisticated simulacra that rely on elaborate behind-the-scenes support systems, without which they are useless.
- serverholic 4y ago
  I don’t think he actually cares about mimicking humans, I think he just wants a truly intelligent AI system that can learn in a sophisticated way like humans do. I think that’s actually pretty doable. Take for example flying. We don’t build airplanes that flap their wings because we have a deeper understanding of flight that allows us to build flying machines far beyond the capabilities of any animal. Likewise, once we understand the mechanics of intelligence we should be able to build something that can learn that is completely computer based.
  - kmoser 4y ago
    AGI not bound to a physical presence is almost an oxymoron. Without basic motivation (hunger, sleep, happiness, etc.), how will it learn? What is its incentive to actually do anything? What's to prevent it from telling us to take a hike? "Intelligence" is not something easily abstracted away from the physical world, especially if you want something to learn on its own. How will an AGI to learn that stoves are hot? The main challenge I see to creating such a system that can truly learn is that you will have to constrain it to have motivation to learn and follow your directions, and nothing else. And even if you could add such constraints, what would "learning" mean to such a device? What would stop it from going off on useless tangents, like attempting to count every grain of sand you show it? Anything with as much autonomy as it takes to have AGI will likely start coming to conclusions we don't want it to. My guess is that in the near future, either we'll create something that is beyond our ability to control effectively, or it will be yet another clever simulation of AGI that is not really AGI.

marviel 4y ago

I don't disagree that our culture gives us a lot. To use ML terms -- Humans have "Foundation Models" which are composed of: - Their Biological makeup - The culture into which they are raised

serverholic 4y ago
I'm not even talking about culture. DNA is only useful because the laws of physical reality are built a certain way. Without that context the information within DNA is meaningless.
- marviel 4y ago
  Well sure but couldn't the same thing be said about a computer program?
  - serverholic 4y ago
    A good analogy is perhaps a config file for a computer program. My interpretation is that Carmack is essentially confusing a config file for the computer program itself, then saying "look how small it is, this shouldn't be that hard to reverse engineer".

kmoser 4y ago

Yes, this is where Carmack's optimism goes awry. He says: "...as soon as you’re at the point where you have the equivalent of a toddler—something that is a being, it’s conscious, it’s not Einstein, it can’t even do multiplication—if you’ve got a creature that can learn, you can interact with and teach it things on some level." He's wrong. There is currently no practical way to produce a software system that possesses the ability for human thought, reasoning, and motivation without that system possessing uniquely human (let alone organic) properties: the biological and chemical makeup, plus the physical characteristics, of a human, and the ability to process using human senses. (Hint: a neural net processing video images is a mere shadow of how a human processes things with their senses of sight, sound, and touch.) Carmack thinks humans can be reliably reduced to spherical cows in a vacuum, but that only holds true on paper. A real human being is not merely a meat machine: we are driven largely by emotions and physical desires, none of which exist in a computer except through elaborate simulation thereof. Now, I'm sure over the next couple of decades we will make huge strides in mimicking a human being's ability to learn, i.e. creating ever more complex LLMs and AI models that act increasingly more humanlike, but they will be nothing but more and more elaborate parlor tricks which, when prodded just the right way, will fail completely, revealing that they were never human at all. They will be like Avatars (from the movie): sophisticated simulacra that rely on elaborate behind-the-scenes support systems, without which they are useless.

serverholic 4y ago
I don’t think he actually cares about mimicking humans, I think he just wants a truly intelligent AI system that can learn in a sophisticated way like humans do. I think that’s actually pretty doable. Take for example flying. We don’t build airplanes that flap their wings because we have a deeper understanding of flight that allows us to build flying machines far beyond the capabilities of any animal. Likewise, once we understand the mechanics of intelligence we should be able to build something that can learn that is completely computer based.
- kmoser 4y ago
  AGI not bound to a physical presence is almost an oxymoron. Without basic motivation (hunger, sleep, happiness, etc.), how will it learn? What is its incentive to actually do anything? What's to prevent it from telling us to take a hike? "Intelligence" is not something easily abstracted away from the physical world, especially if you want something to learn on its own. How will an AGI to learn that stoves are hot? The main challenge I see to creating such a system that can truly learn is that you will have to constrain it to have motivation to learn and follow your directions, and nothing else. And even if you could add such constraints, what would "learning" mean to such a device? What would stop it from going off on useless tangents, like attempting to count every grain of sand you show it? Anything with as much autonomy as it takes to have AGI will likely start coming to conclusions we don't want it to. My guess is that in the near future, either we'll create something that is beyond our ability to control effectively, or it will be yet another clever simulation of AGI that is not really AGI.