4 ms·

Would that actually work, though, or would it just cook itself? I think the general question to ask is, for a given organism or machine, at what point does it

Would that actually work, though, or would it just cook itself? I think the general question to ask is, for a given organism or machine, at what point does it stop being cost effective to make it smarter? Nature doesn't just keep human intelligence at the level it currently is, it's doing so for all other organisms. There seems to be tradeoffs. Which ones would apply to AI? And which ones are not currently relevant but will apply to future AI? For example, AI has no physical predators currently, but you could imagine a future where they do, or where nanobots or special bacteria attack silicon. In such a future, it is possible that very intelligent AI would suddenly become uncompetitive, relative to dumber and leaner ones. Viable superintelligent AI might then turn out to be just as difficult or circumstantial to "evolve" as humans were.

eru 5y ago
> Would that actually work, though, or would it just cook itself? You could add better cooling than what the human hardware provides. Even just taking a normal human head and dunking it in cold water dissipates a lot more head. (If you try this at home, I suggest getting a snorkel.)
gonehome 5y ago
Why would it cook itself if it was replicated in silicon? I think it's a lot less likely humanity just happens to be at some natural limit of intelligence rather than being bounded by other unrelated constraints. AGI tradeoffs would be different because it'd be unconstrained from biological natural selection. If you're interested in this stuff it's worth checking out the work that specifies the AGI goal alignment problem more specifically. The most commonly recommended book is Bostrom's Superintelligence, but it's fairly dry. Tegmark's Life 3.0 is on the other extreme of too pop-sci like. I think Yudkowsky's writing is the best for explaining the issue (specifically AGI goal alignment and what the specific problem is) in a way that's accessible, but not dumbed down.
- coderenegade 5y ago
  The human experience is fundamentally chemical in nature (which is why alcohol has an impact on cognition, for example). What would it even mean to replicate that in silicon? I know food is bad because I can hold it, smell it, and taste it, none of which a computer can do. If you tried to upload someone's consciousness to a machine, you would need to emulate the full sensory experience as well, because there would be zero context otherwise. What's missing from discussions on AGI is the lack of a measure that relates human intelligence with machine intelligence, and the recognition that our own intelligence is fundamentally tied in to our physical bodies. To what extent does it make sense to talk about intelligence without also talking about its embodiment?
  - gonehome 5y ago
    Inputs can be replicated in other forms (cameras for example for visual input and computer vision). There’s a common misconception that AGI would be human like, consciousness is a mostly poorly defined orthogonal rabbit hole to the AGI question (and the goal alignment issue). That said, you’re right the ability to train is dependent on inputs. If you’re curious about this stuff it’s really worth reading the stuff I mentioned to get a sense for what the problem is. I only suggest them because I would have made a similar comment to you a few years ago before getting a better sense for what the problems are actually describing.
    - coderenegade 5y ago
      >A simple thought experiment is the same architecture of the human brain (whatever that is), but just with way more power to run at more operations per second. That's your quote, is it not? I'm pointing out that it's not just a matter of replicating an architecture, because the architecture only really makes sense in the context of everything else in the human body. There's zero reason to expect that we could replicate the architecture of a human brain (whatever that means) and just plug in a few cameras to have a working model. This also implies that it would be exceedingly difficult to make human-like AGI, because doing so would be tantamount to making an artificial human. Anything that we do make will almost certainly not be human-like, and will probably only make sense in the context of its sensory system and the inputs it receives. This goes beyond training, because inference happens on data drawn from the same distribution that you trained on. Note that I haven't said anything about consciousness here, because that's an unrelated issue. I've read bits and pieces of Bostrom, and I would argue that it's "not even wrong", in the sense that, yes, building a child god would be a disaster for humanity, but we don't even know if building such a god is possible, because it's totally undefined and we have no way of measuring it. That's the problem. Without a definition of intelligence as a physical process that maps humans and AI to the same measure on a like-for-like basis, it's essentially an argument on whether or not Thor is stronger than Superman. We have no context for what the limitations are likely to be; it may be that building an angry god just isn't possible.
- breuleux 5y ago
  Silicon-based machines do need cooling, and arguably an AGI-grade processor would be 3D, which is an even greater cooling challenge. Speaking of silicon, I'm curious why it is virtually absent of organic chemistry despite its sheer abundance on Earth. Is it because it's too hard to extract it from its oxides? Or is it just completely outclassed by carbon? Depending on the answer, it is quite possible that future AI will eschew silicon altogether and run on organic chemistry.

eru 5y ago

> Would that actually work, though, or would it just cook itself? You could add better cooling than what the human hardware provides. Even just taking a normal human head and dunking it in cold water dissipates a lot more head. (If you try this at home, I suggest getting a snorkel.)

gonehome 5y ago

Why would it cook itself if it was replicated in silicon? I think it's a lot less likely humanity just happens to be at some natural limit of intelligence rather than being bounded by other unrelated constraints. AGI tradeoffs would be different because it'd be unconstrained from biological natural selection. If you're interested in this stuff it's worth checking out the work that specifies the AGI goal alignment problem more specifically. The most commonly recommended book is Bostrom's Superintelligence, but it's fairly dry. Tegmark's Life 3.0 is on the other extreme of too pop-sci like. I think Yudkowsky's writing is the best for explaining the issue (specifically AGI goal alignment and what the specific problem is) in a way that's accessible, but not dumbed down.

coderenegade 5y ago
The human experience is fundamentally chemical in nature (which is why alcohol has an impact on cognition, for example). What would it even mean to replicate that in silicon? I know food is bad because I can hold it, smell it, and taste it, none of which a computer can do. If you tried to upload someone's consciousness to a machine, you would need to emulate the full sensory experience as well, because there would be zero context otherwise. What's missing from discussions on AGI is the lack of a measure that relates human intelligence with machine intelligence, and the recognition that our own intelligence is fundamentally tied in to our physical bodies. To what extent does it make sense to talk about intelligence without also talking about its embodiment?
- gonehome 5y ago
  Inputs can be replicated in other forms (cameras for example for visual input and computer vision). There’s a common misconception that AGI would be human like, consciousness is a mostly poorly defined orthogonal rabbit hole to the AGI question (and the goal alignment issue). That said, you’re right the ability to train is dependent on inputs. If you’re curious about this stuff it’s really worth reading the stuff I mentioned to get a sense for what the problem is. I only suggest them because I would have made a similar comment to you a few years ago before getting a better sense for what the problems are actually describing.
  - coderenegade 5y ago
    >A simple thought experiment is the same architecture of the human brain (whatever that is), but just with way more power to run at more operations per second. That's your quote, is it not? I'm pointing out that it's not just a matter of replicating an architecture, because the architecture only really makes sense in the context of everything else in the human body. There's zero reason to expect that we could replicate the architecture of a human brain (whatever that means) and just plug in a few cameras to have a working model. This also implies that it would be exceedingly difficult to make human-like AGI, because doing so would be tantamount to making an artificial human. Anything that we do make will almost certainly not be human-like, and will probably only make sense in the context of its sensory system and the inputs it receives. This goes beyond training, because inference happens on data drawn from the same distribution that you trained on. Note that I haven't said anything about consciousness here, because that's an unrelated issue. I've read bits and pieces of Bostrom, and I would argue that it's "not even wrong", in the sense that, yes, building a child god would be a disaster for humanity, but we don't even know if building such a god is possible, because it's totally undefined and we have no way of measuring it. That's the problem. Without a definition of intelligence as a physical process that maps humans and AI to the same measure on a like-for-like basis, it's essentially an argument on whether or not Thor is stronger than Superman. We have no context for what the limitations are likely to be; it may be that building an angry god just isn't possible.
breuleux 5y ago
Silicon-based machines do need cooling, and arguably an AGI-grade processor would be 3D, which is an even greater cooling challenge. Speaking of silicon, I'm curious why it is virtually absent of organic chemistry despite its sheer abundance on Earth. Is it because it's too hard to extract it from its oxides? Or is it just completely outclassed by carbon? Depending on the answer, it is quite possible that future AI will eschew silicon altogether and run on organic chemistry.