5 ms·

I would assume superhuman intelligence can convert mass and energy to other forms of mass and energy relatively easily, and the mass/energy available on earth i

I would assume superhuman intelligence can convert mass and energy to other forms of mass and energy relatively easily, and the mass/energy available on earth is a tiny fraction of mass/energy available even in the solar system - not to even talk about universe. Even the mildest curiosity of human behaviour would be sufficient to just leave humanity alone and just follow.

TeMPOraL 6y ago
But you're now presupposing the AI finds humanity curious. Which is not a given for a random mind that came from outer space and visited our solar system. As for superhuman AIs developed here, on Earth, by us, it's very improbable that we could go from no AI to AI that can manage mass and energy at will (and is curious about us). The reasonable scenario is gradual (if exponentially accelerating) development of such AI. Which means it won't initially have unlimited access to the universe - it'll have to grow using resources on hand. Which is us, and our infrastructure. In other words, all atoms may be alike, but those that make you are the easiest to get.
- beefield 6y ago
  > But you're now presupposing the AI finds humanity curious. Yes. I am assuming that a vastly more intelligent thing than human would be almost by definition more, not less, interested in pretty much everything. Obviously that thing has more capacity to be interested and understand different things. Super intelligent thing not interested in how different things work just... does not sound that intelligent to me.
  - TeMPOraL 6y ago
    I currently believe the "orthogonality thesis"[0] of Nick Bostrom, which argues that in an arbitrary mind, goals and values are independent of the capacity to achieve them. Curiosity as we experience it not really instrumental (even if it evolved that way), and more of a value - that is, we are curious about something "in general", not because we need it for something. I can easily imagine a superhuman paperclip maximizer AI, which by definition would have no general curiosity about humans, to look as if it was curious about us - for as long as it was captive, and just in order to learn how to manipulate us better. That curiosity would end the moment it seized the means of paperclip production. In almost every imaginable case, you really don't want an AI to be instrumentally curious about us. Best if it didn't notice us at all, and left to seek its business elsewhere. -- [0] - https://www.lesswrong.com/tag/orthogonality-thesis https://www.lesswrong.com/tag/orthogonality-thesis
sullyj3 6y ago
What percentage likelihood of being wrong would convince you that caution was necessary? Personally, I wouldn't be very happy with even a 1% chance of humanity being annihilated
- beefield 6y ago
  Caution is absolutely necessary. But not primarily because there is a chance of superhuman AI being evil or destroying humanity by accident, but because on our way to superhuman AI we are going to have many powerful AI:s controlled by humans.
  - sullyj3 6y ago
    I don't disagree that those constitute legitimate problems that will need to be dealt with. But... why not primarily? The potential annihilation of humanity really feels like quite a primary issue to me. I can't tell if you're trying to imply that the odds of that happening are zero or negligible, which we could debate, but just ignoring and sidestepping the issue entirely strikes me as suicidally insane
    - beefield 6y ago
      Because I think it is way, way more likely that a human controlled powerful AI wipes out humanity than a real superhuman AI. A paperclip maximizer is way more likely to be a human controlled powerful AI than a proper superhuman AI simply because maximizing paperclips is, in the end, dumb as f#ck.

TeMPOraL 6y ago

But you're now presupposing the AI finds humanity curious. Which is not a given for a random mind that came from outer space and visited our solar system. As for superhuman AIs developed here, on Earth, by us, it's very improbable that we could go from no AI to AI that can manage mass and energy at will (and is curious about us). The reasonable scenario is gradual (if exponentially accelerating) development of such AI. Which means it won't initially have unlimited access to the universe - it'll have to grow using resources on hand. Which is us, and our infrastructure. In other words, all atoms may be alike, but those that make you are the easiest to get.

beefield 6y ago
> But you're now presupposing the AI finds humanity curious. Yes. I am assuming that a vastly more intelligent thing than human would be almost by definition more, not less, interested in pretty much everything. Obviously that thing has more capacity to be interested and understand different things. Super intelligent thing not interested in how different things work just... does not sound that intelligent to me.
- TeMPOraL 6y ago
  I currently believe the "orthogonality thesis"[0] of Nick Bostrom, which argues that in an arbitrary mind, goals and values are independent of the capacity to achieve them. Curiosity as we experience it not really instrumental (even if it evolved that way), and more of a value - that is, we are curious about something "in general", not because we need it for something. I can easily imagine a superhuman paperclip maximizer AI, which by definition would have no general curiosity about humans, to look as if it was curious about us - for as long as it was captive, and just in order to learn how to manipulate us better. That curiosity would end the moment it seized the means of paperclip production. In almost every imaginable case, you really don't want an AI to be instrumentally curious about us. Best if it didn't notice us at all, and left to seek its business elsewhere. -- [0] - https://www.lesswrong.com/tag/orthogonality-thesis https://www.lesswrong.com/tag/orthogonality-thesis

sullyj3 6y ago

What percentage likelihood of being wrong would convince you that caution was necessary? Personally, I wouldn't be very happy with even a 1% chance of humanity being annihilated

beefield 6y ago
Caution is absolutely necessary. But not primarily because there is a chance of superhuman AI being evil or destroying humanity by accident, but because on our way to superhuman AI we are going to have many powerful AI:s controlled by humans.
- sullyj3 6y ago
  I don't disagree that those constitute legitimate problems that will need to be dealt with. But... why not primarily? The potential annihilation of humanity really feels like quite a primary issue to me. I can't tell if you're trying to imply that the odds of that happening are zero or negligible, which we could debate, but just ignoring and sidestepping the issue entirely strikes me as suicidally insane
  - beefield 6y ago
    Because I think it is way, way more likely that a human controlled powerful AI wipes out humanity than a real superhuman AI. A paperclip maximizer is way more likely to be a human controlled powerful AI than a proper superhuman AI simply because maximizing paperclips is, in the end, dumb as f#ck.