5 ms·

People have already studied this. You make it sound like an open question, but the answer is right there in Bostrom's Superintelligence, or any works on cogniti

People have already studied this. You make it sound like an open question, but the answer is right there in Bostrom's Superintelligence, or any works on cognitive heuristics & biases, the physics of computing, or the mathematics of decision making. The answer is "no. we are nowhere near the smartest creatures possible". And there are multiple independent lines of argument and evidence that point directly to this conclusion.

mindcrime 11y ago
The answer is "no. we are nowhere near the smartest creatures possible". Right, like I said, this whole thing was just "thinking out loud" more than any thoroughly researched and validated idea. And I didn't do a very good job of even explaining what I was getting at. It's not so much that the question is "are we the smartest creatures possible". It's more like: "as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?" See also skybrian's comment which actually states it better than I did originally: https://news.ycombinator.com/item?id=10755109 https://news.ycombinator.com/item?id=10755109
- nshepperd 11y ago
  > "as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?" The answer is still yes. Your "thinking out loud" is privileging an already refuted hypothesis.
  - mindcrime 11y ago
    The answer is still yes. Well that's a fine assertion, but how do you justify it? Your "thinking out loud" is privileging an already refuted hypothesis. WTH does that even mean? There is no hypothesis, there's a vague notion of an area to consider, which - if considered thoroughly - might or might not yield a hypothesis.
    - nshepperd 11y ago
      As I said, Nick Bostrom's Superintelligence talks all about this. The main thing is that the constraints of the physical world are nowhere near the limitations of human capabilities. Yes, there's a theoretical limit to how much computation you can do in a certain amount of space with a certain amount of energy. No biology or technology currently in existence even gets close to those limits. Without any such fundamental limit there's just no reason for AGI not to present a threat. But there is lots of other evidence too.
      - mindcrime 11y ago
        I've been planning to read Superintelligence for a while anyway, so I think I'll move it up the list a bit in response to all this discussion. I'm on vacation once I check out of here today, until the second week of Jan, so I'll probably read it over the holiday.
- lmm 11y ago
  The thing is it's "just thinking out loud" on the same level as "what if the particles aren't really in a superposition and we just don't know which state they're in". These are basic questions, the answers are known, published, and the article even mentions exactly where you can find them.
  - mindcrime 11y ago
    When I say I'm "thinking out loud" what I mean is, the exact words I used may not reflect the underlying point I was getting at, because it was fuzzy in my head when I first started thinking about it. Reading all of these responses, it's clear that most people are responding to something different than the issue I really meant to raise. Fair enough, that's my fault for not being clearer. But that's the value in a discussion, so this whole exercise has been productive (for me at least). These are basic questions, the answers are known, published, and the article even mentions exactly where you can find them. I've read and re-read TFA and I don't find that it addresses the issue I'm thinking about. It's not so much asking "are we the smartest possible creature", or even asking if we're close to that. It's also not about asking whether or not it's possible for a super-AGI to be smarter than humans. The issue I was trying to raise is more of "given how smart humans are (whatever that mean) and given whatever the limit is for how smart a hypothetical super-AGI can be, does the analogy between a super-AGI and a nuclear bomb hold? That is, does a super-AGI really represent an existential threat?" And again, I'm not taking a side on this either way. I honestly haven't spent enough time thinking about it. I will say this though... what I've read on the topic so far (and I haven't yet gotten to Bostrom's book, to be fair) doesn't convince me that this is a settled question. Maybe after I finish Superintelligence I'll feel differently though. I have it on my shelf waiting to be read anyway, so maybe I'll bump it up the priority list a bit and read it over the holiday.

mindcrime 11y ago

The answer is "no. we are nowhere near the smartest creatures possible". Right, like I said, this whole thing was just "thinking out loud" more than any thoroughly researched and validated idea. And I didn't do a very good job of even explaining what I was getting at. It's not so much that the question is "are we the smartest creatures possible". It's more like: "as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?" See also skybrian's comment which actually states it better than I did originally: https://news.ycombinator.com/item?id=10755109 https://news.ycombinator.com/item?id=10755109

nshepperd 11y ago
> "as smart as we are, and given the constraints of the physical world, is there room for a hypothetical 'smarter than human AGI' to represent an existential threat, or something that justifies the analogy to nuclear weapons?" The answer is still yes. Your "thinking out loud" is privileging an already refuted hypothesis.
- mindcrime 11y ago
  The answer is still yes. Well that's a fine assertion, but how do you justify it? Your "thinking out loud" is privileging an already refuted hypothesis. WTH does that even mean? There is no hypothesis, there's a vague notion of an area to consider, which - if considered thoroughly - might or might not yield a hypothesis.
  - nshepperd 11y ago
    As I said, Nick Bostrom's Superintelligence talks all about this. The main thing is that the constraints of the physical world are nowhere near the limitations of human capabilities. Yes, there's a theoretical limit to how much computation you can do in a certain amount of space with a certain amount of energy. No biology or technology currently in existence even gets close to those limits. Without any such fundamental limit there's just no reason for AGI not to present a threat. But there is lots of other evidence too.
    - mindcrime 11y ago
      I've been planning to read Superintelligence for a while anyway, so I think I'll move it up the list a bit in response to all this discussion. I'm on vacation once I check out of here today, until the second week of Jan, so I'll probably read it over the holiday.
lmm 11y ago
The thing is it's "just thinking out loud" on the same level as "what if the particles aren't really in a superposition and we just don't know which state they're in". These are basic questions, the answers are known, published, and the article even mentions exactly where you can find them.
- mindcrime 11y ago
  When I say I'm "thinking out loud" what I mean is, the exact words I used may not reflect the underlying point I was getting at, because it was fuzzy in my head when I first started thinking about it. Reading all of these responses, it's clear that most people are responding to something different than the issue I really meant to raise. Fair enough, that's my fault for not being clearer. But that's the value in a discussion, so this whole exercise has been productive (for me at least). These are basic questions, the answers are known, published, and the article even mentions exactly where you can find them. I've read and re-read TFA and I don't find that it addresses the issue I'm thinking about. It's not so much asking "are we the smartest possible creature", or even asking if we're close to that. It's also not about asking whether or not it's possible for a super-AGI to be smarter than humans. The issue I was trying to raise is more of "given how smart humans are (whatever that mean) and given whatever the limit is for how smart a hypothetical super-AGI can be, does the analogy between a super-AGI and a nuclear bomb hold? That is, does a super-AGI really represent an existential threat?" And again, I'm not taking a side on this either way. I honestly haven't spent enough time thinking about it. I will say this though... what I've read on the topic so far (and I haven't yet gotten to Bostrom's book, to be fair) doesn't convince me that this is a settled question. Maybe after I finish Superintelligence I'll feel differently though. I have it on my shelf waiting to be read anyway, so maybe I'll bump it up the priority list a bit and read it over the holiday.