4 ms·

I think the core is not to agree with the existential risk scenario in the first place. Once you believe some malevolent god will appear and doom us all, anyth

I think the core is not to agree with the existential risk scenario in the first place. Once you believe some malevolent god will appear and doom us all, anything can be justified (and people have done so in the past - no malevolent god appeared then, btw.).

concordDance 3y ago
What's your view on where the X risk idea breaks down? Is it that it's impossible to make something smarter than a human? Or that such a thing wouldn't have goals/plans that require resources us humans want/need? Or that such a thing wouldn't be particularly dangerously powerful?
- RandomLensman 3y ago
  It could break down on any of those, for example (with the first one not forever, but could take much longer than what is mainstream thinking now). A lot of existential risk involves bold hypotheses about machines being able to "solve" a lot of things humans cannot, but we don't know how much is actually just solvable by superior intelligence vs. what just doesn't work in that way. Human collective intelligence failed on a lot of things so far. Even the basic idea of exponential scaling intelligence is a hypothesis which might not hold true. Also, some existential risk ideas involve hypotheses around evolution and on how species dominate - again, might not be right.
  - concordDance 3y ago
    "It could" implies to me that there's a chance they don't. What would you say the odds are on each of those three components? And are those odds independent?
    - RandomLensman 3y ago
      I meant could also that there are other things that could also make the risk not materialise. I'd think they are quite independent. Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.
      - concordDance 3y ago
        To me it seems that the odds of us making something(s) "smarter" (better planning and modelling) than a human in the next 50 years is a coin toss. The odds of us being able to ensure its desires/goals align with what we would like are probably around 30%. And the odds of a "smarter than human" things ending up conquering or killing the humans if they desire to do so being around 90%. This naturally gives a probably of Doom at least 1/3rd. I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.
        RandomLensman 3y ago
        I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
        TeMPOraL 3y ago
        Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
        RandomLensman 3y ago
        I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
  - breuleux 3y ago
    I would argue that intelligence, at the core, is about understanding and manipulating simple systems in order to achieve specific goals. This becomes exponentially difficult with complexity (or level of chaos), so past a certain threshold, systems become essentially unintelligible. A large part of our effectiveness is in strategically simplifying our environment so that it is amenable to intelligent manipulation. Most of the big problems we have are in situations where we need to preserve complex aspects of reality that make the systems too chaotic to properly predict, so I suspect AI won't be able to do much better regardless of how smart it is. The ability to carry out destructive brute force actions worries me more than intelligence.
PoignardAzur 3y ago
> I think the core is not to agree with the existential risk scenario in the first place. I mean, that's motivated reasoning right there, right? "Agreeing that existential risk is there would lead to complex intractable problems we don't want to tackle, so let's agree there's no existential risk". This isn't a novel idea, it's why we started climate change mitigations 20 years too late.
- RandomLensman 3y ago
  No, I meant it as in: there is no reason to agree with that scenario. Stacking a lot of hypotheses on top of each other as to get to something dangerous isn't necessarily something that is convincing.
  - hackinthebochs 3y ago
    There's also no reason to agree that AI will be aligned and everything will be fine. The question is what should our default stance be until proven otherwise? I submit it is not to continue building the potentially world-ending technology. When you see a gun on the table, what do you do? You assume its loaded until proven otherwise. For some reason, those who imagine AI will usher in some tech-utopia not only assume the gun is empty, but that pulling the trigger will bring forth endless prosperity. It's rather insane actually.
    - Chathamization 3y ago
      Whenever I see people jump to alignment they invariably have jumped over the _much_more questionable assumption that AGI’s will be godlike. This doesn’t even match what we observe in reality - drop a human into the middle of a jungle, and it doesn’t simply become a god just because it has an intelligence that’s orders of magnitude greater than the animals around it. In fact, most people wouldn’t even survive. Further, our success as a species doesn’t come from lone geniuses, but from large organizations that are able to harness the capabilities of thousands/millions of individual intelligences. Assuming an AGI that’s better than an individual human is going to automatically be better than millions of humans - and so much better that it’s godlike - is disconnected from what we see in reality. It actually seems to be a reflection of the LessWrong crowd, who (in my experience) greatly overemphasize the role of lone geniuses and end up struggling when it comes social aspects of our society.

concordDance 3y ago

What's your view on where the X risk idea breaks down? Is it that it's impossible to make something smarter than a human? Or that such a thing wouldn't have goals/plans that require resources us humans want/need? Or that such a thing wouldn't be particularly dangerously powerful?

RandomLensman 3y ago
It could break down on any of those, for example (with the first one not forever, but could take much longer than what is mainstream thinking now). A lot of existential risk involves bold hypotheses about machines being able to "solve" a lot of things humans cannot, but we don't know how much is actually just solvable by superior intelligence vs. what just doesn't work in that way. Human collective intelligence failed on a lot of things so far. Even the basic idea of exponential scaling intelligence is a hypothesis which might not hold true. Also, some existential risk ideas involve hypotheses around evolution and on how species dominate - again, might not be right.
- concordDance 3y ago
  "It could" implies to me that there's a chance they don't. What would you say the odds are on each of those three components? And are those odds independent?
  - RandomLensman 3y ago
    I meant could also that there are other things that could also make the risk not materialise. I'd think they are quite independent. Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.
    - concordDance 3y ago
      To me it seems that the odds of us making something(s) "smarter" (better planning and modelling) than a human in the next 50 years is a coin toss. The odds of us being able to ensure its desires/goals align with what we would like are probably around 30%. And the odds of a "smarter than human" things ending up conquering or killing the humans if they desire to do so being around 90%. This naturally gives a probably of Doom at least 1/3rd. I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.
      - RandomLensman 3y ago
        I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
        TeMPOraL 3y ago
        Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
        RandomLensman 3y ago
        I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
- breuleux 3y ago
  I would argue that intelligence, at the core, is about understanding and manipulating simple systems in order to achieve specific goals. This becomes exponentially difficult with complexity (or level of chaos), so past a certain threshold, systems become essentially unintelligible. A large part of our effectiveness is in strategically simplifying our environment so that it is amenable to intelligent manipulation. Most of the big problems we have are in situations where we need to preserve complex aspects of reality that make the systems too chaotic to properly predict, so I suspect AI won't be able to do much better regardless of how smart it is. The ability to carry out destructive brute force actions worries me more than intelligence.

PoignardAzur 3y ago

> I think the core is not to agree with the existential risk scenario in the first place. I mean, that's motivated reasoning right there, right? "Agreeing that existential risk is there would lead to complex intractable problems we don't want to tackle, so let's agree there's no existential risk". This isn't a novel idea, it's why we started climate change mitigations 20 years too late.

RandomLensman 3y ago
No, I meant it as in: there is no reason to agree with that scenario. Stacking a lot of hypotheses on top of each other as to get to something dangerous isn't necessarily something that is convincing.
- hackinthebochs 3y ago
  There's also no reason to agree that AI will be aligned and everything will be fine. The question is what should our default stance be until proven otherwise? I submit it is not to continue building the potentially world-ending technology. When you see a gun on the table, what do you do? You assume its loaded until proven otherwise. For some reason, those who imagine AI will usher in some tech-utopia not only assume the gun is empty, but that pulling the trigger will bring forth endless prosperity. It's rather insane actually.
  - Chathamization 3y ago
    Whenever I see people jump to alignment they invariably have jumped over the _much_more questionable assumption that AGI’s will be godlike. This doesn’t even match what we observe in reality - drop a human into the middle of a jungle, and it doesn’t simply become a god just because it has an intelligence that’s orders of magnitude greater than the animals around it. In fact, most people wouldn’t even survive. Further, our success as a species doesn’t come from lone geniuses, but from large organizations that are able to harness the capabilities of thousands/millions of individual intelligences. Assuming an AGI that’s better than an individual human is going to automatically be better than millions of humans - and so much better that it’s godlike - is disconnected from what we see in reality. It actually seems to be a reflection of the LessWrong crowd, who (in my experience) greatly overemphasize the role of lone geniuses and end up struggling when it comes social aspects of our society.