5 ms·

It could break down on any of those, for example (with the first one not forever, but could take much longer than what is mainstream thinking now). A lot of ex

It could break down on any of those, for example (with the first one not forever, but could take much longer than what is mainstream thinking now). A lot of existential risk involves bold hypotheses about machines being able to "solve" a lot of things humans cannot, but we don't know how much is actually just solvable by superior intelligence vs. what just doesn't work in that way. Human collective intelligence failed on a lot of things so far. Even the basic idea of exponential scaling intelligence is a hypothesis which might not hold true. Also, some existential risk ideas involve hypotheses around evolution and on how species dominate - again, might not be right.

concordDance 3y ago
"It could" implies to me that there's a chance they don't. What would you say the odds are on each of those three components? And are those odds independent?
- RandomLensman 3y ago
  I meant could also that there are other things that could also make the risk not materialise. I'd think they are quite independent. Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.
  - concordDance 3y ago
    To me it seems that the odds of us making something(s) "smarter" (better planning and modelling) than a human in the next 50 years is a coin toss. The odds of us being able to ensure its desires/goals align with what we would like are probably around 30%. And the odds of a "smarter than human" things ending up conquering or killing the humans if they desire to do so being around 90%. This naturally gives a probably of Doom at least 1/3rd. I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.
    - RandomLensman 3y ago
      I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
      - TeMPOraL 3y ago
        Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
        RandomLensman 3y ago
        I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
        JoshTriplett 3y ago
        > Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem. > The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths. (Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)
breuleux 3y ago
I would argue that intelligence, at the core, is about understanding and manipulating simple systems in order to achieve specific goals. This becomes exponentially difficult with complexity (or level of chaos), so past a certain threshold, systems become essentially unintelligible. A large part of our effectiveness is in strategically simplifying our environment so that it is amenable to intelligent manipulation. Most of the big problems we have are in situations where we need to preserve complex aspects of reality that make the systems too chaotic to properly predict, so I suspect AI won't be able to do much better regardless of how smart it is. The ability to carry out destructive brute force actions worries me more than intelligence.

concordDance 3y ago

"It could" implies to me that there's a chance they don't. What would you say the odds are on each of those three components? And are those odds independent?

RandomLensman 3y ago
I meant could also that there are other things that could also make the risk not materialise. I'd think they are quite independent. Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.
- concordDance 3y ago
  To me it seems that the odds of us making something(s) "smarter" (better planning and modelling) than a human in the next 50 years is a coin toss. The odds of us being able to ensure its desires/goals align with what we would like are probably around 30%. And the odds of a "smarter than human" things ending up conquering or killing the humans if they desire to do so being around 90%. This naturally gives a probably of Doom at least 1/3rd. I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.
  - RandomLensman 3y ago
    I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
    - TeMPOraL 3y ago
      Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
      - RandomLensman 3y ago
        I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
        JoshTriplett 3y ago
        > Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem. > The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths. (Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)

breuleux 3y ago

I would argue that intelligence, at the core, is about understanding and manipulating simple systems in order to achieve specific goals. This becomes exponentially difficult with complexity (or level of chaos), so past a certain threshold, systems become essentially unintelligible. A large part of our effectiveness is in strategically simplifying our environment so that it is amenable to intelligent manipulation. Most of the big problems we have are in situations where we need to preserve complex aspects of reality that make the systems too chaotic to properly predict, so I suspect AI won't be able to do much better regardless of how smart it is. The ability to carry out destructive brute force actions worries me more than intelligence.