5 ms·

I meant could also that there are other things that could also make the risk not materialise. I'd think they are quite independent. Not sure I have any odds o

I meant could also that there are other things that could also make the risk not materialise. I'd think they are quite independent. Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.

concordDance 3y ago
To me it seems that the odds of us making something(s) "smarter" (better planning and modelling) than a human in the next 50 years is a coin toss. The odds of us being able to ensure its desires/goals align with what we would like are probably around 30%. And the odds of a "smarter than human" things ending up conquering or killing the humans if they desire to do so being around 90%. This naturally gives a probably of Doom at least 1/3rd. I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.
- RandomLensman 3y ago
  I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
  - TeMPOraL 3y ago
    Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
    - RandomLensman 3y ago
      I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
      - JoshTriplett 3y ago
        > Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem. > The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths. (Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)
        RandomLensman 3y ago
        Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?
- ImPostingOnHN 3y ago
  I would put the numbers much lower, but if your bar for legislation is ">1% risk of damage", we've got a lot of climate change legislation that's decades overdue to pass first, and success or failure there will prove whether we can do something similar with AI.
  - JoshTriplett 3y ago
    > we've got a lot of climate change legislation that's decades overdue to pass first We can do more than one thing at once, and we need to.
    - ImPostingOnHN 3y ago
      can we, though? if we can't even do one thing at once given decades of trying (deal with climate change), we definitely can't do that one thing plus another thing (deal with AI)

concordDance 3y ago

To me it seems that the odds of us making something(s) "smarter" (better planning and modelling) than a human in the next 50 years is a coin toss. The odds of us being able to ensure its desires/goals align with what we would like are probably around 30%. And the odds of a "smarter than human" things ending up conquering or killing the humans if they desire to do so being around 90%. This naturally gives a probably of Doom at least 1/3rd. I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.

RandomLensman 3y ago
I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
- TeMPOraL 3y ago
  Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
  - RandomLensman 3y ago
    I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
    - JoshTriplett 3y ago
      > Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem. > The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths. (Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)
      - RandomLensman 3y ago
        Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?
ImPostingOnHN 3y ago
I would put the numbers much lower, but if your bar for legislation is ">1% risk of damage", we've got a lot of climate change legislation that's decades overdue to pass first, and success or failure there will prove whether we can do something similar with AI.
- JoshTriplett 3y ago
  > we've got a lot of climate change legislation that's decades overdue to pass first We can do more than one thing at once, and we need to.
  - ImPostingOnHN 3y ago
    can we, though? if we can't even do one thing at once given decades of trying (deal with climate change), we definitely can't do that one thing plus another thing (deal with AI)