5 ms·

I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability

I don't even think we need alignment, just containment. The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.

TeMPOraL 3y ago
Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
- RandomLensman 3y ago
  I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
  - JoshTriplett 3y ago
    > Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem. > The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths. (Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)
    - RandomLensman 3y ago
      Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?
      - JoshTriplett 3y ago
        > Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. Those trade off against each other and don't all have to be as easy as possible. Information sufficiently dangerous to destroy the world certainly exists, the question is how close AI gets to the boundary of "possible to summarize from existing literature and/or generate" and "possible for human to apply", given in particular that the AI can model and evaluate "possible for human to apply". > Similar, it is unclear if humans could not contain superhuman intelligence. If you agree that it's not clearly and obviously possible, then we're already most of the way to "what is the risk that it isn't possible to contain, what is the amount of danger posed if it isn't possible, what amount of that risk is acceptable, and should we perhaps have any way at all to limit that risk if we decide the answer isn't 'all of it as fast as we possibly can'". The difference between "90% likely" and "20% likely" and "1% likely" and "0.01% likely" is really not relevant at all when the other factor being multiplied in is "existential risk to humanity". That number needs a lot more zeroes. It's perfectly reasonable for people to disagree whether the number is 90% or 1%; if you think people calling it extremely likely are wrong, fine. What's ridiculous is when people either try to claim (without evidence) that it's 0 or effectively 0, or when people claim it's 1% but act as if that's somehow acceptable risk, or act like anyone should be able to take that risk for all of humanity.
        RandomLensman 3y ago
  - ben_w 3y ago
    First, how is the Khmer Rouge an example of containment, given that regime fell? Second, even if your argument is "genocide of anyone who sounds too smart was the right approach and they just weren't trying hard enough", that only really fits into "neither alignment nor containment, just don't have the AGI at all". Containment, for humans, would look like a prison from which there is no escape; but if this is supposed to represent an AI that you want to use to solve problems, this prison with no escape needs a high-bandwidth internet connection with the outside world… and somehow zero opportunity for anyone outside to become convinced they need to rescue the "people"[0] inside like last year: https://en.wikipedia.org/wiki/LaMDA#Sentience_claims https://en.wikipedia.org/wiki/LaMDA#Sentience_claims [0] or AI who are good at pretending to be people, distinction without a difference in this case
    - RandomLensman 3y ago
      It means intelligence can be killed (and the Khmer were brought down externally). We might not even need to contain, all hypothetical.
      - pixl97 3y ago
        We don't have an answer yet to what the form factor of AGI/ASI will be, but if it's anything like current trends the idea of 'killed' is laughable. You can be killed because your storage medium and execution medium are inseparable. Destroy the brain and you're gone, and you don't even get to copy yourself. With AGI/ASI if we can boot it up from any copy on a disk if we have the right hardware then at the end of the day you've effectively created the undead as long as a drive exists with a copy of it and a computer exists than can run it.
        RandomLensman 3y ago
        No power and it its already dead. Destroy copies, dead. Really not complicated at all.
        pixl97 3y ago

TeMPOraL 3y ago

Since we're in Yudkowsky subthread, I'll remind here that he spent some 20 odd years explaining why containment won't work, period. You can't both contain and put to work something that's smarter than you. Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent. Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.

RandomLensman 3y ago
I find his arguments unconvincing. Humans can and have contained human intelligence, for example (look at the Khmer Rouge, for example). Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
- JoshTriplett 3y ago
  > Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem. > The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths. (Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)
  - RandomLensman 3y ago
    Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?
    - JoshTriplett 3y ago
      > Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. Those trade off against each other and don't all have to be as easy as possible. Information sufficiently dangerous to destroy the world certainly exists, the question is how close AI gets to the boundary of "possible to summarize from existing literature and/or generate" and "possible for human to apply", given in particular that the AI can model and evaluate "possible for human to apply". > Similar, it is unclear if humans could not contain superhuman intelligence. If you agree that it's not clearly and obviously possible, then we're already most of the way to "what is the risk that it isn't possible to contain, what is the amount of danger posed if it isn't possible, what amount of that risk is acceptable, and should we perhaps have any way at all to limit that risk if we decide the answer isn't 'all of it as fast as we possibly can'". The difference between "90% likely" and "20% likely" and "1% likely" and "0.01% likely" is really not relevant at all when the other factor being multiplied in is "existential risk to humanity". That number needs a lot more zeroes. It's perfectly reasonable for people to disagree whether the number is 90% or 1%; if you think people calling it extremely likely are wrong, fine. What's ridiculous is when people either try to claim (without evidence) that it's 0 or effectively 0, or when people claim it's 1% but act as if that's somehow acceptable risk, or act like anyone should be able to take that risk for all of humanity.
      - RandomLensman 3y ago
- ben_w 3y ago
  First, how is the Khmer Rouge an example of containment, given that regime fell? Second, even if your argument is "genocide of anyone who sounds too smart was the right approach and they just weren't trying hard enough", that only really fits into "neither alignment nor containment, just don't have the AGI at all". Containment, for humans, would look like a prison from which there is no escape; but if this is supposed to represent an AI that you want to use to solve problems, this prison with no escape needs a high-bandwidth internet connection with the outside world… and somehow zero opportunity for anyone outside to become convinced they need to rescue the "people"[0] inside like last year: https://en.wikipedia.org/wiki/LaMDA#Sentience_claims https://en.wikipedia.org/wiki/LaMDA#Sentience_claims [0] or AI who are good at pretending to be people, distinction without a difference in this case
  - RandomLensman 3y ago
    It means intelligence can be killed (and the Khmer were brought down externally). We might not even need to contain, all hypothetical.
    - pixl97 3y ago
      We don't have an answer yet to what the form factor of AGI/ASI will be, but if it's anything like current trends the idea of 'killed' is laughable. You can be killed because your storage medium and execution medium are inseparable. Destroy the brain and you're gone, and you don't even get to copy yourself. With AGI/ASI if we can boot it up from any copy on a disk if we have the right hardware then at the end of the day you've effectively created the undead as long as a drive exists with a copy of it and a computer exists than can run it.
      - RandomLensman 3y ago
        No power and it its already dead. Destroy copies, dead. Really not complicated at all.
        pixl97 3y ago