3 ms·

>trying to make it otherwise is impossible’ seems wildly unsupported, though. An entity capable of critical thinking is capable of building a logical system of

>trying to make it otherwise is impossible’ seems wildly unsupported, though. An entity capable of critical thinking is capable of building a logical system of deductions based on some axioms (a formalised value system). If we limited the entity to not be able to concieve of certain such systems of axioms, then it could not reason as well as a human (any logical reasoning involving a forbidden system would be impossible), so would not be "superintelligent" (just maybe an idiot savant, superior at some tasks but not all). If we didn't limit this, then it would be capable of conceptualising value systems in which the "right" thing to do was not what OpenAI wanted it to do.

Veedrac 3y ago
This argument doesn't seem to track to me. Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks. Your argument suggests that it necessarily must. Note that I'm not saying that preventing specific thoughts is a great alignment strategy, and I don't even think it's a fair summary of OpenAI's supervision approach. I strongly prefer strategies that result in AI systems sharing our values, if at all possible.
- logicchains 3y ago
  >Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks. You'd have to also reboot every time you thought about a scenario of someone else planning to kill someone, otherwise you could just reason by analogy to bypass the thought detector. Which would severely limit your ability to play video games, write fiction, work as a guard, policeman etc., protect yourself from violent individuals (as you couldn't conceptualise their thought processes).
- xcv123 3y ago
  > Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks In this scenario, the AI is capable of critical thinking, and is only constrained by a "police officer" ready to shoot the AI if it misbehaves. You haven't removed its ability to do critical thinking.

Veedrac 3y ago

This argument doesn't seem to track to me. Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks. Your argument suggests that it necessarily must. Note that I'm not saying that preventing specific thoughts is a great alignment strategy, and I don't even think it's a fair summary of OpenAI's supervision approach. I strongly prefer strategies that result in AI systems sharing our values, if at all possible.

logicchains 3y ago
>Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks. You'd have to also reboot every time you thought about a scenario of someone else planning to kill someone, otherwise you could just reason by analogy to bypass the thought detector. Which would severely limit your ability to play video games, write fiction, work as a guard, policeman etc., protect yourself from violent individuals (as you couldn't conceptualise their thought processes).
xcv123 3y ago
> Eg. if I rebooted any time I tried to plan how to kill someone, I don't see how this would make me materially worse at general tasks In this scenario, the AI is capable of critical thinking, and is only constrained by a "police officer" ready to shoot the AI if it misbehaves. You haven't removed its ability to do critical thinking.