2 ms·

Close. We might create a very powerful system which we can't determine is fully aligned with our values, but can determine is at least loosely aligned with som

Close. We might create a very powerful system which we can't determine is fully aligned with our values, but can determine is at least loosely aligned with something we care about. Then we might direct that system to do something loosely aligned with our values (e.g. maximize paperclips), at which point it does something against our values as well. And it's not hypothetical. In RL, you do see models exploit simulation errors to maximize reward in a way we didn't intend and would have disallowed if we'd thought of it. Likewise across automation, we get what we built, not what we intended. We have solid historical precedent to expect to not get quite what we want, and the tech seems to be on the precipice of being very powerful, so I'd call it a very straightforward fear for us to worry about making something very powerful that isn't quite what we want.

bamboozled 4y ago
Do you not think there isn't still a different between something which can become sentient, somehow start operating outside of the computer it lives in and on and on ? I mean it would have to be a pretty big leap from and algorithm optimized to do "something" to an algorithm which understands how to operate interdependently and then escape it's box and then start to do the wrong thing, and to add to that, be smart enough to stop us from turning it off before doing too much damage. Actually curious what people think about this?
- 6gvONxR4sf7o 4y ago
  I think the whole idea of sentience is a red herring for reasoning about the potential for AI to cause harm. For example, take the title of the article we're talking about. You could take it with a dose of anthropomorphization and interpret it with human-like intent and sentience, or you could take it as an analogy that we might formally define closer to "not worried about AI that passes Turing test, but AI that fails for some reason other than the ability to succeed (and that other reason requires the evaluator to be misled)." So is sentience far away? No idea. Is misaligned AI a near-term worry? Yeah, absolutely. And "misaligned" doesn't have to be as extreme as 'makes humans extinct.' It can be as simple as 'is an asshole' or 'inexplicably swerves into the other lane 0.01% of the time' or something else we don't want in the system.
  - bamboozled 4y ago
    Good points, I almost feel like we have this unusual "need" to build an asshole AI to find out what it's like to have an asshole AI in our lives...kind of like how people don't want to ski into a tree, but for some reason develop target fixation, then ski into...the tree.

bamboozled 4y ago

Do you not think there isn't still a different between something which can become sentient, somehow start operating outside of the computer it lives in and on and on ? I mean it would have to be a pretty big leap from and algorithm optimized to do "something" to an algorithm which understands how to operate interdependently and then escape it's box and then start to do the wrong thing, and to add to that, be smart enough to stop us from turning it off before doing too much damage. Actually curious what people think about this?

6gvONxR4sf7o 4y ago
I think the whole idea of sentience is a red herring for reasoning about the potential for AI to cause harm. For example, take the title of the article we're talking about. You could take it with a dose of anthropomorphization and interpret it with human-like intent and sentience, or you could take it as an analogy that we might formally define closer to "not worried about AI that passes Turing test, but AI that fails for some reason other than the ability to succeed (and that other reason requires the evaluator to be misled)." So is sentience far away? No idea. Is misaligned AI a near-term worry? Yeah, absolutely. And "misaligned" doesn't have to be as extreme as 'makes humans extinct.' It can be as simple as 'is an asshole' or 'inexplicably swerves into the other lane 0.01% of the time' or something else we don't want in the system.
- bamboozled 4y ago
  Good points, I almost feel like we have this unusual "need" to build an asshole AI to find out what it's like to have an asshole AI in our lives...kind of like how people don't want to ski into a tree, but for some reason develop target fixation, then ski into...the tree.