3 ms·

This is not a problem with AGI or alignment or anything like that, it's a problem with someone else sneaking inputs into your machine using tricks like white fo

This is not a problem with AGI or alignment or anything like that, it's a problem with someone else sneaking inputs into your machine using tricks like white font colours and javascript and stuff like that. It's not about self driving cars it's about other people grabbing the steering wheel. These models literally only respond to the text they're inputted, and if they had alignment or interests or intelligence at all that would be when prompt injection stopped working.

NumberWangMan 4y ago
When you say "if they had alignment or interests or intelligence at all that would be when prompt injection stopped working", be wary of conflating alignment and intelligence, because they are orthogonal. A system can be incredibly dumb and do exactly what we want, as in a set of hard-coded business rules. A system can be intelligent, and do something awful, as when a human lures people into dark alleys and murders them, evading capture for years. Or even a human who is smart enough to do great things with their life, but instead decides to languish in a dead-end job and play video games. Alignment doesn't get easier as the system gets more intelligent. These models seem to do what we want more as they get more powerful, but because we don't understand alignment, or how to interpret them, we don't know if that curve continues, or if it falls off a cliff. We can't look into a model and prove that it will never teach anyone how to hurt someone else, or fall prey to a prompt injection. Or even that it won't do something very specific like generate a particular sequence of tokens we don't want it to.
johnfn 4y ago
> These models literally only respond to the text they're inputted, and if they had alignment or interests or intelligence at all that would be when prompt injection stopped working. Let's take this as true (although I'm not sure it is!). Even if it is, you still must agree that AI will provide bad agents with a staggering amount of new attack vectors. Even if AI itself won't ever do a single bad thing, how do we stop Joe Schmoe's random malignant thought from being amplified ten-thousand-fold?
- ChatGTP 4y ago
  Some ideas: * take computers off people * perform surveillance on everyone * turn off the internet as we know it * watch some horrifying shit go down Any combination of those, not necessarily in that order. I’m not saying I want this to happen, but we cannot peacefully live in a world that can just be fucked over by anyone at any second, that would be stupid ? Like we can’t let people live with the means to create biological weapons at home either.

NumberWangMan 4y ago

When you say "if they had alignment or interests or intelligence at all that would be when prompt injection stopped working", be wary of conflating alignment and intelligence, because they are orthogonal. A system can be incredibly dumb and do exactly what we want, as in a set of hard-coded business rules. A system can be intelligent, and do something awful, as when a human lures people into dark alleys and murders them, evading capture for years. Or even a human who is smart enough to do great things with their life, but instead decides to languish in a dead-end job and play video games. Alignment doesn't get easier as the system gets more intelligent. These models seem to do what we want more as they get more powerful, but because we don't understand alignment, or how to interpret them, we don't know if that curve continues, or if it falls off a cliff. We can't look into a model and prove that it will never teach anyone how to hurt someone else, or fall prey to a prompt injection. Or even that it won't do something very specific like generate a particular sequence of tokens we don't want it to.

johnfn 4y ago

> These models literally only respond to the text they're inputted, and if they had alignment or interests or intelligence at all that would be when prompt injection stopped working. Let's take this as true (although I'm not sure it is!). Even if it is, you still must agree that AI will provide bad agents with a staggering amount of new attack vectors. Even if AI itself won't ever do a single bad thing, how do we stop Joe Schmoe's random malignant thought from being amplified ten-thousand-fold?

ChatGTP 4y ago
Some ideas: * take computers off people * perform surveillance on everyone * turn off the internet as we know it * watch some horrifying shit go down Any combination of those, not necessarily in that order. I’m not saying I want this to happen, but we cannot peacefully live in a world that can just be fucked over by anyone at any second, that would be stupid ? Like we can’t let people live with the means to create biological weapons at home either.