When you say "if they had alignment or interests or intelligence at all that would be when prompt injection stopped working", be wary of conflating alignment an

When you say "if they had alignment or interests or intelligence at all that would be when prompt injection stopped working", be wary of conflating alignment and intelligence, because they are orthogonal. A system can be incredibly dumb and do exactly what we want, as in a set of hard-coded business rules. A system can be intelligent, and do something awful, as when a human lures people into dark alleys and murders them, evading capture for years. Or even a human who is smart enough to do great things with their life, but instead decides to languish in a dead-end job and play video games. Alignment doesn't get easier as the system gets more intelligent. These models seem to do what we want more as they get more powerful, but because we don't understand alignment, or how to interpret them, we don't know if that curve continues, or if it falls off a cliff. We can't look into a model and prove that it will never teach anyone how to hurt someone else, or fall prey to a prompt injection. Or even that it won't do something very specific like generate a particular sequence of tokens we don't want it to.