2 ms·
It seems intuitively that that should work to mitigate some prompt injection, but even if it would, it’s quite possible that a good chunk of the power of moder
by dragonwriter 3y ago
It seems intuitively that that should work to mitigate some prompt injection, but even if it would, it’s quite possible that a good chunk of the power of modern models comes from not making that distinction, and the model in effect determining what kind of weight to give data by context. We might make much safer but also far less useful models by doing that.
It also isn’t likely to eliminate all bad data –> undesirable actions within the scope of what is enabled by the tools the model has access to problems. Because as long as the intent is for behavior to depend on data, you can manipulate behavior by manipulating data in unexpected ways.