4 ms·
It's probably fuzzily fixable by including instruction authority levels in the training data. Can't expect much more than that, given that the model itself is f
by xpct 2mo ago
It's probably fuzzily fixable by including instruction authority levels in the training data. Can't expect much more than that, given that the model itself is fuzzy.
- QuercusMax 2mo agoWe've already seen that it's possible to trick models into seeing user input as their own "thinking" if you make it sound like what the model writes. While it may appear that it's looking at the tags on the input, in practice that's not as strong a guarantee as you'd hope.
- altcognito 2mo agoDo you have the reference for this, I remember seeing it recently but can't dig it up
- aw1621107 2mo agoI think GP is referring to "Prompt Injection as Role Confusion" (https://role-confusion.github.io/ https://role-confusion.github.io/). It was discussed on HN several weeks ago (https://news.ycombinator.com/item?id=48631888 https://news.ycombinator.com/item?id=48631888)
- QuercusMax 2mo agoThat's the one!
- altcognito 2mo agoYep, thanks
- jonnycomputer 2mo agoMight be this: https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-mechanistic-explanation-of-prompt-injection-and-why-you https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-mechanis...
- tp3358 2mo agoMaybe this is an elementary angle given my lack of security experience, but couldn't Microsoft figure out a wat to parse the documents prior to model analysis/action? Implement some form of deterministic layer that resides between the user and the model?
- raggi 2mo agoParse for what? The model has “arbitrary understanding” of “arbitrary input”. The filter is unbounded and the only actually safe result is to filter everything.
- hulitu 2mo ago> but couldn't Microsoft figure out a wat to parse the documents prior to model analysis/action If Apple and others "couldn't", why would Microsoft ? Throwing errors for bad input is so 90's.
- HlessClaudesman 2mo agoWhat would non fuzzy AI look like? Seems to me the fuzzyness is where all the fun is.
- veganmosfet 2mo agoI think this is already implemented and called message roles: system, user, tool and assistant (ordered by authority). Untrusted data shall only be injected in tool responses, which are less authoritative. But at the end, this all ends in the big token stream - so this is not a perfect separation.