3 ms·
How do we know these are real rules or were just made up on the spot of what it thinks the rules should be for some hypothetic AI engine?
by rdtsc 3y ago
How do we know these are real rules or were just made up on the spot of what it thinks the rules should be for some hypothetic AI engine?
- simonw 3y agoI wrote about that here: https://news.ycombinator.com/item?id=35925239 https://news.ycombinator.com/item?id=35925239
- wll 3y agoAgreed. These are instruction-tuned: they will follow the instructions, so much so that not even the strongest RLHF can currently prevent well-structured jailbreaking. In my experience their attention is strongest towards the end of the last message, which could be the reason for injections being so effective with little effort. Within the OpenAI models as of today the first user message is much stronger than the system message. Given the ChatML spec and their end-to-end control over the models, I wonder whether the system message could end up being sandboxed by architecture and/or training.