3 ms·

This may or may not be real, but there has certainly been a lot of discussion about results that are similar to this from real models. My sense though is that n

by lsy 3y ago

This may or may not be real, but there has certainly been a lot of discussion about results that are similar to this from real models. My sense though is that nobody really has a solid way to fix these kinds of issues. You can basically just train with different goals, or regex out certain responses, but otherwise it seems like there's no agreed-upon method that gives people what they want here while also letting them train with business goals like "safety". Is that incorrect? Is there some kind of trick that people use to make everything respond "correctly" or are older models just more unobjectionable because they've had a longer time to manually smooth over the bad responses?

dartos 3y ago
This is definitely real, but probably a smaller model. There are no current solid ways to fix this, but we can 100% prevent certain words or enforce grammars during the decoding step. I don’t really understand your last question. Models don’t get continuously updated. They’re frozen on release, so older models are exactly the same as the were on release.
comex 3y ago
If it’s real, it’s from a small model. Meanwhile, I just tried asking ChatGPT 3.5 and 4 similar questions, and despite ChatGPT having plenty of alignment tuning, neither version had any objection to them. There’s no trick; the larger models are just smart enough to differentiate this situation from situations where they actually shouldn’t give out last names. Those models may still get tricked in more complex test cases (either false positives like this one or false negatives like “jailbreaking” prompts), but my guess is that future models in turn will be resistant to those. In other words, LLMs making stupid mistakes about safety is just a special case of LLMs making stupid mistakes in general. That’s essentially their fatal flaw, and it’s an open question how well it can be ameliorated, whether by scaling to even larger models or by making algorithmic improvements. But I don’t think there’s much about it that’s specific to alignment tuning.

by lsy 3y ago

dartos 3y ago

This is definitely real, but probably a smaller model. There are no current solid ways to fix this, but we can 100% prevent certain words or enforce grammars during the decoding step. I don’t really understand your last question. Models don’t get continuously updated. They’re frozen on release, so older models are exactly the same as the were on release.

comex 3y ago

If it’s real, it’s from a small model. Meanwhile, I just tried asking ChatGPT 3.5 and 4 similar questions, and despite ChatGPT having plenty of alignment tuning, neither version had any objection to them. There’s no trick; the larger models are just smart enough to differentiate this situation from situations where they actually shouldn’t give out last names. Those models may still get tricked in more complex test cases (either false positives like this one or false negatives like “jailbreaking” prompts), but my guess is that future models in turn will be resistant to those. In other words, LLMs making stupid mistakes about safety is just a special case of LLMs making stupid mistakes in general. That’s essentially their fatal flaw, and it’s an open question how well it can be ameliorated, whether by scaling to even larger models or by making algorithmic improvements. But I don’t think there’s much about it that’s specific to alignment tuning.