2 ms·
I wonder if this is due to abliteration actually "damaging" the model, or just an artifact of the model never having been properly trained on "forbidden" topics
by lxgr 7mo ago
I wonder if this is due to abliteration actually "damaging" the model, or just an artifact of the model never having been properly trained on "forbidden" topics (as it's enough for them to recognize them, and there's no point in dedicating neurons to something that will never be exercised anyway).
- zozbot234 7mo agoModern abliteration is quite good at not damaging the model on ordinary topics. But yes, on many of the weirdest "forbidden" topics (excluding the mild stuff like ordinary erotica) there's not going to be any real training of any sort and it's basically hallucinations running wild. You even see this claim repeated explicitly on every model release "safety card": 'no, this model does not have the sort of fiddly tacit know-how it would need to actually advise anyone nefarious on this dangerous stuff'.
- saagarjha 7mo agoI’m not an expert and this is just a guess from the behavior but I feel like it’s kind of like giving the model a kind of Williams Syndrome, where (AFAIK) a common training process is that you keep asking it sensitive topics and every time it rejects the request you train it away from that and towards “Yeah, absolutely! Let’s go!!!” I feel like the natural result of this is that you’ll get something that is excited about whatever ideas you have?