Sam Altman calls it the "alignment tax". There were some papers on ArXiV showing that as GPT-4 was aligned, it got measurably stupider. For example, the test w

Sam Altman calls it the "alignment tax". There were some papers on ArXiV showing that as GPT-4 was aligned, it got measurably stupider. For example, the test where it was asked to draw a pink unicorn using a the TikZ vector art format: during training it got better and better, and then worse and worse again as it was aligned. A friend of mine put it well: biases are just our Bayesian priors of reality. If you remove biases, then you're erasing some of those priors. Bad priors lead to bad predictions.