I believe they are referring to the original observations of code output degradation due to adding training epochs for safety. These observations were made by M

I believe they are referring to the original observations of code output degradation due to adding training epochs for safety. These observations were made by Microsoft working directly with OpenAI early on. They comment on it in the technical lecture for GPT-4 in the famous 'Tikz Unicorn' example, wherein the model got better and better at making the unicorn during typical training, but then regressed when training after that for better safety. It is unclear why this may be, but to speculate, it may be due to classifying large regions of the distribution as 'off limits' and therefore less structure about these regions is modeled in detail (since it is now not needed, and summed up as 'as a LLM, I cannot'). It is certainly strange that making a model less deplorable will make it worse at coding in general, but it does appear to be a real observation.