I think the main issue is that since we don't have access to the implementation of GPT-4, there's no way for researchers to know how it works, so either they're

I think the main issue is that since we don't have access to the implementation of GPT-4, there's no way for researchers to know how it works, so either they're not saying anything or they're just making things up. GPT-4 is "multimodal" and RLHF'd, so it was trained with some tasks other than next word prediction. I don't remember if it's been trained for code correctness (by running unit tests etc.), but other models have been.