That's not true yet. The MoE of GPT-4 is not in open models is it? OpenAI has done extensive RLHF. If you compare the code generation ability of GPT-4 with Fal

That's not true yet. The MoE of GPT-4 is not in open models is it? OpenAI has done extensive RLHF. If you compare the code generation ability of GPT-4 with Falcon 140B, it is clear that all of their efforts are still paying off. There is still a gap in efficiency and performance. I mean, the gap has definitely narrowed though. And I think it's true that they will have trouble recovering all of the billions. But at least some of it has been paid back already.