I would say much of it is data. I believe at this point, you can feed it the whole world wide web and it would still be a little smaller than what GPT-4 is trai

I would say much of it is data. I believe at this point, you can feed it the whole world wide web and it would still be a little smaller than what GPT-4 is trained on. I'd speculate OpenAI hires a lot of people to make new data for them. There's stories of people who were interviewed where the job is just solving questions to be fed to GPT-4. They're likely looking at things that were thumbed down and adding several other data points to handle those. There's also the quality of this data. You could feed it data from private convos (which Meta probably does) but most of that data would be low quality. We're finding that Claude, GPT, Llama, even DeepL consistently mistranslate "you" in Indonesian as "Anda". It's never capitalized in the real world and it's a safe word, the equivalent of "they/them" pronouns in English. Imagine how weird it would be if someone wrote "Alice walks Their dog every afternoon." It's likely they were trained on government docs or perhaps some kind of ads. But the point is there's lots of holes in the data and the perception of quality is how often it doesn't fail these. None of the benchmarks catch this bug either. It's not just Indonesian - most of the code benchmarks use Python. So we'll also likely need better benchmarks in the future.