4 ms·
Especially if you consider that the more one writes online about the CodeContests benchmark, the more training data including the CodeContests benchmark data is
by aprilthird2021 2y ago
Especially if you consider that the more one writes online about the CodeContests benchmark, the more training data including the CodeContests benchmark data is fed to the next iteration of GPT. We could be doing massive scale over-fitting without realizing it.
- falcor84 2y agoI haven't seen any evidence of over-fitting in the sense of the model getting worse on new examples. Conversely, it seems that we're gradually fitting the models on more and more types of coding questions, such that it's gradually becoming more and more difficult to find well-defined problems that humans can solve easily but AIs cannot.