5 ms·
some interesting stuff from one of the authors. could only solve a third of MIT undergrad problems but did well on other tests. https://twitter.com/alewkowycz/
by alphabetting 4y ago
some interesting stuff from one of the authors. could only solve a third of MIT undergrad problems but did well on other tests.
https://twitter.com/alewkowycz/status/1542559200051593219 https://twitter.com/alewkowycz/status/1542559200051593219
- riku_iki 4y agoLooks like Polish tests were included into training dataset.
- guyga 4y agoFrom the paper, the Polish tests are from May 2022, and the training dataset was from January 2022, so this is unlikely.
- riku_iki 4y agoUnless new tests consist of many questions similar to questions from the previous years. I think this is huge issue with many current benchmarks. They need to create some handcrafted secret questions and never disclose them like it has been done with SquAD benchmark.
- rafaelero 4y agoStudents have the same advantage. What's your point?
- riku_iki 4y agoFirst point is that LLM not necessary "solve", but potentially "memorize", while students can still be able to solve questions they didn't see. Second point is proposal how benchmarks can be improved.
- lumost 4y agoIt’s worth recalling that a structurally similar problem presented somewhere else on the internet could enter the training set. A non-trivial portion of problems likely just have variable names replaced.