3 ms·
I see that they do some decontamination of the datasets, in the hope that the models won't just recite answers from the training data. But in the recent intervi
by ziofill 2y ago
I see that they do some decontamination of the datasets, in the hope that the models won't just recite answers from the training data. But in the recent interview with Subbarao Kambhampati on MLST (https://www.youtube.com/watch?v=y1WnHpedi2A https://www.youtube.com/watch?v=y1WnHpedi2A) they explain that models fail as soon as one slightly rephrases the test problems (indicating that they are indeed mostly reciting). I expect this to be the case with this model too.
- eightysixfour 2y agoThis is just not true. There are plenty of private tests with problems that are not in the training set. GPQA is an excellent example.
- refulgentis 2y agoThere's a lot of bunko 3rd-time-rephrased "wisdom" about AI, and enough interest in it that if you have the right title, you can get away with repeating it. I'm a bit surprised that an HN reader, who presumably has first hand experience with them, isn't sure if its just a hashmap lookup.