4 ms·
It tops benchmarks because it uses them in its training data. https://x.com/eliebakouch/status/2077425801633427919 https://x.com/eliebakouch/status/207742580163
by jbellis 3mo ago
It tops benchmarks because it uses them in its training data. https://x.com/eliebakouch/status/2077425801633427919 https://x.com/eliebakouch/status/2077425801633427919
- sva_ 3mo agohttps://nitter.net/eliebakouch/status/2077425801633427919 https://nitter.net/eliebakouch/status/2077425801633427919 That is very disappointing.
- Tostino 3mo agoThat's just embarrassing. It keeps happening though.
- sajithdilshan 3mo agoDisappointing. Sounds like Volkswagen 2.0
- deleted 3mo ago[deleted]
- krackers 3mo agotraining on the test set is all you need
- ch4s3 3mo agoIt's ready for law school!
- AtlasBarfed 3mo agoscrape the scrapers... all's fair...
- fxwin 3mo agolooks like they have since fixed the dataset, hope they release an updated report (either without the leaked evals or a new checkpoint/weights) https://huggingface.co/datasets/AIML-TUDA/QA-base/commit/2f96f4ba8f4576934ac8b066841c05b8f7f4b044 https://huggingface.co/datasets/AIML-TUDA/QA-base/commit/2f9...
- istvan0 3mo agoReminds me of: https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal
- amai 3mo agoIt seems they corrected this already: https://nitter.net/jphme/status/2077468512025477226#m https://nitter.net/jphme/status/2077468512025477226#m
- jbellis 3mo agoI guess you didn't read ellie's reply directly underneath that?