3 ms·
This ^ I am suspicious of contamination in every finetune I see, and very suspicious in a new foundational model like this. (For those reading and not followi
by brucethemoose2 3y ago
This ^
I am suspicious of contamination in every finetune I see, and very suspicious in a new foundational model like this.
(For those reading and not following, "contamination" is training a model/finetune on the very test it will be tested on. Normally these known tests are specifically excluded from training datasets so the models can be properly evaluated, but throwing them in is an easy way to "cheat" and claim a model is better than it is.
In a foundational model with a huge dataset, there's also a high probability that well-known evaluation questions snuck into the dataset by accident).
- Workaccount2 3y agoWe need an AI that can generate bespoke AI tests...
- brucethemoose2 3y agoThe existing models are technically bad candidates for this, as they will inevitably produce tests they can more effectively answer themselves.
- londons_explore 3y agoTheres also a good chance that simply people discussing the test questions and answers on forums like reddit sneaks into the dataset, even if the exact question is filtered out.
- brucethemoose2 3y agoYeah, this is exactly what I was thinking. Its supposed to be searched for and filtered out in the training datasets, but I can even see an active effort to weed them out missing some test Q/A pairs.