4 ms·
How do you even put together an open evaluation benchmark without it being slurped up into the training set the minute someone puts it on Github? Maybe it need
by tveita 3y ago
How do you even put together an open evaluation benchmark without it being slurped up into the training set the minute someone puts it on Github?
Maybe it needs to be obfuscated, or all the questions should be obscene so they don't pass the content filtering.
- gwern 3y agoThere are 'canary tokens' that you can add to your dataset which are supposed to be respected by self-regulating scrapers & responsible ML practitioners to grep them out before doing any training. How well it works is unknown, of course...