3 ms·
One explaination https://x.com/yampeleg/status/1707127722743325106?s=46&t=CxaRJxotO6kj3op6jNibrw https://x.com/yampeleg/status/1707127722743325106?s=46&t=Cxa...
by mufasachan 3y ago
One explaination https://x.com/yampeleg/status/1707127722743325106?s=46&t=CxaRJxotO6kj3op6jNibrw https://x.com/yampeleg/status/1707127722743325106?s=46&t=Cxa...
I would be curious what does he mean by "semi-automated system for detecting benchmark leaks. " though.
- brucethemoose2 3y agoAFAIK, such tests just feed the model chopped up bits of the evaluation data as raw strings with zero temperature. If it completes them verbatim, its probably in the training dataset.