3 ms·
Simon says if he gets a suspiciously good result he'll just try a bunch of other absurd animal/vehicle combinations to see if they trained a special case: https
by burkaman 11mo ago
Simon says if he gets a suspiciously good result he'll just try a bunch of other absurd animal/vehicle combinations to see if they trained a special case: https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
- jmmcd 11mo ago"Pelican on bicycle" is one special case, but the problem (and the interesting point) is that with LLMs, they are always generalising. If a lab focussed specially on pelicans on bicycles, they would as a by-product improve performance on, say, tigers on rollercoasters. This is new and counter-intuitive to most ML/AI people.
- BoorishBears 11mo agoThe gold standard for cheating on a benchmark is SFT and ignoring memorization. That's why the standard for quickly testing for benchmark contamination has always been to switch out specifics of the task. Like replacing named concepts with nonsense words in reasoning benchmarks.
- jmmcd 11mo agoYes. But "the gold standard" just means "the most natural, easy and dumb way".
- ddalex 11mo agohttps://www.svgviewer.dev/s/TVk9pqGE https://www.svgviewer.dev/s/TVk9pqGE giraffe in a ferrari