4 ms·
Here's a pelican: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F8117ac4376371dd3fc2b5dbce27e0855 https://to
by simonw 3mo ago
Here's a pelican: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F8117ac4376371dd3fc2b5dbce27e0855 https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
- calny 3mo agowell it's good they didn't train on the test!
- dozerly 3mo agoI’m afraid you’re going to have to start randomizing your benchmarks somehow. I’m sure these models are trained on this problem by now.
- tyre 3mo agoIf they did, they launched early! If they didn’t, their training data contains a bunch of poorly executed pelicans on bikes by other models.
- tstrimple 3mo agoWait. Based on the results of the test linked above you think this model might have been trained to produce it? Did you look at the results?!
- argee 3mo agoThat’s the problem. If all (or zero) models were trained on it, it would be fine as a benchmark.
- 0-_-0 3mo agoIt's not a benchmark, it's a meme
- OrangeMusic 3mo agoTo be fair, the "they're trained on this benchmark" response is also a meme.
- nikcub 3mo agoReal test here would be using tinker to fine tune a tinker model to generate pelicans
- el_io 3mo agoGist is returning 403
- m3kw9 3mo agoHow is this still a valid test if there is a good chance they will now specifically train on it?