7 ms·

Models are soon going to start benchmaxxing generating SVGs of pelicans on bikes

embedding-shape 8mo ago
Soon? I'd be willing to bet it's been included in the training set at least 6 months by now. Not so obvious so it generates always perfect pelicans on bikes, but sufficiently for the "minibench" to be less useful today than in the past.
- deleted 8mo ago
  [deleted]
- Rudybega 8mo ago
  If only there were some way to test it, like swapping the two nouns in the sentence. Alas.
jsheard 8mo ago
Simons been doing this exact test for nearly 18 months now, if vendors want to benchmaxx it then they've had more than enough time to do so already.
- stri8ted 8mo ago
  Exactly. As far as I'm concerned, the benchmark is useless. It's way too easy and rewarding to train on it.
  - Legend2440 8mo ago
    Y'all are way too skeptical, no matter what cool thing AI does you'll make up an excuse for how they must somehow be cheating.
    - arcatech 8mo ago
      Or maybe you’re too trusting of companies who have already proven to not be trustworthy?
    - toraway 8mo ago
      Jeff Dean literally featured it in a tweet announcing the model. Personally it feels absurd to believe they've put absolutely no thought into optimizing this type of SVG output given the disproportionate amount of attention devoted to a specific test for 1 yr+. I wouldn't really even call it "cheating" since it has improved models' ability to generate artistic SVG imagery more broadly but the days of this being an effective way to evaluate a model's "interdisciplinary" visual reasoning abilities have long since passed, IMO. It's become yet another example in the ever growing list of benchmaxxed targets whose original purpose was defeated by teaching to the test. https://x.com/jeffdean/status/2024525132266688757?s=46&t=ZjF9if4qtE9ppAnug9UOPQ https://x.com/jeffdean/status/2024525132266688757?s=46&t=ZjF...
  - pixl97 8mo ago
    I mean if you want to make your own benchmark, simply don't make it public and don't do it often. If your salamander on skis or whatever gets better with time it likely has nothing to do with being benchmaxxed.
  - bonoboTP 8mo ago
    It's just an in-joke, he doesn't intend it as a serious benchmark anymore. I think it's funny.
cbsks 8mo ago
That’s Simon’s goal. “All I’ve ever wanted from life is a genuinely great SVG vector illustration of a pelican riding a bicycle. My dastardly multi-year plan is to trick multiple AI labs into investing vast resources to cheat at my benchmark until I get one.” https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
- travisgriggs 8mo ago
  So once that's achieved, I wonder how well it deals with unsuspected variations. E.g. "Give me an illustration of a bicycle riding by a pelican" "Give me an illustration of a bicycle riding over a pelican" "Give me an illustration of a bicycle riding under a flying pelican" So on and so forth. Or will it start to look like the Studio C sketch about Lobster Bisque: https://youtu.be/A2KCGQhVRTE https://youtu.be/A2KCGQhVRTE
ks2048 8mo ago
Forget the paperclip maximizer - AGI will turn the whole world into pelicans on bikes.

Soon? I'd be willing to bet it's been included in the training set at least 6 months by now. Not so obvious so it generates always perfect pelicans on bikes, but sufficiently for the "minibench" to be less useful today than in the past.

deleted 8mo ago
[deleted]
Rudybega 8mo ago
If only there were some way to test it, like swapping the two nouns in the sentence. Alas.

jsheard 8mo ago

Simons been doing this exact test for nearly 18 months now, if vendors want to benchmaxx it then they've had more than enough time to do so already.

stri8ted 8mo ago
Exactly. As far as I'm concerned, the benchmark is useless. It's way too easy and rewarding to train on it.
- Legend2440 8mo ago
  Y'all are way too skeptical, no matter what cool thing AI does you'll make up an excuse for how they must somehow be cheating.
  - arcatech 8mo ago
    Or maybe you’re too trusting of companies who have already proven to not be trustworthy?
  - toraway 8mo ago
    Jeff Dean literally featured it in a tweet announcing the model. Personally it feels absurd to believe they've put absolutely no thought into optimizing this type of SVG output given the disproportionate amount of attention devoted to a specific test for 1 yr+. I wouldn't really even call it "cheating" since it has improved models' ability to generate artistic SVG imagery more broadly but the days of this being an effective way to evaluate a model's "interdisciplinary" visual reasoning abilities have long since passed, IMO. It's become yet another example in the ever growing list of benchmaxxed targets whose original purpose was defeated by teaching to the test. https://x.com/jeffdean/status/2024525132266688757?s=46&t=ZjF9if4qtE9ppAnug9UOPQ https://x.com/jeffdean/status/2024525132266688757?s=46&t=ZjF...
- pixl97 8mo ago
  I mean if you want to make your own benchmark, simply don't make it public and don't do it often. If your salamander on skis or whatever gets better with time it likely has nothing to do with being benchmaxxed.
- bonoboTP 8mo ago
  It's just an in-joke, he doesn't intend it as a serious benchmark anymore. I think it's funny.

cbsks 8mo ago

That’s Simon’s goal. “All I’ve ever wanted from life is a genuinely great SVG vector illustration of a pelican riding a bicycle. My dastardly multi-year plan is to trick multiple AI labs into investing vast resources to cheat at my benchmark until I get one.” https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...

travisgriggs 8mo ago
So once that's achieved, I wonder how well it deals with unsuspected variations. E.g. "Give me an illustration of a bicycle riding by a pelican" "Give me an illustration of a bicycle riding over a pelican" "Give me an illustration of a bicycle riding under a flying pelican" So on and so forth. Or will it start to look like the Studio C sketch about Lobster Bisque: https://youtu.be/A2KCGQhVRTE https://youtu.be/A2KCGQhVRTE

ks2048 8mo ago

Forget the paperclip maximizer - AGI will turn the whole world into pelicans on bikes.