62 ms·
This is fantastic I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demon
by simonw 3mo ago
This is fantastic
I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.
Catching a lab cheating specifically on my one dumb benchmark would be really funny.
Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.
His conclusion:
> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.
- zkmon 3mo agoWhy is it hard to come up with tests that closely resemble real-world usecases where the models are being used?
- philipwhiuk 3mo agoWould it be any more applicable? We see the same problem in database benchmarking. The good thing about the deliberately-non-real world pelican case is that it gives a general impression of how much the model is improving because it's not likely that it's being specifically targeted at it, rather than a 'real world case' which might have been specifically optimised for.
- simonw 3mo agoIt's not that hard - there are plenty of those. They're mostly not very funny though.
- gilleain 3mo agoPerhaps also vary the bird? Wikipedia tells me pelicans are in the order _Pelecaniformes_ so shoebills or herons might do.
- mattertoast 3mo ago[dead]
- lukev 3mo agoWhat if they’re not pelicanmaxxing, but svgmaxxxing in general? Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge. Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
- charcircuit 3mo agoI agree, other formats, both textual and binary should be tested.
- netsec_burn 3mo agoAddressed in the article, in case you're curious.
- lukev 3mo agoWell, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.) That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)
- Dylan16807 3mo agoAddressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.
- qq66 3mo agoBut that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.
- ryukoposting 3mo agoHow useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good. AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.
- gopalv 3mo ago> Catching a lab cheating specifically on my one dumb benchmark would be really funny. Similar thing happened when TPC came up with SQL benchmarks. If you're not good at TPC, your engineering team is no good. If you're good at TPC, then (as a customer) we will actually include you in a bake-off benchmark for our specific problem. Winning on it is the price of admittance into the game, especially in a crowded market. But how narrowly you benchmarket matters, you can't just hard-code that specific scenario & not fix anything adjacent while you're at it. For example when it comes to GPUs, the "Quack3" (sic) benchmark on ATI cards comes to mind.
- ryandrake 3mo agoYea, I worked for a competitor to ATI back in the day and we were definitely Quakemaxxing. We were not so unethical as to try to detect the .EXE name so to put the GPU into a "cheating" mode only when that benchmark was run, but we did run Quake 3 pretty much constantly while trying to eke out a few more FPS...
- exhaze 3mo agoWasn’t that almost the norm for both new OS drivers and subsequently GPU SoC firmware of ‘gamer cards’ to literally optimize for the latest AAA titles? I feel in that case, the interests aligned - in a world where I had a chance to actually play Crysis, I wouldn’t care why the frame rate was decent, no?
- docheinestages 3mo agoI think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.
- zahlman 3mo agoFidelity to the original is definitely not how humans would measure "art" in this context.
- docheinestages 3mo agoTrue, maybe we can call the generated SVG something else than art.
- cyberax 3mo ago> I've been casually spot-checking other animals in other vehicles Snakes on a plane, weasels on a diesel, spiders on a glider, baboons on a balloon, goats on a boat.
- eob 3mo agoSimon I hope from this day hence, your bio always includes: "Simon Willison, among other things, is an advocate for the inclusion of pelican geometry in LLM training datasets."
- exhaze 3mo ago“Today, the NASDAQ fell by 18% moments after Simon Willison published his latest blog evaluating the Legend 7.4 model that rendered an animated SVG of a pelican on a bicycle and the pelican fell off”
- shawabawa3 3mo agoWhen asked for comment an anthropic spokesperson said "it's very unusual for the pelican to fall off. They are built to very rigorous SVG engineering standards"
- exhaze 3mo agoWhat if they’re just unsuccessful at it and something about a model grokking how to create a realistic visual representation of a pelican on a bicycle ends being key to the next OOM of capability unhobbling. And since we have established this silly routine once, I must keep going and ask - yes, you have quite a collection of bona fide pelicans you’ve seen and photographed. Have you physically seen all the other animals you’ve evaluated as well? I was born in Soviet Russia, so trust but verify and if still around, maybe there pelican make LLM draw Simon ride bicycle and notice Python code improve.
- JadoJodo 3mo agoIt's a shame you started with pelicans on a bicycle and not snakes on a plane.
- usef- 3mo agoIt's better to be original
- packageman 3mo agoI worked with image analytics years ago. In 2024 and cynical about LLM image capability, I tested different models to create "Two female octopi in a bar each having their own drink. One is wearing a yellow hat. The other is wearing a red hat. They have no human features." It went badly I think because the training visual training data associated with the word "female" was overwhelmingly biased in quantity and obviated the 'no human features' instruction even after iterated instructions to remove. So I tried "Make an image of a Djibouti cab with a camel sitting in the passenger seat. Give the camel no human anatomical features." Still bad. Two years later the graphics and perspectives of the tools are so much better. But that isn't the big story. What really stands out is the models are no longer adding human female anatomies to camels and octopi. The gates and filtering based on instructions have matured enough so that the pelican/bicycle deductive reasoning puzzle is less problematic. But until an LLM anticipates something like impressionism from Parisian artists rebelling against the rules of the French Academy, continue to reserve a place for human artistry. tl;dr simonw, your spot-check nailed it just as well as any extensive methodology. LLM's no longer need to cheat this part of the test (better to hack the question than the tool).
- Sharlin 3mo agoBack in 2024 we didn’t really yet have image models that used bigger language models as their text encoders, so prompt understanding was still rather hit and miss, especially with negations involved.
- julianz 3mo agoI asked Gemini 3.6 Flash for a pūkeko on a unicycle yesterday and the results were very stylish. The thinking notes were cool too, making sure it got all the high points of what it means to be a pūkeko (distinctive swamp hen found in NZ and Australia for anyone not familiar) and then figuring out how the legs would work with the pedals.
- Melatonic 3mo agoIt's all fun and games until that Pelican becomes self aware and asks themselves - why am I riding this bicycle ? I could be creating a Terminator 2 esque dystopian future myself. And that's how we end up with PelicanSkynet
- DonHopkins 3mo agoA pelican on a bicycle is like a canary in a coal mine. And a man without religion is like a fish without a bicycle.
- graemep 3mo agoI am not sure being better at pelicans on bicycles would be proof of cheating. Could all the commentary on the web on pelican on bicycle SVGs not influence training without anything being done deliberately?