5 ms·
Wow, there's a lot going on with this pelican riding a bicycle: https://gist.github.com/simonw/c31d7afc95fe6b40506a9562b5e83bcc?permalink_comment_id=5901194#gis
by simonw 11mo ago
Wow, there's a lot going on with this pelican riding a bicycle: https://gist.github.com/simonw/c31d7afc95fe6b40506a9562b5e83bcc?permalink_comment_id=5901194#gistcomment-5901194 https://gist.github.com/simonw/c31d7afc95fe6b40506a9562b5e83...
- minimaxir 11mo agoIs that the first SVG pelican with drop shadows?
- simonw 11mo agoNo, I got drop shadows from DeepSeek 3.2 recently https://simonwillison.net/2025/Dec/1/deepseek-v32/ https://simonwillison.net/2025/Dec/1/deepseek-v32/ (probably others as well.)
- tmaly 11mo agoseems to be eating something
- danans 11mo agoProbably a jellyfish. You're seeing the tentacles
- belter 11mo agoWhat happens if you ask for a pterodactyl on a motorbike? Would like to know how much they are optimizing for your pelican....
- simonkagedal 11mo agoHe commented on this here: https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
- irthomasthomas 11mo agoI was expecting to see a pterodactyl :(
- fxwin 11mo agothe only benchmark i trust
- BeetleB 11mo agoThey probably saw your complaint that 5.1 was too spartan and a regression (I had the same experience with 5.1 in the POV-Ray version - have yet to try 5.2 out...).
- Stevvo 11mo agoThe variance is way too high for this test to have any value at all. I ran it 10 times, and each pelican on a bicycle was a better rendition than that, about half of them you could say were perfect.
- golly_ned 11mo agoCompared to the other benchmarks which are much more gameable, I trust PelicanBikeEval way more.
- deleted 11mo ago[deleted]
- refulgentis 11mo ago[flagged]
- throwaway102398 11mo ago[dead]
- getnormality 11mo agoWell, the variance is itself interesting.
- AstroBen 11mo agoSeems to be getting more aerodynamic. A clear sign of AI intelligence
- sroussey 11mo agoWhat is good at SVG design?
- azinman2 11mo agoGraphic designers?
- culi 11mo agoNot svg, but basically the same challenge: https://clocks.brianmoore.com/ https://clocks.brianmoore.com/ Probably Kimi or Deepseek are best
- KellyCriterion 11mo agoIve not seen any model being good in graphic/svg creation so far - all of the stuff mostly looks ugly and somewhat "synthetic-disorted". And lately, Claude (web) started to draw ascii charts from one day to another indstead of colorful infographicstyled-images as it did before (they were only slightly better than the ascii charts)
- nightshift1 11mo agobenchmarks probably should not be used for so long.
- alechewitt 11mo agoNice work on these benchmarks Simon. I’ve followed your blog closely since your great talk at the AI Engineers World Fair, and I want to say thank you for all the high quality content you share for free. It’s become my primary source for keeping up to date. I’ve been working on a few benchmarks to test how well LLMs can recreate interfaces from screenshots. (https://github.com/alechewitt/llm-ui-challenge https://github.com/alechewitt/llm-ui-challenge). From my basic tests, it seems GPT-5.2 is slightly better at these UI recreations. For example, in the MS Word replica, it implemented the undo/redo buttons as well as the bold/italic formatting that GPT-5.1 handled, and it generally seemed a bit closer to the original screenshot (https://alechewitt.github.io/llm-ui-challenge/outputs/microsoft_word/gpt_5_2.html https://alechewitt.github.io/llm-ui-challenge/outputs/micros...). In the VS Code test, it also added the tabs that weren’t visible in the screenshot! (https://alechewitt.github.io/llm-ui-challenge/outputs/vs_code/gpt_5_2.html https://alechewitt.github.io/llm-ui-challenge/outputs/vs_cod...).
- simonw 11mo agoThat is a very good benchmark. Interesting to see GPT-5.2 delivering on the promise of better vision support there.
- tkgally 11mo agoI added GPT-5.2 Pro to my pelican-alternatives benchmark for the first three prompts: Generate an SVG of an octopus operating a pipe organ Generate an SVG of a giraffe assembling a grandfather clock Generate an SVG of a starfish driving a bulldozer https://gally.net/temp/20251107pelican-alternatives/index.html https://gally.net/temp/20251107pelican-alternatives/index.ht... GPT-5.2 Pro cost about 80 cents per prompt through OpenRouter, so I stopped there. I don’t feel like spending that much on all thirty prompts.
- smusamashah 11mo agoHi, it doesn't have Gemini 3.5 Pro which seems to be the best at this
- svantana 11mo agoThat's probably because "Gemini 3.5 Pro" doesn't exist
- philipgross 11mo agoThat gallery is an excellent advertisement for Gemini 3.0 Pro.
- tootie 11mo agoDo you think the big guys are on to your game and have been adding extra pelicans to the training data?