4 ms·
Here are 18 pelicans - six each for Luna, Terra and Sol at the six different reasoning effort levels (plus the price to generate each one): https://static.simon
by simonw 4mo ago
Here are 18 pelicans - six each for Luna, Terra and Sol at the six different reasoning effort levels (plus the price to generate each one): https://static.simonwillison.net/static/2026/gpt-5.6-pelicans.html https://static.simonwillison.net/static/2026/gpt-5.6-pelican...
Or if you want to see some in 3D, OpenAI featured a pelican riding a tricycle, bicycle, pony and another pelican in their livestream this morning: https://www.youtube.com/live/Wq45rvPGNHs?t=1070s https://www.youtube.com/live/Wq45rvPGNHs?t=1070s
- semiquaver 4mo agoThe quality of sol on effort=none makes me think this test is saturated or they are benchmarkmaxxing this exercise.
- yokoprime 4mo agoThank you Simon! Luna is surprisingly decent across all reasoning levels.
- vecter 4mo agoI think all of Luna's are bad. The only decent one is sol @ xhigh. Even sol @ max is weird. Sol @ high and @ medium are ok, and every other single one across every model is bad.
- hn_throwaway_99 4mo agoStrong disagree, but to each their own. For sol I really like how only medium uses the wings on the handlebars to ride the bike. For all the other sols the pelican evolved a new set of arms separate from the wings.
- emehex 4mo agogpt-5.6-sol x XHIGH is my favourite
- throwaw12 4mo agosomething is wrong with Terra model series, most pelicans, except Max, looks bad
- CryZe 4mo agoSeems to match the pareto frontier on Artificial Analysis as well. Terra is nowhere on it.
- siva7 4mo agoOk, I'll never use max effort again on OAI models..
- isoprophlex 4mo agoIs that... an x-rated, censored pelican?
- cma256 4mo agoIs the direction of the pelicans encoded in your prompt? Curious why they are all left to right with the exception of terra xhigh.
- hirvi74 4mo agoI like Terra High the best. That pelican is utterly yoked.
- deleted 4mo ago[deleted]
- miohtama 4mo agoAt what stages will models start to internally reflect the drawn SVG and automatically fix their own mistakes? I assume multimodal models can do it already do it today if constantly asked "make it better"
- firebot 4mo agoAI really sucks at bicycles...
- rsoto2 4mo agoLLMs really suck*
- algoth1 4mo agogpt-5.6-sol Max pelican didn’t skip neck day
- aappleby 4mo agoSurely "how to draw a SVG pelican on a bike" has made it into the training data by now ...
- conception 4mo agoAnd yet... only one is any good.
- huseyinkeles 4mo agoSurely this comment is literally on every new model release post.
- simonw 4mo agoIt's part of the pelican tradition at this point.
- dannyobrien 4mo agoI eagerly await the models replying with that: "I'd be happy to create a pelican riding a bicycle, but just a note that this might already be in my training data. Simon."
- simonw 4mo agoSomeone told me that a model said "oh, the classic" when they asked for this recently. I think that was Opus 4.8.
- astrange 4mo agoIf that was the case in a non-trivial way you'd see mode collapse, but you don't, they come out differently.
- tlamponi 4mo agoIt's because all the frequent comment that this pelican is in the trainings set now also got into the trainings set and models adapt. /joking (I hope)
- ismailmaj 4mo agoApparently plus users do not have access to Sol, so I'm really worried about the ugly Terra Pelican.
- pupppet 4mo agoTime to dump this test. Probably not a coincidence every version has the same rolling green hills, gradient blue sky, sun in the corner, etc.
- fastball 4mo agoOn the one hand: yes, pelicans on bikes are definitely in the training set at this point. On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.
- pupppet 4mo agoI sort of agree, but within the same model I expect the reasoning effort to be reflected in the quality of output and that's basically how it played out. When you're comparing different models, then it's just who benchmaxxed the best and there's not a lot of value there.
- fastball 4mo agoBut that is my point: if benchmaxxing was all the labs were doing, then surely the dumber model could/would have equivalent performance? Rather than noticeably worse perf on a (somewhat trivial to game) test.
- dbt00 4mo agoGoodhart's law.
- modriano 4mo agoI don't know. If they were training on this, I feel like they would be able to get the shape of a bike frame right; it's a pretty simple polygon, and a lot of the bike frames that are getting generated would be impossible to steer.
- mbauman 4mo agoI mean, humans can't draw bikes! https://themagnet.substack.com/p/why-is-it-so-hard-to-draw-a-bike https://themagnet.substack.com/p/why-is-it-so-hard-to-draw-a...
- Night_Thastus 4mo agoI think the 'pelican test' is becoming useless. It's been around long enough that now I'm sure good examples are in the training data, and hell they might even do some hand tuning to make it do a decent job since they know people will ask about it. But either way, with no real way to visualize the result of the text it starts with - it will always be stabbing in the dark. It can't understand conceptually what any of it should look like and then refine the SVG to improve it gradually. It just throws darts at a wall and hopes it comes out alright.
- skybrian 4mo agoI think it's still useful in a "hello world" sort of way. It means you actually tried the new model.
- simonw 4mo agoHonestly that's the main value I get from it myself - making a pelican means I have to figure out API keys and how to talk to the provider, or how to run it locally for the local models.
- observationist 4mo agoPelicans, maybe, but the point is to measure how good the "internal visualization" abilities are. Throw curveballs, like a unicorn with a duck bill serving coffee at a basketball court. An elephant playing a piano while its trunk swings a baseball bat at a tiny alien spaceship buzzing its head. Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint. Compositionality and visualization are generally much, much better at each new generation / release cycle. It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.
- tandr 4mo agoStill fun to watch models trying their best :). I think "money spent" metric in this test is becoming the most interesting to watch. It is like looking at RT cost of "Hello, world!"...
- alecco 4mo ago[flagged]
- dang 4mo agoYou can't post like this to HN, regardless of how you feel about someone else's posts. Doing it repeatedly crosses into harassment, and you've done it more than 3 times now - e.g. https://news.ycombinator.com/item?id=48504823 https://news.ycombinator.com/item?id=48504823 https://news.ycombinator.com/item?id=48504654 https://news.ycombinator.com/item?id=48504654 So please especially stop doing that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- alecco 4mo agoThose 2 comments are from a month ago, on the same day, and my point was he was spamming those threads with basically the same comment, and he links often to his site and his posts are all about his software. Both of my replies you link have positive counts (with sure had plenty of downvotes as he had positive upvotes), and even now after your link probably added more downvotes to mine. And other commenters in those threads agree with me. HN search shows 21 comments in the past month from simonw+pelican: https://hn.algolia.com/?dateRange=pastMonth&page=0&prefix=true&query=simonw%20pelican&sort=byDate&type=comment https://hn.algolia.com/?dateRange=pastMonth&page=0&prefix=tr... You can choose to look the other way but 3 comments out of 21, where 2 were kind of the same one on the same day, and 27 days ago... But you say "repeatedly crosses into harassment" as if I'm a troll chasing him around. I was caring about the quality of this site with my 18 year old account... Unbelievable. But as you bless him, I give up. It's your site. Quite a bummer after I defended you many times over the years, though. I remember in the beginning your interventions were not generally welcome. Edit: His comment is now 12th in this thread and there are comments saying his pelican SVG test is pointless. At least there's some sanity with other users. Or are they harassing him, too? Sadly his other comment on Hy3 thread is still the top comment. And of course it also has a link to his site. https://news.ycombinator.com/item?id=48848950 https://news.ycombinator.com/item?id=48848950
- irthomasthomas 4mo agoCool. I still find these a useful visualization of some the qualities of llms. Even if they did train for [animal] on [vehicle] svg, it's still nice to see at a glance how the different models and reasoning levels perform. Lunar misses part of the frame, except on max reasoning. While most of the others have a mostly correct bike at all reasoning levels. I once used something like karpathy's auto-scientist to mutate the prompts and rank them with a vison model. Some of the winners where pretty neat. I think they have a lot more style than the gpt-5.6 ones. https://xcancel.com/xundecidability/status/2044918567414419626?s=20 https://xcancel.com/xundecidability/status/20449185674144196...
- ashu1461 4mo agoThey said in the AI community, a pelican riding a bicycle is a good test to measure effectiveness of the model, wondering if they were referring to you, or is it really a standard in the AI community ? Also would be good to have a tool where users can select models and instantly see each model's generated pelicans. That will make it easy to compare the output of different models.
- jjice 4mo agoSimon did start the pelicans on bicycles as an SVG, but I think it's more of a fun goofy thing to see how the model performs at. I don't think it has a direct correlation to a model's performance though.
- JeremyHerrman 4mo agopeople are saying this is benchmark is saturated but all of these have occlusion issues, even sol max. A skilled human artist wouldn't have both legs in front of the bike, or a single straight line representing both leg's crank arms.
- MichaelZuo 4mo agoYeah it makes no sense at all to dismiss the test, when even the very best examples are noticeably below what a skilled teenager could produce. Dead internet theory? Semi-random parroting by real people? Or something else.
- rsoto2 4mo agothis looks like the same shit from 4 years ago. give it up.
- simonw 4mo agoThe first time I did this was actually less than two years ago - in October 2024 - and it's fun seeing how much better they've got since then: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/ https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
- tezza 4mo agoAbsolutely, the change in quality over time is a great yardstick. Nice to see you did the quality level comparisons and did three passes. I've been using that technique myself on my image gen reviews[1] and it also works well in presentations and for personal study. [1] https://generative-ai.review/2025/12/beast-mode-activated-openai-image-1-5/#bm-rearing-horse https://generative-ai.review/2025/12/beast-mode-activated-op...
- zniturah 4mo agomax effort sol clearly over-engineered
- attentive 4mo agosomehow Terra really struggles here even compare to Luna.
- walrus01 4mo agoI'm waiting for the day that the "generate a Pelican" test comes back with a SVG-art like illustration of a Pelican equipment case, like a model 1620 or similar. https://www.google.com/search?client=firefox-b-d&q=pelican+1620 https://www.google.com/search?client=firefox-b-d&q=pelican+1...
- bombcar 4mo agoWhat's strange with this is the prompt "Photorealistic photograph of a pelican riding a bicycle down a coastal boardwalk, wings gripping the handlebars, webbed feet on the pedals, large orange bill, detailed feather texture, golden hour lighting, shallow depth of field, shot on a DSLR with 85mm lens, natural motion blur on the wheels" produced, well, exactly what I asked it for. I wonder if I tell it then to make it SVG ... https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d0442 https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d04...
- deleted 4mo ago[deleted]
- bansimonw 4mo ago[dead]
- rldjbpin 4mo agoterra is just weird. in this nothingburger test, time nor higher costs seem to not strongly correlate with the aesthetics.