6 ms·
I generated pelicans riding bicycles on both thinking level low and thinking level high: https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304dc6?permal
by simonw 5mo ago
I generated pelicans riding bicycles on both thinking level low and thinking level high:
https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304dc6?permalink_comment_id=6172953#gistcomment-6172953 https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304...
The high one is notably better - the bicycle frame is the correct shape, unlike thinking level low.
For comparison, here's Opus 4.7: https://gist.github.com/simonw/afcb19addf3f38eb1996e1ebe749c118?permalink_comment_id=6104087#gistcomment-6104087 https://gist.github.com/simonw/afcb19addf3f38eb1996e1ebe749c...
- onlyrealcuzzo 5mo ago4.7 reigns supreme IMO.
- nickvec 5mo agoIs the "opossum riding an e-scooter" benchmark in the works for Opus 4.8? ;)
- simonw 5mo agoGood call, it's cute: https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304dc6?permalink_comment_id=6172972#gistcomment-6172972 https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304... - but nothing like GLM-5.1: shttps://static.simonwillison.net/static/2026/glm-possum-escooter.html https://static.simonwillison.net/static/2026/glm-possum-esco...
- glm51_wfstc 5mo ago[dead]
- yanis_t 5mo agoSimon, is your pelican test really captures differences among models or should you at least try like 10 times or something to average the random effects
- simonw 5mo agoI've been meaning to do a "run 3 times and pick the best" version for quite a while, I should really pull the trigger on that one. Currently it's one-shot only.
- xiphias2 5mo agoBest-of-3 would be cheating, ruin the test, middle of 3 makes more sense
- nik736 5mo agoWhy would you need the 3rd run if you pick the "one in the middle"?
- jmaw 5mo agoMiddle as in not the best, and not the worst. As opposed to the second generated in sequence. But not the best/not the worst is somewhat subjective.. so not sure how well that would work.
- BrokenCogs 5mo agoI think GP meant picking the median pelican
- notaharvardmba 5mo agoYou could run 3 times and overlay/average the images to show how consistent they are
- 1attice 5mo agoThat little red hat on hard mode is sending me. 4.8 has whimsy
- jonas21 5mo agoGlad to see that the "high thinking" level adds a helmet. Always a smart choice.
- usef- 5mo agoAnd yet some people doubt Anthropic's commitment to AI safety
- Xunjin 5mo agoHey simonw I love your test, do you think using thinking level "max" makes sense for this test? I would love to see the results about it.
- simonw 5mo agoI don't think the API supports "max" as an option, that might just be a Claude Code harness thing. UPDATE: My mistake, the API does support max. I added a max one at the bottom of this page (cost 43 cents): https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2Ffea4f7546626d627862dc241a4e3a86a https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
- Xunjin 5mo agoThe legend.
- spmartin823 5mo agoYou've peed in the pool Simon, this has to be a part of the internal evals by now! You got to try something new - maybe a panda in a canoe?
- phainopepla2 5mo agoIf these were in the internal evals then the output would be much better. The 4.8 pelicans are pretty meh
- HDThoreaun 5mo agoClick the link
- ceroxylon 5mo agoI really like that thinking level high gave the pelican a helmet.
- timsuchanek 5mo agothanks for always providing this very much on time. I'm wondering what the next, harder challenge could be? Maybe some animated svg?
- highwaylights 5mo agoAm I allowed to say that pelican's little helmet is adorable? I can't provide a strong computational proof, or even a shred of anecdata... ...but that pelican's little helmet is adorable.
- GistNoesis 5mo ago> the bicycle frame is the correct shape No, the handlebar is wrong. The handle bar is rotating the frame instead of rotating the front wheel. The handle bar should be mounted on the same line as the front wheel is. Hopefully 4.9 will read my comments :)
- loeg 5mo agoCould be an extremely high angle stem that just happens to match the downtube angle.
- Venkatesh10 5mo agoMaybe the pelican is just riding a road bike/gravel bike
- toastmaster11 5mo agoI find the most miraculous thing about 4.7 to be that the pelican is facing left, wonder why the right facing everything is so ubiquitous in these images.
- i000 5mo agoThis happened to me in elementary school. We were doing fingerpaintings using plasticine. After all the bikes were hung on the wall, mine was racing the other way... Somehow it really stuck with me.
- sunnybeetroot 5mo agoWhat do you think it means?
- gboss 5mo agoIt's facing left but looking right...
- toastmaster11 5mo agoProfound political commentary?
- tancop 5mo ago[dead]
- whalesalad 5mo agoEventually the frontier model folks are going to pick up on your pelican on a bike test and bake-in flawless results for that particular request.
- silisili 5mo agoThe vast majority (if not all) of these make it impossible to turn, among other fun things. Only out of curiosity, have you tried prompting further with how a bike must operate to see if it does the right thing?
- fendy3002 5mo agotried it myself, not much of difference https://gist.github.com/fendy3002/3026a8c4d67d1301666ec40fc02c785b https://gist.github.com/fendy3002/3026a8c4d67d1301666ec40fc0... looks like the model already trained well on both bicycle and pelicans
- simonw 5mo agoHere's pelicans in all of the thinking levels - low, medium, high, xhigh, max https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2Ffea4f7546626d627862dc241a4e3a86a https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
- stratos123 5mo agoIs the output on the max level meant to be missing?
- simonw 5mo agoI just fixed that (force refresh). It hit my default 8,000 output token limit, it worked when I bumped that up. For max I used 25 input, 17,167 output which cost me 43 cents! https://www.llm-prices.com/#it=25&ot=17167&ic=5&oc=25&sel=claude-opus-4-5 https://www.llm-prices.com/#it=25&ot=17167&ic=5&oc=25&sel=cl...
- ionwake 5mo agoI like the way the max pelican has a stern look on his face
- deleted 5mo ago[deleted]
- motza 5mo agolow: yolo medium: redesign bike so peli can reach bars high: redesign bike so peli can rest on frame xhigh: yolo max: big peli reach bars
- virgildotcodes 5mo agoMax seems to me to be notably better than the others.
- eminence32 5mo agoI bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so: https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/ Turns out even humans can be pretty bad at drawing bicycles :)
- skydhash 5mo agoBut if you need to draw a bicycle, you wouldn’t pick a random person in the street. You would hire an artist and you’d be guaranteed to have at least a believable one if not a perfect rendering. No guarantees is why LLM is akin to gambling. Every new context is essentially picking someone out of the crowd.
- walthamstow 5mo agoOn a new model release, you can guarantee two things are in the replies to Simon. One is your link, the other is "surely the models are being trained on this now"
- kvirani 5mo ago> The most unintelligible drawing has also the most unintelligible handwriting. It was made by a doctor. Haha
- saghm 5mo agoSure, but no one is trying to force art from most people into about every area in the economy where anyone ever pays for something visual. If you asked professional artists to draw a realistic bicycle, I'm guessing few of them would try to just randomly guess what the mechanical parts looked like
- jodrellblank 5mo agoAs an aside, some of the renders have only a single side connection to the wheel and that is a valid bike design, the Cannondale Lefty front fork only has a left leg: https://duckduckgo.com/?q=cannondale+lefty&iar=images&t=ffab https://duckduckgo.com/?q=cannondale+lefty&iar=images&t=ffab
- fragmede 5mo agoFor comparison, what's GPT-5.5 producing today?
- simonw 5mo agoThe reasoning xhigh one is pretty solid: https://simonwillison.net/2026/Apr/23/gpt-5-5/#and-some-pelicans https://simonwillison.net/2026/Apr/23/gpt-5-5/#and-some-peli...
- fragmede 5mo agoLends credence to my vibe-based assertion that GPT-5.5 > Opus 4.7 (and now 4.8), which is why I've cancelled my Claude plan. Opus 4.8 is them seeing it reflected in their own numbers and having to pull stopgap measures to avoid falling behind while they embargo Mythos.
- impalallama 5mo agoI actually like the 4.7 the most, interestingly enough. Not like you can "objectively" weight artistic output like this.
- prmoustache 5mo agoI don't see how a frame without a headtube can be "the correct shape".
- keyle 5mo agoIt's pretty safe to say that AI will be used on the battlefield making real life and death decisions before it will be able to render a decent pelican on a bike in SVG.
- Kiro 5mo agoI think it's beyond decent. I don't understand how people are not more impressed by this. Just a few years ago the only expectation would be garbled nonsense.
- notatoad 5mo agothe battlefield sounds much easier. worst case scenario you kill somebody, but that's what you're trying to do anyways. if you kill somebody while trying to render a pelican on a bicycle it's a real problem.
- ares623 5mo ago"shift left" on the battlefield. break down those silos. if you have to ask for permission it's already too late. remember the goal. find the bottlenecks in your system and remove them.
- pwagland 5mo agoIn many battlefield scenarios, there is more than one "somebody" on it. The "somebody" that you kill might not be the "somebody" that you intended to kill. Depending on the how pelicans are created, it is entirely possible to indirectly kill "somebody" due to the externalised costs of global warming etc.
- culi 5mo agoIt already has been and this has been widely written about. AI was used to identify and prioritize targets for the US to bomb in Iran. Here's an article from 2 months ago for example: https://www.theguardian.com/technology/commentisfree/2026/mar/06/the-guardian-view-on-ai-in-war-the-iran-conflict-shows-that-the-paradigm-shift-has-already-begun https://www.theguardian.com/technology/commentisfree/2026/ma... It was also implicated in the bombing of a girls elementary school which left 168 dead. The US did a "triple tap" to kill any first responders. https://www.theguardian.com/news/2026/mar/26/ai-got-the-blame-for-the-iran-school-bombing-the-truth-is-far-more-worrying https://www.theguardian.com/news/2026/mar/26/ai-got-the-blam... https://www.theguardian.com/technology/2026/apr/01/dont-blame-ai-for-the-iran-school-bombing https://www.theguardian.com/technology/2026/apr/01/dont-blam...
- alex_duf 5mo agoIt's funny that we've reached the level where LLMs draw more correct bikes than any random person
- lysecret 5mo agoSadly I think the correlation between this benchmark and performance is starting to break down imo. Still a legendary idea will be remembered and ingrained in the models forever haha