4 ms·
I posted this in the other Astra thread but it's just fallen off the homepage, so... Pelicans from Astra, plus 5.6 Sol, Terra, Luna for comparison: https://sta
by simonw 1mo ago
I posted this in the other Astra thread but it's just fallen off the homepage, so...
Pelicans from Astra, plus 5.6 Sol, Terra, Luna for comparison: https://static.simonwillison.net/static/2026/gpt-6-and-5.6-pelicans.html https://static.simonwillison.net/static/2026/gpt-6-and-5.6-p...
I think this is a genuinely interesting comparison grid. Astra may be more expensive, but if you have a budget of 10 cents for a Pelican Astra low gives you something SO much better than the other models.
Astra uses less tokens overall too, for better results.
Astra transcript here: https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2Ff789d2784fc6c5b870cc80f0b7cd9d01 https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
- leoqa 1mo ago[flagged]
- ComplexSystems 1mo agoIt's absolutely related.
- samuelknight 1mo agoHow are we supposed to know if Astra is frontier without the pelican?
- satvikpendem 1mo agoIt's simonw. It's interesting to see their pelican benchmark, another comment by a different author elsewhere here shows some very good SVG generation too.
- Xunjin 1mo agoDo you have other ideas of "combinations" for this kind of benchmark? I'm wondering if this is being trained on by the models today.
- r_lee 1mo agoI have a feeling they've been doing that for a while now, even if unintentional, as it's such a well known benchmark
- BrokenCogs 1mo agoInteresting that all of the bikes are turquoise colored, except for the medium effort
- deleted 1mo ago[deleted]
- lspears 1mo agoLuna's price seems off
- droidjj 1mo agoThe price is probably coming from when Luna was first released. OpenAI slashed the price by 80% at the end of July.
- simonw 1mo agoYes! Good catch, thanks - I'll fix that.
- simonw 1mo agoI had the prices wrong on Sol and Terra as well - they've all had price drops: https://openai.com/index/advancing-the-price-performance-frontier-with-gpt-5-6/ https://openai.com/index/advancing-the-price-performance-fro... Sol discount is until November 21, 2026 according to https://developers.openai.com/api/docs/changelog https://developers.openai.com/api/docs/changelog Luna — costs in cents +--------+--------+---------+ | Effort | Before | After | +--------+--------+---------+ | max | 7.83 | 1.57 | | xhigh | 4.24 | 0.85 | | high | 2.46 | 0.49 | | medium | 1.26 | 0.25 | | low | 0.76 | 0.15 | | none | 0.71 | 0.14 | +--------+--------+---------+ Per million tokens: Before: $1 input / $6 output After: $0.20 input / $1.20 output Sol — costs in cents +--------+--------+---------+ | Effort | Before | After | +--------+--------+---------+ | max | 48.55 | 32.37 | | xhigh | 24.11 | 16.08 | | high | 10.38 | 6.92 | | medium | 10.55 | 7.03 | | low | 8.33 | 5.55 | | none | 5.90 | 3.93 | +--------+--------+---------+ Per million tokens: Before: $5 input / $30 output After: $4 input / $20 output Terra — costs in cents +--------+--------+---------+ | Effort | Before | After | +--------+--------+---------+ | max | 32.09 | 25.67 | | xhigh | 14.67 | 11.74 | | high | 3.74 | 2.99 | | medium | 3.46 | 2.77 | | low | 3.47 | 2.78 | | none | 2.60 | 2.08 | +--------+--------+---------+ Per million tokens: Before: $2.50 input / $15 output After: $2 input / $12 output
- deleted 1mo ago[deleted]
- StopTheCringe 1mo ago[flagged]
- andai 1mo agoOh my goodness. I was not prepared for Luna on "none". Reminded me of https://clocks.brianmoore.com/ https://clocks.brianmoore.com/
- vb-8448 1mo agoI find curios that Astra's pelicans are basically the same (yellow sun top right corner, green bike, same bike shape, same legs style, very similar background) while in other there is more randomness.
- thimabi 1mo agoThis tracks with what OpenAI has been saying about Astra: that it tends to do things in a certain way and it’s up to you to prompt it to change its style.
- pizza234 1mo agoAstra is the only model that correctly depicts occlusion of crank and leg, although interestingly, at max and medium levels (not in between).
- vessenes 1mo agoLooked to me like it missed the chain though.
- TiredOfLife 1mo agoKimi k3 and Fable max also
- jsdalton 1mo agoSimon do you have a page somewhere that shows _all_ of the penguins created by various models across time? You’ve been doing this public service for so long (well, for so long in “AI hype” years anyway) that it’d be fascinating to see the evolution of this artifact across time.
- simonw 1mo agoFor the moment the best place to see them all is to browse the tag on my blog - 140 posts now! https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/ I'm running out of excuses not to build a proper comparison site though. Maybe I'll have Astra do rhat.
- bahmboo 1mo agoWould be great to spend some money (other people's money) on paying some illustrators and artists to do the same for comparison. It could be straight images converted to SVG or direct vector graphics.
- Gander5739 1mo agoDoesn't that somewhat miss the point? The models can't see and check their work, for instance.
- bahmboo 1mo agoI'm not clear what you are saying. I am curious about what humans can do in the same context. It's becoming a standard I think for some tasks. E.g. how does a Gen AI stack up against a human in quality and "cost". Perhaps I'm missing what you are asking.
- spockz 1mo agoThey can. Ask them to generate SVG , call a renderer and let them inspect the bitmap/png. They can learn about it.
- fHr 1mo agoLuna is my go to daily model it's great value
- bnorton 1mo agoYou’re closer to this than I am but do you find it odd that helmets are almost never included? Biking is almost always accompanied by helmets
- maxlapdev 1mo agoBut pelicans are almost never accompanied by helmets, so it cancels out.
- neutronicus 1mo agoAt least one reasoning trace I saw considered a helmet and discarded the idea because it was worried about obscuring some detail
- sumedh 1mo ago> Biking is almost always accompanied by helmets Isnt Netherlands the leader in bike riders and they dont wear helmets.
- kart23 1mo agoradial spokes on the back wheel is not a thing irl. i wonder if any model has gotten that right?
- gpugreg 1mo agoI scrolled through https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/ but did not see any image where the spokes were correct. For a moment, I thought that the text-to-image model might have gotten it right, but on closer look, the spokes fork https://static.simonwillison.net/static/2026/why-are-you-like-this.jpeg https://static.simonwillison.net/static/2026/why-are-you-lik... But I enjoyed the image anyway.
- petilon 1mo agoAstra pelican looks amazing: it looks like it was made by a professional artist. All the others look like they were made by kindergartners.
- steve-atx-7600 1mo agoYou would not expect the developers of the model to optimize for a well known benchmark?
- y1n0 1mo agoI would expect it happens more organically where people’s discussion of the benchmark and posted results find their way into the training data, like anything else on the internet.
- Kranar 1mo agoWhat is this supposed to mean exactly? Do you think there are developers at OpenAI or Anthropic whose job is to train these state of the art models to draw pelicans riding bicycles? Like how exactly do you expect them to be doing that anyways? Hiring graphic artists to create SVGs of bike riding pelicans and feeding thousands of them into the model's training set?
- awakeasleep 1mo agoYou gotta look up how RLHF works before you ask a demanding question like this.
- WarmWash 1mo agoRight, but that's not the crazy part. The crazy part is thinking they do it all specifically for pelicans on bikes. If that was the case, the models would have been producing near perfect outputs for it a year ago. Instead they are just training on general SVG generation, which in no way should be viewed as "benchmaxxing".
- mudkipdev 1mo agoGenerate two images, get a vision model to judge, then RL reward the better one. And yes, OpenAI employees on twitter bragged about the model's SVG capabilities. So there are clearly people working there who care about it.
- dgellow 1mo ago
- jedjjfjf 1mo ago[dead]
- man4 1mo ago[dead]
- PacificSpecific 1mo agoFallen off the homepage so what? I genuinely don't understand
- yreg 1mo agoSo he is posting it again because he believes it is crucial information for us. Apparently the crowd agrees because they keep upvoting these.
- dolebirchwood 1mo agoGotta double-dip on those internet points.
- threatripper 1mo agoIt seems to start understanding how bicycles work. Look at the front fork. There are reasons why it's shaped the way it is in real bicycles. If you do physical simulations in 3D with reinforcement learning you will start understanding that most of the arrangements that the other agents use just won't work.
- epihelix 1mo agoYes, the curve of the front fork on the max pelican was what I noticed also. It's impressive (if not an accident), and a remarkably accurate bike overall. The pelican just needs to raise their seat a little.
- jayd16 1mo agoProbably just luck seeing at the wheel intersects the frame and the penguin's legs are caught in the chain.
- CamperBob2 1mo agoCrazy how much the Astra pelicans resemble GLM 5.3's (https://crimson-jeri-74.tiiny.site/ https://crimson-jeri-74.tiiny.site/), down to the color of the bike and the scarf. The improved front fork design mentioned by Threatripper is about the only thing Astra is doing better, IMO.
- dector 1mo agoBtw, they had 3D pelican-on-the-bicycle easter egg in one of the promo videos: https://youtu.be/bOC3DisEOfg?t=117 https://youtu.be/bOC3DisEOfg?t=117 so I'm pretty sure that they spent some small amount of resources to train the model to produce good svg version as well. :D
- theturtletalks 1mo agoThis is one of those benchmarks I don’t mind if they benchmax because it would mean models are actually good at creating svg graphics.
- theturtletalks 1mo agoLook at how well it made this Xbox controller SVG: https://www.svgviewer.dev/s/i8t1VXzQ https://www.svgviewer.dev/s/i8t1VXzQ
- alexgoodhart 1mo agoI’m not sure that this follows. And I don’t understand why the pelican bro is not purposefully demonstrating variety of svg tasks to begin with
- anthonyrstevens 1mo agoBecause sequential comparison is part of the point of doing them?
- SkiFire13 1mo agoNo, it would mean models are good at pelicans svgs, not svgs in general.
- alastairr 1mo agoIt's mildly interesting that the bikes always seem to be front wheel facing the right.
- poopiokaka 1mo ago[dead]
- mkagenius 1mo agoDo you ever randomize on a particular model - like try and get 3 outputs and pick one at random? Coz who knows if astra low will produce max like output if tried once more.
- jeffybefffy519 1mo agoShouldnt you give the AI a different task each time? otherwise the model companies just optimise for this benchmark because its in their best interest to...
- EugeneOZ 1mo agoI love Astra pelicans! Awesome results!
- justinbaker84 1mo agoYour comparison grid is one of the most useful comparison grids I have ever seen in terms of measuring AI. It feels silly to say that about making a pelican image but it really shows the difference in output and cost in an easy to understand way.
- scotty79 1mo agoThey really can't decide which way pelican's knees should bend, do they?
- mkl 1mo agoWhy do Sol and Terra use 26 input tokens when the others use 16?
- simonw 1mo agoI don't know and I'm really Interested in the answer. I heard a rumor that Luna is a slightly different architecture from Sol and Terra, which makes me wonder if Luna and Astra might be more related to each other than to Sol and Terra. Bit of a big leap to make from a token count though! I just checked the tiktoken library and couldn't see any changes relating to Luna: https://github.com/openai/tiktoken https://github.com/openai/tiktoken
- az226 1mo agoThe biggest difference is that Astra was pre-trained with reasoning traces, not just post-trained.