6 ms·
We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It
by Jimmc414 11mo ago
We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. It would be a great way to ensure an initial thumbs up from a prominent reviewer. It's a good benchmark but it seems like it would be a good idea to include an additional random or unannounced similar test to catch any benchmaxxing.
- deleted 11mo ago[deleted]
- simonw 11mo agoI wrote about that possibility here: https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
- th0ma5 11mo ago[flagged]
- dugidugout 11mo agoCondescending and disrespectful to whom? Everybody wholsale? This doesnt seem reasonable? Please elaborate.
- th0ma5 11mo agoNo, when did I say that?
- deleted 11mo ago[deleted]
- dugidugout 11mo agoIt isn't clear what you said. You asserted a pattern of conduct on the user simonw: > I think constantly replying to everybody with some link which doesn't address their concerns Then claimed that conduct was: > condescending and disrespectful. I am asking you to elaborate to whom simonw is condescending and disrespecting. I don't see how it follows.
- bravetraveler 11mo agoNot sure if I'd use the same descriptions so pointedly, but I can see what they mean. It's perfectly fine to link for convenience, but it does feel a little disrespectful/SEO-y to not 'continue the conversation'. A summary in the very least, how exactly it pertains. Sell us. In a sense, link-dropping [alone] is saying: "go read this and establish my rhetorical/social position, I'm done here" Imagine meeting an author/producer/whatever you liked. You'd want to talk about their work, how they created it, the impact it had, and so on. Now imagine if they did that... or if they waved their hand vaguely at a catalog.
- tomrod 11mo agoHell, I would consider myself graced that simonw, yes, THAT simonw, the LLM whisperer, took time out of his busy schedule to send me to a discussion I might have expressed interest in.
- bravetraveler 11mo ago> send me to a discussion I might have expressed interest in No, no, remember? Points to the blog you were already reading! Working diligently to build a brand: podcast, paid newsletter, the works.
- tomrod 11mo agoI wasn't speaking to this interaction, and my point is genuine. Simonw has done fantastic work in the LLM space
- bravetraveler 11mo ago... and my point remains: he's fine. Could be better. If he does grace us, he can choose to bait the hook more effectively. Or not. The stakes are silly-low. This interaction is, effectively, a link dropped with an announcement of an announcement. For what has already occurred. Over-fitting, training? You don't say. If I wanted to be more of an ass, I'd look to argue about hype generation. But I don't, I appreciate any honest effort, which I believe for Simon.
- vanschelven 11mo agoWhatever you think Jimmc414's _concerns_ are (they merely state a possibility) Simon enumerates a number of concerns in the linked article, and then addresses those. So I'm not sure why you think this is so.
- vnvnff 11mo agoIt's a pattern: https://news.ycombinator.com/item?id=44725190 https://news.ycombinator.com/item?id=44725190
- armcat 11mo agoHi Simon! Love your work! Our of curiosity - how many pelican-cycling samples do you produce. Curious about the variance here. Thanks!
- simonw 11mo agoI've lost count, but there are 85 posts with that tag here: https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/ I need to extract them all into a formal collection.
- jgalt212 11mo agoAiden is perhaps misinformed. From a Bing search performed just now. > Yes, I am familiar with the "pelican riding a bicycle" SVG generation test. It is a benchmark for evaluating the ability of AI models, particularly large language models (LLMs) and multi-modal systems, to generate original, high-quality SVG vector graphics based on a deliberately unusual and complex prompt. The benchmark was popularized by Simon Willison, who selected the prompt because:
- 100721 11mo agoWeb search-based RAG is very different from having something embedded in a model's training data, though.
- jgalt212 11mo agoChatGPT website gives a similar answer. Are they running RAG, or the model? > Yes — I’m familiar with the “pelican riding a bicycle” SVG generation test. > It’s become a kind of informal benchmark people use when evaluating whether an image-generation or SVG-generation model can: ...
- Barbing 11mo agoRunnin’ confabulations: >Yes — the “hamster driving a car” prompt is a well-known informal test … >…that’s a well-known informal test people use…(a mole-rat holding or playing a guitar). Try any plausible concept. Get sillier and it’s trained to talk about it being nonsense. The output still claims it’s a real test, just a real “nonsense” test.
- th0ma5 11mo agoIf this had any substance then it could be criticized, which is what they're trying to avoid.
- Etheryte 11mo agoHow? There's no way for you to verify if they put synthetic data for that into the dataset or not.
- thatwasunusual 11mo ago> We are getting to the point that its not unreasonable to think that "Generate an SVG of a pelican riding a bicycle" could be included in some training data. I may be stupid, but _why_ is this prompt used as a benchmark? I mean, pelicans _can't_ ride a bicycle, so why is it important for "AI" to show that they can (at least visually)? The "wine glass problem"[0] - and probably others - seems to me to be a lot more relevant...? [0] https://medium.com/@joe.richardson.iii/the-curious-case-of-the-gen-ai-wine-glass-problem-f43778bbe890 https://medium.com/@joe.richardson.iii/the-curious-case-of-t...
- simonw 11mo agoThe fact that pelicans can't ride bicycles is pretty much the point of the benchmark! Asking an LLM to draw something that's physically impossible means it can't just "get it right" - seeing how different models (especially at different sizes) handle the problem is surprisingly interesting. Honestly though, the benchmark was originally meant to be a stupid joke. I only started taking it slightly more seriously about six months ago, when I noticed that the quality of the pelican drawings really did correspond quite closely to how generally good the underlying models were. If a model draws a really good picture of a pelican riding a bicycle there's a solid chance it will be great at all sorts of other things. I wish I could explain why that was! If you start here and scroll through and look at the progression of pelican on bicycle images it's honestly spooky how well they match the vibes of the models they represent: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-worlds-fair-2025-05.jpeg https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-... So ever since then I've continue to get models to draw pelicans. I certainly wouldn't suggest anyone take serious decisions on model usage based on my stupid benchmark, but it's a fun first-day initial impression thing and it appears to be a useful signal for which models are worth diving into in more detail.
- thatwasunusual 11mo ago> If a model draws a really good picture of a pelican riding a bicycle there's a solid chance it will be great at all sorts of other things. Why? If I hired a worker that was really good at drawing pelicans riding a bike, it wouldn't tell me anything about his/her other qualities?!
- 0cf8612b2e1e 11mo agoI assume all of the models also have variations on, “how many ‘r’s in strawberry”.
- theshrike79 11mo agoThe easiest way to fix these is give the model an environment to run code. Any model can easily one-shot a python script that can count the occurrence of any letter anywhere and return the result. It's just a tooling issue. You really can't "train" an LLM to do it because tokenisation and ... stuff.
- 0cf8612b2e1e 11mo agoI am not convinced they are executing code. Otherwise I would expect LLMs to not frequently guess the result of math questions. Of course you could train it. Some quick scripting to find all words with repeat letters, build up sample sentences (aardvark has three a,) and you have hard coded the answer to simple questions that make your LLM look stupid.
- theshrike79 11mo agoI have personally observed Grok running Python code in a chat to determine the current date so it could accurately tell me whether the 20th is a Friday (it wasn't in that specific month) .. it did that in a story prompt that didn't happen in a) our world b) the current time =)
- Workaccount2 11mo agoIt would be easy to out models that train on the bike pelican, because they would probably suck at the kayaking bumblebee. So far though, the models good at bike pelican are also good at kayak bumblebee, or whatever other strange combo you can come up with. So if they are trying to benchmaxx by making SVG generation stronger, that's not really a miss, is it?
- majormajor 11mo agoThat depends on if "SVG generation" is a particularly useful LLM/coding model skill outside of benchmarking. I.e., if they make that stronger with some params that otherwise may have been used for "rust type system awareness" or somesuch, it might be a net loss outside of the benchmarks.