6 ms·
The bicycle frame is a bit wonky but the pelican itself is great: https://gist.github.com/simonw/a6806ce41b4c721e240a4548ecdbe216?permalink_comment_id=5972110#g
by simonw 9mo ago
The bicycle frame is a bit wonky but the pelican itself is great: https://gist.github.com/simonw/a6806ce41b4c721e240a4548ecdbe216?permalink_comment_id=5972110#gistcomment-5972110 https://gist.github.com/simonw/a6806ce41b4c721e240a4548ecdbe...
- stkai 9mo agoWould love to find out they're overfitting for pelican drawings.
- andy_ppp 9mo agoYes, Racoon on a unicycle? Magpie on a pedalo?
- deleted 9mo ago[deleted]
- throw310822 9mo agoCorrect horse battery staple: https://claude.ai/public/artifacts/14a23d7f-8a10-4cde-89fe-05f108da236d https://claude.ai/public/artifacts/14a23d7f-8a10-4cde-89fe-0...
- ta988 9mo agono staple?
- iwontberude 9mo agoit looks like a bodge wire
- Schlagbohrer 9mo agoThat is the nastiest, ugliest horse ever
- HappMacDonald 9mo agowait, how do you know my pw?
- _kb 9mo agoPlatypus on a penny farthing.
- fragmede 9mo agoThe estimation I did 4 months ago: > there are approximately 200k common nouns in English, and then we square that, we get 40 billion combinations. At one second per, that's ~1200 years, but then if we parallelize it on a supercomputer that can do 100,000 per second that would only take 3 days. Given that ChatGPT was trained on all of the Internet and every book written, I'm not sure that still seems infeasible. https://news.ycombinator.com/item?id=45455786 https://news.ycombinator.com/item?id=45455786
- eli 9mo agoHow would you generate a picture of Noun + Noun in the first place in order to train the LLM with what it would look like? What's happening during that 1 estimated second?
- Terretta 9mo agoThis is why everyone trains their LLM on another LLM. It's all about the pelicans.
- metalliqaz 9mo agoits pelicans all the way down
- fragmede 9mo agoUse any of the image generation models (eg Nanobanana, Midjourney, or ChatGPT) to generate a picture of a noun on a noun. Simonw's test is to have a Language (text) model generate a Scalar Vector Graphic, which the language model has to do by writing curves and colors, like draw a spline from point 150,100 to 200,300 of type cubic, using width 20, color orange. In that hypothetical second is freaking fascinating. It's a denoising algorithm, and then a bunch of linear algebra, and out pops a picture of a pelican on a bicycle. Stable diffusion does this quite handily. https://stablediffusionweb.com/image/6520628-pelican-bicycle-ride https://stablediffusionweb.com/image/6520628-pelican-bicycle...
- AnimalMuppet 9mo agoBut you need to also include the number of prepositions. "A pelican on a bicycle" is not at all the same as "a pelican inside a bicycle". There are estimated to be 100 or so prepositions in English. That gets you to 4 trillion combinations.
- theanonymousone 9mo agoEven if not intentionally, it is probably leaking into training sets.
- fdeage 9mo agoOpenAI claims not to: https://x.com/aidan_mclau/status/1986255202132042164 https://x.com/aidan_mclau/status/1986255202132042164
- mattacular 9mo agoThat settles it
- DetroitThrow 9mo agoThe ears on top are a cute touch
- eaf7e281 9mo agoThere's no way they actually work on training this.
- KeplerBoy 9mo agoThere is no way they are not training on this.
- collinmanderson 9mo agoI suspect they have generic SVG drawing that they focus on.
- deleted 9mo ago[deleted]
- margalabargala 9mo agoI suspect they're training on this. I asked Opus 4.6 for a pelican riding a recumbent bicycle and got this. https://i.imgur.com/UvlEBs8.png https://i.imgur.com/UvlEBs8.png
- mrandish 9mo agoInteresting that it seems better. Maybe something about adding a highly specific yet unusual qualifier focusing attention?
- WarmWash 9mo agoIt would be way way better if they were benchmaxxing this. The pelican in the image (both images) has arms. Pelicans don't have arms, and a pelican riding a bike would use it's wings.
- seanhunter 9mo agoPelicans don’t ride bikes. You can’t have scruples about whether or not the image of a pelican riding a bike has arms.
- ares623 9mo agoCan it draw a different bird on a bike?
- simonw 9mo agoHere's a kākāpō riding a bicycle instead: https://gist.github.com/simonw/19574e1c6c61fc2456ee413a24528488?permalink_comment_id=5972119#gistcomment-5972119 https://gist.github.com/simonw/19574e1c6c61fc2456ee413a24528... I don't think it quite captures their majesty: https://en.wikipedia.org/wiki/K%C4%81k%C4%81p%C5%8D https://en.wikipedia.org/wiki/K%C4%81k%C4%81p%C5%8D
- zahlman 9mo agoNow that I've looked it all up, I feel like that's much more accurate to a real kākāpō than the pelican is to a real pelican. It's almost as if it thinks a pelican is just a white flamingo with a different beak.
- vikkymelani 9mo ago[dead]
- nubg 9mo agoWhat about the Pelo2 benchmark? (the gray bird that is not gray)
- hoeoek 9mo agoThis really is my favorite benchmark
- einrealist 9mo agoThey trained for it. That's the +0.1!
- 7777777phil 9mo agobest pelican so far would you say? Or where does it rank in the pelican benchmark?
- mrandish 9mo agoIn other words, is it a pelican or a pelican't?
- canadiantim 9mo agoYou’ve been sitting on that pun just waiting for it to take flight
- athrowaway3z 9mo agoThis benchmark inspired me to have codex/claude build a DnD battlemap tool with svg's. They got surprisingly far, but i did need to iterate a few times to have it build tools that would check for things like; dont put walls on roads or water. What I think might be the next obstacle is self-knowledge. The new agents seem to have picked up ever more vocabulary about their context and compaction, etc. As a next benchmark you could try having 1 agent and tell it to use a coding agent (via tmux) to build you a pelican.
- copilot_king_2 9mo agoI'm firing all of my developers this afternoon.
- RGamma 9mo agoOpus 6 will fire you instead for being too slow with the ideas.
- insane_dreamer 9mo agoToo late. You’ve already been fired by a moltbot agent from your PHB.
- behnamoh 9mo ago[flagged]
- smokel 9mo agoI'll bite. The benchmark is actually pretty good. It shows in an extremely comprehensible way how far LLMs have come. Someone not in the know has a hard time understanding what 65.4% means on "Terminal-Bench 2.0". Comparing some crappy pelicans on bicycles is a lot easier.
- blibble 9mo agoit ceases to be a useful benchmark of general ability when you post it publicly for them to train against
- quinnjh 9mo agothe field is advancing so fast it's hard to do real science as their will be a new SOTA by the time you're ready to publish results. i think this is a combination of that and people having a laugh. Would you mind sharing which benchmarks you think are useful measures for multimodal reasoning?
- techpression 9mo agoA benchmark only tests what the benchmark is doing, the goal is to make that task correlate with actually valuable things. Graphic benchmarks is a good example, extremely hard to know what you will get in a game by looking at 3D Mark scores, it varies by a lot. Making a SVG of a single thing doesn’t help much unless that applies to all SVG tasks.
- gcanyon 9mo agoOne aspect of this is that apparently most people can't draw a bicycle much better than this: they get the elements of the frame wrong, mess up the geometry, etc.
- gnatolf 9mo agoAbsolutely. A technically correct bike is very hard to draw in SVG without going overboard in details
- falloutx 9mo agoIts not. There are thousands of examples on the internet but good SVG sites do have monetary blocks. https://www.freepik.com/free-photos-vectors/bicycle-svg https://www.freepik.com/free-photos-vectors/bicycle-svg
- jefftk 9mo agoSeveral of those have incorrect frames: https://www.freepik.com/free-vector/cyclist_23714264.htm https://www.freepik.com/free-vector/cyclist_23714264.htm https://www.freepik.com/premium-vector/bicycle-icon-black-line-bike-two-wheel-vehicle_24065169.htm https://www.freepik.com/premium-vector/bicycle-icon-black-li... Or missing/broken pedals: https://www.freepik.com/premium-vector/bicycle-silhouette-icon-white-background_390337074.htm https://www.freepik.com/premium-vector/bicycle-silhouette-ic... https://www.freepik.com/premium-vector/bicycle-silhouette-vector-illustration_396261558.htm https://www.freepik.com/premium-vector/bicycle-silhouette-ve... http://freepik.com/premium-vector/bicycle-silhouette-vector-illustration_396261558.htm http://freepik.com/premium-vector/bicycle-silhouette-vector-...
- gnatolf 9mo agoFrom smaller to larger nitpick, there's basically something wrong with all of the first 15 or so of these drawings. Thanks for agreeing :)
- RussianCow 9mo agoI'm not positive I could draw a technically correct bike with pen and paper (without a reference), let alone with SVG!
- bityard 9mo agoWell, the clouds are upside-down, so I don't think I can give it a pass.
- nine_k 9mo agoI suppose the pelican must be now specifically trained for, since it's a well-known benchmark.
- risyachka 9mo agoPretty sure at this point they train it on pelicans
- 6thbit 9mo agodo you have a gif? i need an evolving pelican gif
- Kye 9mo agoA pelican GIF in a Pelican(TM) MP4 container.
- franze 9mo agohere the animated version https://claude.ai/public/artifacts/3db12520-eaea-4769-82be-7e4cda778aba https://claude.ai/public/artifacts/3db12520-eaea-4769-82be-7...
- gryfft 9mo agoThat's hilarious. It's so close!
- beemboy 9mo agoIsn't there a point at which it trains itself on these various outputs, or someone somewhere draws one and feeds it into the model so as to pass this benchmark?
- zahlman 9mo agoDo you find that word choices like "generate" (as opposed to "create", "author", "write" etc.) influence the model's success? Also, is it bad that I almost immediately noticed that both of the pelican's legs are on the same side of the bicycle, but I had to look up an image on Wikipedia to confirm that they shouldn't have long necks? Also, have you tried iterating prompts on this test to see if you can get more realistic results? (How much does it help to make them look up reference images first?)
- simonw 9mo agoI've stuck with "Generate an SVG of a pelican riding a bicycle" because it's the same prompt I've been using for over a year now and I want results that are sort-of comparable to each other. I think when I first tried this I iterated a few times to get to something that reliably output SVG, but honestly I didn't keep the notes I should ahve.
- iujasdkjfasf 9mo ago[dead]
- fullstackchris 9mo ago[flagged]
- dang 9mo agoPersonal attacks are not allowed on HN. No more of this, please.
- etwigg 9mo agoIf we do get paperclipped, I hope it is of the "cycling pelican" variety. Thanks for your important contribution to alignment Simon!
- MaysonL 9mo agoExcept for both its legs being on the same side of the bike.