10 ms·
The last six months in LLMs, illustrated by pelicans on bicycles
- neepi 1y agoMy only take home is they are all terrible and I should hire a professional.
- dist-epoch 1y agoMost of them are text-only models. Like asking a person born blind to draw a pelican, based on what they heard it looks like.
- neepi 1y agoThat seems to be a completely inappropriate use case? I would not hire a blind artist or a deaf musician.
- dist-epoch 1y agoThe point is about exploring the capabilities of the model. Like asking you to draw a 2D projection of 4D sphere intersected with a 4D torus or something.
- deleted 1y ago[deleted]
- kevindamm 1y agoYeah, I suppose it is similar.. I don't know their diameters, rotations, nor the distance between their centers, nor which two dimensions, so I would have to guess a lot about what you meant.
- namibj 1y agoIt's a proxy for abstract designing, like writing software or designing in a parametric CAD. Most the non-math design work of applied engineering AFAIK falls under the umbrella that's tested with the pelican riding the bicycle. You have to make a mental model and then turn it into applicable instructions. Program code/SVG markup/parametric CAD instructions don't really differ in that aspect.
- neepi 1y agoI would not assume that this methodology applies to applied engineering, as a former actual real tangible meat space engineer. Things are a little nuanced and the nuances come from a combination of communication and experience, neither of which any LLM has any insight into at all. It's not out there on the internet to train it with and it's not even easy to put it into abstract terms which can be used as training data. And engineering itself in isolation doesn't exist - there is a whole world around it. Ergo no you can't just say throw a bicycle into an LLM and a parametric model drops out into solidworks, then a machine makes it. And everyone buys it. That is the hope really isn't it? You end up with a useless shitty bike with a shit pelican on it. The biggest problem we have in the LLM space is the fact that no one really knows any of the proposed use cases enough and neither does anyone being told that it works for the use cases.
- dist-epoch 1y agohttps://www.solidworks.com/lp/evolve-your-design-workflows-ai https://www.solidworks.com/lp/evolve-your-design-workflows-a...
- neepi 1y agoYeah good luck with that. Seriously.
- rjsw 1y agoI don't think any of that matters, CEOs will decide to use it anyway.
- neepi 1y agoThis is sad but true.
- __alexs 1y agoI guess the idea is that by asking the model to do something that is inherently hard for it we might learn something about the baseline smartness of each model which could be considered a predictor for performance at other tasks too.
- dmd 1y agoSorry, Beethoven, you just don’t seem to be a match for our org. Best of luck on your search! You too, Monet. Scram.
- simonw 1y agoYeah, that's part of the point of this. Getting a state of the art text generating LLM to generate SVG illustrations is an inappropriate application of them. It's a fun way to deflate the hype. Sure, your new LLM may have cost XX million to train and beat all the others on the benchmarks, but when you ask it to draw a pelican on a bicycle it still outputs total junk.
- dist-epoch 1y agotried starting from an image: https://chatgpt.com/share/684582a0-03cc-8006-b5b5-de51e5cd89b0 https://chatgpt.com/share/684582a0-03cc-8006-b5b5-de51e5cd89... lol: https://gemini.google.com/share/4d1746a234a8 https://gemini.google.com/share/4d1746a234a8
- wongogue 1y agoEven Beethoven?
- matkoniecz 1y agoit depends on quality you need and your budget
- neepi 1y agoAh yes the race to the bottom argument.
- ben_w 1y agoWhen I was at university, they got some people from industry to talk to us all about our CVs and how to do interviews. My CV had a stupid cliché, "committed to quality", which they correctly picked up on — "What do you mean?" one of them asked me, directly. I thought this meant I was focussed on being the best. He didn't like this answer. His example, blurred by 20 years of my imperfect human memory, was to ask me which is better: a Porsche, or a go-kart. Now, obviously (or I wouldn't be saying this), Porsche was a trick answer. Less obviously is that both were trick answers, because their point was that the question was under-specified — quality is the match between the product and what the user actually wants, so if the user is a 10 year old who physically isn't big enough to sit in a real car's driver's seat and just wants to rush down a hill or along a track, none of "quality" stuff that makes a Porsche a Porsche is of any relevance at all, but what does matter is the stuff that makes a go-kart into a go-kart… one of which is the affordability. LLMs are go-karts of the mind. Sometimes that's all you need.
- neepi 1y agoI disagree. Quality depends on your market position and what you are bringing to the market. Thus I would start with market conditions and work back to quality. If you can't reach your standards in the market then you shouldn't enter it. And if your standards are poor, you should be ashamed. Go kart or porsche is irrelevant.
- ben_w 1y ago> Quality depends on your market position and what you are bringing to the market. That's the point. The market for go-karts does not support Porche. If you bring a Porche sales team to a go-kart race, nobody will be interested. Porche doesn't care about this market. It goes both ways: this market doesn't care about Porche, either.
- keiferski 1y agoAs the other guy said, these are text models. If you want to make images use something like Midjourney. Promoting a pelican riding a bicycle makes a decent image there.
- keiferski 1y ago* Prompting
- GaggiX 1y agoAn expert at writing SVGs?
- jug 1y agoBefore that, you might ask ChatGPT to create a vector image of a pelican riding a bicycle and then running the output through a PNG to SVG converter... Result: https://www.dropbox.com/scl/fi/8b03yu5v58w0o5he1zayh/pelican-on-bicycle.svg?rlkey=3evrjz7fiaa940842zxinanyl&st=rfbc5z9w&dl=0 https://www.dropbox.com/scl/fi/8b03yu5v58w0o5he1zayh/pelican... These are tough benchmarks to trial reasoning by having it _write_ an SVG file by hand and understanding how it's to be written to achieve this. Even a professional would struggle with that! It's _not_ a benchmark to give an AI the best tools to actually do this.
- YuccaGloriosa 1y agoI think you made an error there png is a bitmap format
- sethaurus 1y agoYou've misunderstood. The parent was making a specific point — if you want an SVG of a penguin, the easiest way to AI-generate it is to get an image generator to create a (vector-styled) bitmap, then auto-vectorize it to SVG. But the point of this benchmark is that it's asking models to create an SVG the hard way, by writing its code directly.
- spaceman_2020 1y agoMy only take home is that a spanner can work as a hammer, but you probably should just get a hammer
- vunderba 1y agoThis test isn't really about the quality of the image itself (multimodals like gpt-image-1 or even standard diffusion models would be far superior) - it's about following a spec that describes how to draw. A similar test would be if you asked for the pelican on a bicycle through a series of LOGO instructions.
- joshstrange 1y agoI really enjoy Simon’s work in this space. I’ve read almost every blog post they’ve posted on this and I love seeing them poke and prod the models to see what pops out. The CLI tools are all very easy to use and complement each other nicely all without trying to do too much by themselves. And at the end of the day, it’s just so much fun to see someone else having so much fun. He’s like a kid in a candy store and that excitement is contagious. After reading every one of his blog posts, I’m inspired to go play with LLMs in some new and interesting way. Thank you Simon!
- blackhaj7 1y agoSame sentiment!
- dotemacs 1y agoThe same here. Because of him, I installed a RSS reader so that I don't miss any of his posts. And I know that he shares the same ones across Twitter, Mastodon & Bsky...
- alanmoraes 1y agoI also like what he writes and the way he does it.
- nathan_phoenix 1y agoMy biggest gripe is that he's comparing probabilistic models (LLMs) by a single sample. You wouldn't compare different random number generators by taking one sample from each and then concluding that generator 5 generates the highest numbers... Would be nicer to run the comparison with 10 images (or more) for each LLM and then average.
- puttycat 1y agoYou are right, but the companies making these models invest a lot of effort in marketing them as anything but probabilistic, i.e. making people think that these models work discretely like humans. In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. In any case, even if a model is probabilistic, if it had correctly learned the relevant knowledge you'd expect the output to be perfect because it would serve to lower the model's loss. These outputs clearly indicate flawed knowledge.
- ben_w 1y ago> In that case we'd expect a human with perfect drawing skills and perfect knowledge about bikes and birds to output such a simple drawing correctly 100% of the time. Look upon these works, ye mighty, and despair: https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/
- jodrellblank 1y agoYou claim those are drawn by people with "perfect knowledge about bikes" and "perfect drawing skills"?
- ben_w 1y agoMore that "these models work … like humans" (discretely or otherwise) does not imply the quotation. Most humans do not have perfect drawing skills and perfect knowledge about bikes and birds, they do not output such a simple drawing correctly 100% of the time. "Average human" is a much lower bar than most people want to believe, mainly because most of us are average on most skills, and also overestimate our own competence — the modal human has just a handful of things they're good at, and one of those is the language they use, another is their day job. Most of us can't draw, and demonstrably can't remember (or figure out from first principles) how a bike works. But this also applies to "smart" subsets of the population: physicists have https://xkcd.com/793/ https://xkcd.com/793/, and there's this famous rocket scientist who weighed in on rescuing kids from a flooded cave, they come up with some nonsense about a submarine.
- anon373839 1y agoEnjoyable write-up, but why is Qwen 3 conspicuously absent? It was a really strong release, especially the fine-grained MoE which is unlike anything that’s come before (in terms of capability and speed on consumer hardware).
- Maxious 1y agoCut for time - qwen3 was pelican tested too https://simonwillison.net/2025/Apr/29/qwen-3/ https://simonwillison.net/2025/Apr/29/qwen-3/
- simonw 1y agoOmitting Qwen 3 is my great regret about this talk. Honestly I only realized I had missed it after I had delivered the talk! It's one of my favorite local models right now, I'm not sure how I missed it when I was reviewing my highlights of the last six months.
- qwertytyyuu 1y agohttps://imgur.com/a/mzZ77xI https://imgur.com/a/mzZ77xI here are a few i tried the models, looks like the newer vesion of gemini is another improvement?
- puttycat 1y agoThe bicycle are still very far from actual ones.
- JimDabell 1y agoSee also: The recent history of AI in 32 otters https://www.oneusefulthing.org/p/the-recent-history-of-ai-in-32-otters https://www.oneusefulthing.org/p/the-recent-history-of-ai-in...
- pbhjpbhj 1y agoThat is otterly fantastic. The post there shows the breadth too - both otters generated via text representations (in TikZ) and by image generators. The video at the end, wow (and funny too). Thanks for sharing.
- bravesoul2 1y agoIs there a good model (any architecture) for vector graphics out of interest?
- simonw 1y agoI was impressed by Recraft v3, which gave me an editable vector illustration with different layers - https://simonwillison.net/2024/Nov/15/recraft-v3/ https://simonwillison.net/2024/Nov/15/recraft-v3/ - but as I understand it that one is actually still a raster image generator with a separate step to convert to vector at the end.
- bravesoul2 1y agoNow that is a pelican on a bicycle! Thanks
- dirtyhippiefree 1y agoHere’s the spot where we see who’s TL;DR… > Claude 4 will rat you out to the feds! >If you expose it to evidence of malfeasance in your company, and you tell it it should act ethically, and you give it the ability to send email, it’ll rat you out.
- ben_w 1y agoI'd say that's too short. > But it’s not just Claude. Theo Browne put together a new benchmark called SnitchBench, inspired by the Claude 4 System Card. > It turns out nearly all of the models do the same thing.
- dirtyhippiefree 1y agoI totally agree, but I needed you to post the other half because of TL;DR…
- yubblegum 1y agoI was looking at that and wondering about swatting via LLMs by malicious users.
- gscott 1y agoI am interested in this ratting you out thing. At some point you have a video feed into AI from a Jarvis like headset device, you walking down the street and cross the street in the middle not at a sidewalk... does it rat you out? Does it make a list of every crime no matter how small? Or just the big ones?
- atxtechbro 1y agoThank you, Simon! I really enjoyed your PyBay 2023 talk on embeddings and this is great too! I like the personalized benchmark. Hopefully the big LLM providers don't start gaming the pelican index!
- deadbabe 1y agoAs a control, he should go on fiver and have a human generate a pelican riding a bicycle, just to see what the eventual goal is.
- gus_massa 1y agoSomeone did this. Look at this sibling comment by ben_w https://news.ycombinator.com/item?id=44216284 https://news.ycombinator.com/item?id=44216284 about an old similar project.
- zahlman 1y ago> back in 2009 I began pestering friends and random strangers. I would walk up to them with a pen and a sheet of paper asking that they immediately draw me a men’s bicycle, by heart. Someone commissioned to draw a bicycle on Fiverr would not have to rely on memory of what it should look like. It would take barely any time to just look up a reference.
- franze 1y agoHere Claude Opus Extended Thinking https://claude.ai/public/artifacts/707c2459-05a1-4a32-b393-c61192b9e55e https://claude.ai/public/artifacts/707c2459-05a1-4a32-b393-c...
- big_hacker 1y agoHonestly the metric which increased the most is the marketing and astroturfing budget of the major players (OpenAI, Anthropic, Google and Deepseek). Say what you want about Facebook but at least they released their flagship model fully open.
- mdaniel 1y ago> model fully open. uh-huh https://www.llama.com/llama4/license/ https://www.llama.com/llama4/license/
- spaceman_2020 1y agoI don’t know what secret sauce Anthropic has, but in real world use, Sonnet is somehow still the best model around. Better than Opus and Gemini Pro
- diggan 1y agoStatements like these are useless without sharing exactly all the models you've tried. Sonnet beats O1 Pro Mode for example? Not in my experience, but I haven't tried the latest Sonnet versions, only the one before, so wouldn't claim O1 Pro Mode beats everything out there. Besides, it's so heavily context-dependent that you really need your own private benchmarks to make head or tails out of this whole thing.
- wohoef 1y agoQuite a detailed image using claude sonnet 4: https://ibb.co/39RbRm5W https://ibb.co/39RbRm5W
- landgenoot 1y agoIf you would give a human the SVG documentation and ask to write an SVG, I think the results would be quite similar.
- diggan 1y agoLets give it a try, if you're willing to be the experiment subject :) The prompt is "Generate an SVG of a pelican riding a bicycle" and you're supposed to write it by hand, so no graphical editor. The specification is here: https://www.w3.org/TR/SVG2/ https://www.w3.org/TR/SVG2/ I'm fairly certain I'd lose interest in getting it right before I got something better than most of those.
- mormegil 1y agoDid the testing prompt for LLMs include a clause forbidding the use of any tools? If not, why are you adding it here?
- diggan 1y agoThe models that are being put under the "Pelican" testing don't use a GUI to create SVGs (either via "tools" or anything else), they're all Text Generation models so they exclusively use text for creating the graphics. There are 31 posts listed under "pelican-riding-a-bicycle" in case you wanna inspect the methodology even closer: https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/
- simonw 1y agoThe way I run the pelican on a bicycle benchmark is to use this exact prompt: Generate an SVG of a pelican riding a bicycle And execute it via the model's API with all default settings, not via their user-facing interface. Currently none of the model APIs enable tools unless you ask them to, so this method excludes the use of additional tools.
- zahlman 1y ago> The colors use traditional bicycle brown (#8B4513) and a classic blue for the pelican (#4169E1) with gold accents for the beak (#FFD700). The output pelican is indeed blue. I can't fathom where the idea that this is "classic", or suitable for a pelican, could have come from.
- adrian17 1y ago> This was one of the most successful product launches of all time. They signed up 100 million new user accounts in a week! They had a single hour where they signed up a million new accounts, as this thing kept on going viral again and again and again. Awkwardly, I never heard of it until now. I was aware that at some point they added ability to generate images to the app, but I never realized it was a major thing (plus I already had an offline stable diffusion app on my phone, so it felt less of an upgrade to me personally). With so much AI news each week, feels like unless you're really invested in the space, it's almost impossible to not accidentally miss or dismiss some big release.
- azinman2 1y agoExcept this went very mainstream. Lots of turn myself into a muppet, what is the human equivalent for my dog, etc. TikTok is all over this. It really is incredible.
- thierrydamiba 1y agoThe big trend was around the ghiblification of images. Those images were everywhere for a period of time.
- Jedd 1y agoYeah, but so were the bored ape NFTs - none of these ephemeral fads are any indication of quality, longevity, legitimacy, or interest.
- mrkurt 1y agoIf we try really hard, I think we can make an exhaustive list of what viral fads on the internet are not. You made a small start. none of these ephemeral fads are any indication of quality, longevity, legitimacy, interest, substance, endurance, prestige, relevance, credibility, allure, staying-power, refinement, or depth.
- 1y ago
- nowayno583 1y agoThat was a very fun recap, thanks for sharing. It's easy to forget how much better these things have gotten. And this was in just six months! Crazy!
- mromanuk 1y agoThe last animation is hilarious, represents very well the AI Hype cycle vs reality.
- bredren 1y agoGreat writeup. This measure of LLM capability could be extended by taking it into the 3D domain. That is, having the model write Python code for Blender, then running blender in headless mode behind an API. The talk hints at this but one shot prompting likely won’t be a broad enough measurement of capability by this time next year. (Or perhaps now, even) So the test could also include an agentic portion that includes consultation of the latest blender documentation or even use of a search engine for blog entries detailing syntax and technique. For multimodal input processing, it could take into account a particular photo of a pelican as the test subject. For usability, the objects can be converted to iOS’s native 3d format that can be viewed in mobile safari. I built this workflow, including a service for blender as an initial test of what was possible in October of 2022. It took post processing for common syntax errors back then but id imagine the newer LLMs would make those mistakes less often now.
- jfengel 1y agoIt's not so great at bicycles, either. None of those are close to rideable. But bicycles are famously hard for artists as well. Cyclists can identify all of the parts, but if you don't ride a lot it can be surprisingly difficult to get all of the major bits of geometry right.
- mattlondon 1y agoMost recent Gemini 2.5 one looks pretty good. Certainly rideable.
- pier25 1y agoDefinitely getting better but even the best result is not very impressive.
- nine_k 1y agoAm I the only one who can't but see these attempts much like attempts of a kid learning to draw?
- Ygg2 1y agoYes. Kids don't draw that good of a line at the start. Here is better example of start https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTfTfAA45vJOFzPUt8UrxlfRXGUj0PfqIpfOA&s https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTfTfAA...
- nine_k 1y agoHave you tried giving a kid a vector-drawing tool? I did that to my daughter when she was not even 6 years old. The results were somehow similar: https://photos.app.goo.gl/XSLnTEUkmtW2n7cX8 https://photos.app.goo.gl/XSLnTEUkmtW2n7cX8 (Now she's much better, but prefers raster tools, e.g. https://www.deviantart.com/sofiac9/art/Ivy-with-riding-gear-1166659744 https://www.deviantart.com/sofiac9/art/Ivy-with-riding-gear-...)
- djherbis 1y agoKaggle recently ran a competition to do just this (draw SVGs from prompts, using fairly small models under the hood). The top results (click on the top Solutions) were pretty impressive: https://www.kaggle.com/competitions/drawing-with-llms/leaderboard https://www.kaggle.com/competitions/drawing-with-llms/leader...
- username223 1y agoInteresting timeline, though the most relevant part was at the end, where Simon mentions that Google is now aware of the "pelican on bicycle" question, so it is no longer useful as a benchmark. FWIW, many things outside of the training data will pants these models. I just tried this query, which probably has no examples online, and Gemini gave me the standard puzzle answer, which is wrong: "Say I have a wolf, a goat, and some cabbage, and I want to get them across a river. The wolf will eat the goat if they're left alone, which is bad. The goat will eat some cabbage, and will starve otherwise. How do I get them all across the river in the fewest trips?" A child would pick up that you have plenty of cabbage, but can't leave the goat without it, lest it starve. Also, there's no mention of boat capacity, so you could just bring them all over at once. Useful? Sometimes. Intelligent? No.
- zahlman 1y ago> If you lost interest in local models—like I did eight months ago—it’s worth paying attention to them again. They’ve got good now! > As a power user of these tools, I want to stay in complete control of what the inputs are. Features like ChatGPT memory are taking that control away from me. You reap what you sow.... > I already have a tool I built called shot-scraper, a CLI app that lets me take screenshots of web pages and save them as images. I had Claude build me a web page that accepts ?left= and ?right= parameters pointing to image URLs and then embeds them side-by-side on a page. Then I could take screenshots of those two images side-by-side. I generated one of those for every possible match-up of my 34 pelican pictures—560 matches in total. Surely it would have been easier to use a local tool like ImageMagick? You could even have the AI write a Bash script for you. > ... but prompt injection is still a thing. ...Why wouldn't it always be? There's no quoting or escaping mechanism that's actually out-of-band. > There’s this thing I’m calling the lethal trifecta, which is when you have an AI system that has access to private data, and potential exposure to malicious instructions—so other people can trick it into doing things... and there’s a mechanism to exfiltrate stuff. People in 2025 actually need to be told this. Franklin missed the mark - people today will trip over themselves to give up both their security and their liberty for mere convenience.
- simonw 1y agoI had the LLM write a bash script for me that used my https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ tool - on the basis that it was a neat opportunity to demonstrate another of my own projects. And honestly, even with LLM assistance getting Image Magick to output a 1200x600 image with two SVGs next to each other that are correctly resized to fill their half of the image sounds pretty tricky. Probably easier (for Claude) to achieve with HTML and CSS.
- zahlman 1y ago> And honestly, even with LLM assistance getting Image Magick to output a 1200x600 image with two SVGs next to each other that are correctly resized to fill their half of the image sounds pretty tricky. FWIW, the next project I want to look at after my current two, is a command-line tool to make this sort of thing easier. Likely featuring some sort of Lisp-like DSL to describe what to do with the input images.
- Joker_vD 1y ago> most people find it difficult to remember the exact orientation of the frame. Isn't it Δ∇Λ welded together? The bottom left and right vertices are where the wheels are attached to, the middle bottom point is where the big gear with the pedals is. The lambda is for the front wheel because you wouldn't be able to turn it if it was attached to a delta. Right? I guess having my first bicycle be a cheap Soviet-era produced one paid off: I spent loads of time fidgeting with the chain tension, and pulling the chain back onto the gears, so I guess I had to stare at the frame way too much to forget even by today the way it looks.
- pbronez 1y agoThere are a lot of structural details that people tend to gloss over. This was illustrated by an Italian art project: https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/ > back in 2009 I began pestering friends and random strangers. I would walk up to them with a pen and a sheet of paper asking that they immediately draw me a men’s bicycle, by heart. Soon I found out that when confronted with this odd request most people have a very hard time remembering exactly how a bike is made.
- joshuajooste05 1y agoDoes anyone have any thoughts on privacy/safety regarding what he said about GPT memory. I had heard of prompt injection already. But, this seems different, completely out of humans control. Like even when you consider web search functionality, he is actually right, more and more, users are losing control over context. Is this dangerous atm? Do you think it will become more dangerous in the future when we chuck even more data into context?
- ActorNightly 1y agoSort of. The thing is with agentic models, you are basically entering probability space where it can do real actions in the form of http requests if the statistical output leads it to it.
- threeseed 1y agoI've had Cursor/Claude try to call rm -rf on my entire User directory before. The issue is that LLMs have no ability to organise their memory by importance. Especially as the context size gets larger. So when they are using tools they will become more dangerous over time.
- isx726552 1y ago> I’ve been feeling pretty good about my benchmark! It should stay useful for a long time... provided none of the big AI labs catch on. > And then I saw this in the Google I/O keynote a few weeks ago, in a blink and you’ll miss it moment! There’s a pelican riding a bicycle! They’re on to me. I’m going to have to switch to something else. Yeah this touches on an issue that makes it very difficult to have a discussion in public about AI capabilities. Any specific test you talk about, no matter how small … if the big companies get wind of it, it will be RLHF’d away, sometimes to the point of absurdity. Just refer to the old “count the ‘r’s in strawberry” canard for one example.
- Choco31415 1y agoJust tried that canard on GPT-4o and it failed: "The word "strawberry" contains 2 letter r’s."
- belter 1y agoI tried strawberry -> DeepSeek, GeminiPro and ChatGPT4o all correctly said three strawberrry -> DeepSeek, GeminiPro and ChatGPT4o all correctly said four stawberrry -> DeepSeek, GeminiPro all correctly said three ChatGPT4o even in a new Chat, incorrectly said the word "stawberrry" contains 4 letter "r" characters. Even provided this useful breakdown to let me know :-) Breakdown: stawberrry → s, t, a, w, b, e, r, r, r, y → 4 r's And then asked if I meant "strawberry" instead and said because that one has 2 r's....
- MattRix 1y agoThis is why things like the ARC Prize are better ways of approaching this: https://arcprize.org https://arcprize.org
- whiplash451 1y agoWell, ARC-1 did not end well for the competitors of tech giants and it’s very unclear that ARC-2 won’t follow the same trajectory.
- zurichisstained 1y agoWow, I love this benchmark - I've been doing something similar (as a joke for and much less frequently), where I ask multiple models to attempt to create a data structure like: ``` const melody = [ { freq: 261.63, duration: 'quarter' }, // C4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 293.66, duration: 'triplet' }, // D4 { freq: 0, duration: 'triplet' }, // triplet rest { freq: 329.63, duration: 'half' }, // E4 ] ``` But with the intro to Smoke on the Water by Deep Purple. Then I run it through the Web Audio API and see how it sounds. It's never quite gotten it right, but it's gotten better, to the point where I can ask it to make a website that can play it. I think yours is a lot more thoughtful about testing novelty, but its interesting to see them attempt to do things that they aren't really built for (in theory!). https://codepen.io/mvattuone/pen/qEdPaoW https://codepen.io/mvattuone/pen/qEdPaoW - ChatGPT 4 Turbo https://codepen.io/mvattuone/pen/ogXGzdg https://codepen.io/mvattuone/pen/ogXGzdg - Claude Sonnet 3.7 https://codepen.io/mvattuone/pen/ZYGXpom https://codepen.io/mvattuone/pen/ZYGXpom - Gemini 2.5 Pro Gemini is by far the best sounding one, but it's still off. I'd be curious how the latest and greatest (paid) versions fare. (And just for comparison, here's the first time I did it... you can tell I did the front-end because there isn't much to it!) https://nitter.space/mvattuone/status/1646610228748730368#m https://nitter.space/mvattuone/status/1646610228748730368#m
- dr_kretyn 1y agoI'm slightly confused by your example. What's the actual prompt? Is your expectation that a text model is going to know how to perform the exact song in audio?
- zurichisstained 1y agoOhhh absolutely not, that would be pretty wild - I just wanted to see if it could understand musical notation enough to come up with the correct melody. I know there are far better ways to do gen AI with music, this was just a joke prompt that worked far better than I expected. My naive guess is all of the guitar tabs and signal processing info it's trained on gives it the ability to do stuff like this (albeit not very well).
- 1y ago
- NicoSchwandner 1y agoNice post, thanks!
- NohatCoder 1y agoIf you calculate ELO based on a round-robin tournament with all participants starting out on the same score, then the resulting ratings should simply correspond to the win count. I guess the algorithm in use take into account the order of the matches, but taking order into account is only meaningful when competitors are expected to develop significantly, otherwise it is just added noise, so we never want to do so in competitions between bots. I also can't help but notice that the competition is exactly one match short, for some reason exactly one of the 561 possible pairings has not been included.
- simonw 1y agoYeah, that's a good call out: Elo isn't actually necessary if you can have every competitor battle every other competitor exactly once. The missing match is because one single round was declared a draw by the model, and I didn't have time to run it again (the Elo stuff was very much rushed at the last minute.)
- tazjin 1y ago[flagged]
- m3047 1y agoTIL: Snitchbench!
- irthomasthomas 1y agoThe best pelicans come from running a consortium of models. I use pelicans as evals now. https://x.com/xundecidability/status/1921009133077053462 https://x.com/xundecidability/status/1921009133077053462 Test it using VibeLab (wip) https://x.com/xundecidability/status/1926779393633857715 https://x.com/xundecidability/status/1926779393633857715
- Klaus_ 1y ago[dead]
- 0points 1y agoSo the only bird slightly resembling a pelican beak was drawn by gemini 2.5 pro. In general, none of the output resembles a pelican enough so you could separate it from "a bird". OP seem to ignore that pelican has a distinct look when evaluating these doodles.
- simonw 1y agoThe pelican's distinct look - and the fact that none of the models can capture it - is the whole point.
- 0points 1y ago> The pelican's distinct look - and the fact that none of the models can capture it - is the whole point. You didn't even mention the beak or the lack of similarities in your blog. Your text is centered around this rather peculiar statement: > Most importantly: pelicans can’t ride bicycles.
- simonw 1y agoThe blog was my attempt to capture the key ideas from the talk, which was full of jokes that don't come across as well in text as they do out loud. "Pelicans can't ride bicycles" is a good joke.
- darkoob12 1y agoShould we be that excited about AI and calling a fraud and plagiarism machine "ChatGPT Mischief Buddy" without any moral deliberation?
- simonw 1y agoThe "mischief buddy" joke is a poke at exactly that.
- buserror 1y agoThe hilarious bit is that this page will soon be scraped by ai-bots as learning material, and they'll all learn to draw pelicans on bicycles using this as their primary example material, as they'll be the only examples. GIGO in motion :-)
- beefnugs 1y agoI think its hilarious how humans can make mistakes interpreting the crazy drawings : He says "I like how it solved the problem of pelicans not fitting on bicycles by adding a second smaller bicycle to the stack." no... that is an attempt at it actually drawing the pedals, and putting the pelicans feet right on the pedals!