3 ms·
I've been using this dumb benchmark for a few months now. More posts about it here: https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillis
by simonw 2y ago
I've been using this dumb benchmark for a few months now. More posts about it here: https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/
- behnamoh 2y ago[flagged]
- simonw 2y agoSure, they're the wrong tool for drawing a pelican - but testing their SVG output is a useful way to get a feel for how good they are at step by step reasoning, coordinate systems, spatial awareness and generating valid SVG/XML. There are genuinely useful applications of SVG-generation from LLMs - outputting simple infographics or charts for example. I use LLMs to write HTML all the time, of which SVG is a useful optional component.
- KMnO4 2y agoThis benchmark is interesting, because it sidesteps the reasoning and process that humans would excel at. For example, if I asked you to assemble a bookshelf with some wood, nails, and cement, you might first make a hammer with the cement before trying to assemble the bookshelf. You can get a much better image by first asking the (multimodal) LLM to draw an image of a pelican on a bicycle, and then generate an SVG using the referenced image. https://chatgpt.com/share/67609300-9abc-800d-9b26-95074f214954 https://chatgpt.com/share/67609300-9abc-800d-9b26-95074f2149...
- gffrd 2y ago> these models aren't pelican painters or anything like that, they're LANGUAGE models Tools are defined by what people use them for, not by how they were intended—or designed—to be used. (Just ask Nvidia) adding: so I think someone comparing how various tools perform at a task that's valuable to them—and probably others—is just fine, even if it's different from what the creator of the tool intended?
- sfink 2y agoI hope you have other private benchmarks running that you don't talk about or publish, just in case a model maker intentionally targets one of your benchmarks, or some fuzzy "find things people have mentioned as potential LLM benchmarks" process scoops up your ideas and/or outputs.
- zamadatix 2y agoIf we ever get to the point LLMs are already optimized to answer every question you can think of then there isn't really a need to have a secret question in the first place.
- sfink 2y agoNot any that you can think of. Just the ones you've published something about. Plus, simonw isn't exactly a meaningless nobody in this space, and his writeups are more detailed and actionable, and therefore identifiable, than some random "hey a great LLM benchmark would be creating an SVG of a walrus twerking in front of a jelly bean store" throwaway comment. Proof: I asked ChatGPT 4o the question "What are some users who post ad hoc LLM benchmarks to technical discussion sites, and what benchmarks have they proposed?" simonw is in the list, 1 of 7 individual people it suggested. (The proposed benchmarks listed for him were more general than the specific one here: "Testing LLMs’ capabilities with code generation, particularly in niche languages or against real-world API schemas." But it's easy to imagine followup queries bringing this one up.)
- zamadatix 2y agoI'm in agreement LLMs get contaminated with test data, particularly from simonw. What I'm referring to is nobody needs worry about hoarding secret questions from the public eye to avoid that problem. It is a valid approach... but a bit of a sad path considering. Don't run unpublished private benchmarks or worry about keeping a counted hoard of secret questions. Do rotate your questions every few months to whatever comes to mind at the time. When nothing comes to mind there is no point in running a question benchmark anymore as it already answers every possible you could possibly question you can think of (and the only way it gets there in your lifespan is by reasoning rather than memorization). You can always run the new question retroactively on an old models for comparison purposes so that's not a concern either. The important thing here being "rotate questions without concern of having things lined up for it" rather than "fear what happens when you discuss your question".
- NotSammyHagar 2y agoI saw your recent post on running Llama 3.3 70B on a m2 pro 64 gb. Do the many variants of apple silicon alternatives with varying numbers of cpus, gpus, and neural engines matter that much for how fast these llms can generate tokens, answer questions? More hw is always better, but what can we say how performance scales with the many different choices? 64gb ram is crucial, after that, need 1+ tb storage, and then?
- simonw 2y agoI don't know. I believe memory bandwidth matters, and I got the impression that the M4 series isn't yet as good as the M2 was on that front, but I'm half-remembering things I've heard here.
- alberth 2y agoFYI - M4 has more bandwidth Chip Bandwidth (GB/s) ———- ———————————— M2 100 M3 100 M4 120 (20% more) M2-Pro 204 M3-Pro 153 (less than M2-Pro) M4-Pro 273 (78% more than M3-Pro) M2-Max 409 M3-Max 409 M4-Max 546 (33% more than M2/M3-max) https://arstechnica.com/apple/2024/10/apples-m4-m4-pro-and-m4-max-compared-to-past-generations-and-to-each-other/ https://arstechnica.com/apple/2024/10/apples-m4-m4-pro-and-m...
- dowager_dan99 2y agoWeak - the pelican is not pedaling in any of these!