3 ms·
I wonder when pelican riding a bicycle will be useless as an evaluation task. The point was that it was something weird nobody had ever really thought about bef
by jubilanti 6mo ago
I wonder when pelican riding a bicycle will be useless as an evaluation task. The point was that it was something weird nobody had ever really thought about before, not in the benchmarks or even something a team would run internally. But now I'd bet internally this is one of the new Shirley Cards.
- abustamam 6mo agoSimon has an article on this https://simonwillison.net/2025/Nov/13/training-for-pelicans-riding-bicycles/ https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
- rafaelmn 6mo agoI mean look at the result where he asked about a unicycle - the model couldn't even keep the spokes inside the wheels - would be rudimentary if it "learned" what it means to draw a bicycle wheel and could transfer that to unicycle.
- duzer65657 6mo agoit's the frame that's surprisingly - and consistentnly - wrong. You'd think two triangles would be pretty easy to repro; once you get that the rest is easy. It's not like he's asking "draw a pelican on a four-bar linkage suspension mountainbike..."
- Reddit_MLP2 6mo agoThis is older, but even humans don't have a great concept of how a bicycle works... https://twistedsifter.com/2016/04/artist-asks-people-to-draw-bicycle-from-memory-and-renders-results/ https://twistedsifter.com/2016/04/artist-asks-people-to-draw...
- yndoendo 6mo agoWouldn't this be more about being capable of mentally remembering how a bicycle looks versus how it works? This reminds me of Pictionary. [0] Some people are good and some are really bad. I am really bad a remembering how items look in my head and fail at drawing in Pictionary. My drawing skills are tied to being able to copy what I see. [0] https://en.wikipedia.org/wiki/Pictionary https://en.wikipedia.org/wiki/Pictionary
- johanvts 6mo agoI think it’s difficult to draw a bike exactly because you remember how it works rather than how it looks, so you worry about placing all the functional parts and get the overall composition wrong. Similar to drawing faces, without training, people will consistently dedicate too much area to the lower part of the face and draw some kind of neanderthal with no forehead.
- quinnjh 6mo agois it possible to have greater success with the specificity? I don't think i ever drew a bike frame properly as a kid despite riding them and understanding the concept of spokes and wheels...
- MagicMoonlight 6mo agoThey’ll hardcode it in 4.8, just like they do when they need to “fix” other issues
- amelius 6mo agoYeah try it with something else, or e.g. add a tiger to the back seat.
- SwellJoe 6mo agoPelicanmaxxing
- survirtual 6mo agoI use this metric now, and I suggest you change it per your imagination: "Make a single-page HTML file using threejs from a CDN. Render a scene of a flying dinosaur orbiting a planet. There are clouds with thunder and lightning, and the background is a beautiful starscape with twinkling stars and a colorful nebula" This allows me to evaluate several factors across models. It is novel and creative. I generally run it multiple times, though now that I have shared it here, I will come up with new scenes personally to evaluate. I also consider how well it one shots, errors generated, response to errors being corrected, and velocity of iteration to improvement. Generally speaking, Claude Sonnet has done the best, Qwen3.5 122B does second, and I have nice results from Qwen3.5 35B. ChatGPT does not do well. It can complete the task without errors but the creativity is atrocious.
- hansmayer 6mo agoValid points, but you"d think "superintelligence" would "know" how to draw a pelican on a bike?