5 ms·
Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model. In the sam
by OtherShrezzing 3mo ago
Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model.
In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.
Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.
- simonw 3mo agoYou know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass. Blue sky and green grass aren't that surprising, but the color and direction are interesting. When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like which direction, color of bicycle, general pelican geometry etc. It will be interesting to see if other creatures end up with coincidentally similar design choices or if that's unique to the pelican-bicycle combination.
- pterhx 3mo agoIn photography (and probably art in general), there's a composition "rule" to frame moving subjects from left to right. So the direction may not be that interesting!
- copperx 3mo agoIs it culture dependent? Is it because in English we read left to right?
- simonw 3mo agoThere was a glorious moment when I thought that the Chinese models were more likely to produce right-to-left cycling pelicans, but sadly that trend didn't seem to hold up.
- BeetleB 3mo agoFor almost the last 70 years, Chinese has been left to right. Before that it was vertical (although the ordering of the columns was right to left).
- CorrectHorseBat 3mo agoChinese is also written left to right
- valleyer 3mo agoArabic or Hebrew would be better tests for that.
- ahtihn 3mo agoI wonder if that changes in countries where the main language is written right to left?
- filoleg 3mo agoThat was my first thought too, I wonder if it works the same in countries speaking arabic (as that's the first one i could think of that's a language with truly no-buts right to left writing).
- elashri 3mo agoArabic native speaker here. Yes, people will usually post or draw a bicycle right to left which is going to ve opposite of what normally is drawn. I tried the prompt in arabic for many models and I don't recall any adjusting it based on that difference at least culturally speaking.
- pclmulqdq 3mo agoIt is. All over the Arab world, imagery in ads is “backwards” and I believe several companies will flip their ads horizontally, and UI localization involves flipping graphics.
- embedding-shape 3mo agoDo the gears and stuff sit on the other side of the bike in the Arab world? Otherwise I'd expect cycling ads to still show a bike going from left to the right, considering https://news.ycombinator.com/item?id=48951828 https://news.ycombinator.com/item?id=48951828
- nick3000 3mo agoThe other thing to consider (as someone who frequently take a photos of their bike) the common direction has the drive side out! In cycling forums it is sacrilegious to post a photo of your bicycle without showing the drive side.
- ray_kay777 3mo agoBeat me to it - but I had the same thought. Most amateur and nearly all professional studio photographs of a bicycle will have it drive side out so I expect this plays some role in it.
- embedding-shape 3mo ago> the common direction has the drive side out Took some searching and sleuthing to actually figure out what "drive side out" means, as I'm just a casual "from A to B" cyclist: apparently this is referring to the side the chainset, chainwheels and all those things are on.
- talloaktrees 3mo agoside scrolling video games were always moving from left to right
- sehugg 3mo agoWell except for Jungle King
- exhaze 3mo agoI thought my joke post was silly and then I read new comments and I'm like, "I didn't try hard enough" lol
- cuttothechase 3mo agoThe art styling is more or less uniform too. I haven't seen many AI works that produces a pelican on a bicycle done in a "Ligne Claire" style, for example. I guess AI's narrows down the output probability space drastically and converge on some agreed upon aesthetics. Works great for computer programs but bad for art.
- forgot-my-pw 3mo agoI have done some variation of the other animals, also for something more tricky where they need to calculate things, I ask them to draw an SVG at a certain angle. For example: "generate an SVG of a chessboard seen from a 45 degree angle slightly higher POV" or "generate an SVG of a basketball court from a TV broadcast perspective". I find Gemini is still the best at creating SVGs.
- lIl-IIIl 3mo agoBicycle color, grass color and sky color are all part of the prompt. >Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground
- segmondy 3mo agoThere's a bias in the direction all things face. You can ask these models to generate a thing animal, car etc and you will notice that 90% of them will converge towards the same sort of results. If you ask for something rotating, 90% of them will rotate right and a few odd ones will rotate left.
- walrus01 3mo agoWhat's interesting is that given the fairly general and short in length prompt for the test, none of the models are attempting things like more discrete details of the bike. Such as showing V-brakes or dual 160mm disc rotors, rear derailleur, water bottle in a bottle cage, panniers, lights, saddlebag, the rider wearing a helmet, or other details that might be found on as vague a description as "a bicycle".
- cogman10 3mo agoIt'd be hard to fully compare, but I think a truly random "creature-vehicle" along side the pelican test would catch who's gaming and who's not. I'd also enjoy the absurdism of "Herring on a pogostick"
- aenis 3mo agoThe models are already brilliant at that. My own todo app generates 128x128 pixel art icons for my todo items. They are mind blowingly creative and funny.
- InsideOutSanta 3mo ago> the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model You would not expect that to happen if the models trained on the unrecognizable mess, right? > model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts And the labs clearly did focus on improving image rendering. > they have a uniform style SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.
- OtherShrezzing 3mo agoI’m not suggesting Simon’s pelicans in the dataset are having a meaningful impact. I’m expecting that a company like ScaleAI has a product along the lines of “benchmax dataset: SimonW’s Pelican on Bikes test” which is a private curated series of well-drawn SVGs of animals riding vehicles for training and RL.
- simonw 3mo agoIf such a product existed I'm reasonably confident someone would have tipped me off by now, NDAs be damned.
- evilduck 3mo agoIf they're benchmaxed on SVG pelicans then the outcome of that has still produced a surprisingly good generic SVG image generator. Go invent your own random alternatives and the AI models have across the board gotten better over time. Insects playing sports, anthropomorphic fruits performing martial arts, wizards conjuring weapons of WWII, whatever you can imagine. I've tried a lot of these, well beyond what I think would be a reasonable thing to specifically train as combinations. If they have given it a corpus of SVG drawings it has learned to extrapolate. (note: wizards conjuring a tank got me a surprise animated SVG with my Qwen 3.6 35B model)
- pegasus 3mo agoWatercolors in SVG?
- hombre_fatal 3mo agoAlso, I'd assume the ideal output for an underspecified, generic prompt is the most expected, generic result. Not something that defaults off the rails with creative license.
- jefftk 3mo ago> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts That doesn't seem right. I use these models as research assistants when writing lots of random blog posts (including in economically ~useless areas like the history of contra dance) and Fable 5 is a serious improvement (when I don't get downgraded!) over Opus 4.6-4.8 which was a serious improvement over Opus 4.
- conception 3mo agoIf you’ve been keeping track of all of the pelicans, there is actually stylistic differences - sometimes pretty big differences as far as watercolors go. It’s an SVG so I’m not sure what you’re looking for there. Most look the same because the prompt is to make a pelican on a bicycle as an SVG. It’s not some giant image prompt.
- doctorpangloss 3mo agobeing able to draw a picture of a pelican is really cool and it requires intelligence but i don't think it's a good measure of improving capabilities of these models nor AGI. we don't have to spend so much breath on it.
- vunderba 3mo ago> Moreover, they have a uniform style, even though your prompt doesn’t ask for one. This shouldn’t really come as a surprise, particularly to anyone who’s used diffusion models. The same thing happens when you ask an LLM for a short story [1] without providing any specific details. Even cranking up the temperature or top_p values is no panacea. The more generic your prompt, the more pedestrian the response. [1] - https://news.ycombinator.com/item?id=42093394 https://news.ycombinator.com/item?id=42093394