8 ms·

Pelicans on a bicycle

102 points by colejohnson66 2y ago

flippyhead 2y ago
Once again, Claude wins.
twiss 2y ago
I think it's interesting to consider how humans would go about this task ("Generate an SVG of a pelican riding a bicycle"), and how well they would do if they had to output the SVG into a text box without any other tools. Considering that, I think Claude 3.5 Sonnet and GPT-4o did incredibly well, and even the others might be commended for making a valid SVG at all..
- nemomarx 2y ago
  depends on the human, right? I imagine an artist who specializes in SVG would do pretty well and might make it a professional logo?
  - egypturnash 2y ago
    Pro artist who specializes in vector work here: I would use Adobe Illustrator to draw it, while looking at actual photos of pelicans and bicycles, and export an SVG. If it needed to have a lot of named parts I could make that happen. If I had the latest version of Illustrator then I would consider seeing how well its image generation does, but I do not because it has a lot of exciting new bugs that break my normal workflow. I believe that under the hood that works by feeding your text prompt to a bitmap image generator and running the same old autotrace on it, which results in some pretty messy and hard-to-edit shapes.
danielcorin 2y ago
More recently, `gemini-exp-1206` did quite well [1]. [1]: https://github.com/simonw/pelican-bicycle/blob/main/README.md#gemini-exp-1206 https://github.com/simonw/pelican-bicycle/blob/main/README.m...
- toxik 2y ago
  I feel like ”quite well” is overselling it a bit. It did maybe better.
- ttul 2y ago
  Gemini 1206 is the new hotness in my books. I've moved my day to day LLM needs over to Google's tab for the first time. I'm not sure what they changed, but it deserves a good look. Claude 3.5-Sonnet (New) is fantastic as well, but the 2M token context window offered by Google allows you to suck in an entire code repository and reason effectively across the whole thing. Google is catching up...
simonw 2y ago
I've been using this dumb benchmark for a few months now. More posts about it here: https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/
- behnamoh 2y ago
  [flagged]
  - simonw 2y ago
    Sure, they're the wrong tool for drawing a pelican - but testing their SVG output is a useful way to get a feel for how good they are at step by step reasoning, coordinate systems, spatial awareness and generating valid SVG/XML. There are genuinely useful applications of SVG-generation from LLMs - outputting simple infographics or charts for example. I use LLMs to write HTML all the time, of which SVG is a useful optional component.
    - KMnO4 2y ago
      This benchmark is interesting, because it sidesteps the reasoning and process that humans would excel at. For example, if I asked you to assemble a bookshelf with some wood, nails, and cement, you might first make a hammer with the cement before trying to assemble the bookshelf. You can get a much better image by first asking the (multimodal) LLM to draw an image of a pelican on a bicycle, and then generate an SVG using the referenced image. https://chatgpt.com/share/67609300-9abc-800d-9b26-95074f214954 https://chatgpt.com/share/67609300-9abc-800d-9b26-95074f2149...
  - gffrd 2y ago
    > these models aren't pelican painters or anything like that, they're LANGUAGE models Tools are defined by what people use them for, not by how they were intended—or designed—to be used. (Just ask Nvidia) adding: so I think someone comparing how various tools perform at a task that's valuable to them—and probably others—is just fine, even if it's different from what the creator of the tool intended?
- 2y ago
eminence32 2y ago
I like how one of them is clearly (to my eyes, at least) a person holding a gun
MarkusWandel 2y ago
I'm focusing on the bike part here because, as a bike geek, I could draw one from memory that's correct in all details. But to a non-bikie that's more difficult than you'd think. I can't find the picture gallery right now but an article about it, which links another article: https://web.archive.org/web/20240419001426/https://www.wired.com/2016/04/can-draw-bikes-memory-definitely-cant/ https://web.archive.org/web/20240419001426/https://www.wired... So the fact that the AI models screw this up so badly is understandable. Sure, they screw up in ways that humans wouldn't, such as the beak backwards in one of the pictures (pointy end toward the bird!) because they don't know or care about something every human would know: What a beak is for and what it looks like in general. Or for that matter the biodynamics of how a pelican's long, spindly legs could, in fact, work a pair pedals. But ask me to draw a pelican from memory, and have a good laugh (if you're better at it than me) because to me, they're just kind of a peripheral vision, pink abstraction, not something I focus on understanding. And that's what they are to the AI model too.
- parpfish 2y ago
  > ... they're just kind of a peripheral vision, pink abstraction, not something I focus on understanding are there pink pelicans, or are you thinking of flamingoes?
  - MarkusWandel 2y ago
    Ha, see, even missed that part! Honestly.
    - codetrotter 2y ago
      You had me wondering at > a pelican's long, spindly legs and when you then also described it as pink, it started to make sense and I too understood that you must be thinking of flamingos :^)
- larubbio 2y ago
  This is the the artist you are thinking of. https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/
oatsandsugar 2y ago
It looks like no LLM has seen a pelican before at all.
eitally 2y ago
Those output examples are absolutely horrifically bad compared to what I get with a cursory request to Gemini 2.0 using Imagen3 via gemini.google.com.
- toxik 2y ago
  This is generating SVG data, not using an image generator.
- alwa 2y ago
  Was yours an SVG? I think that’s what makes Simon feel that this test is useful: the LLM has to generate functioning SVG code describing these shapes.
  - simonw 2y ago
    Yeah, this test is to see if a pure LLM can output SVG that renders well. It's effectively a test of their "spatial reasoning" capabilities.
    - openrisk 2y ago
      Its sneakier than that because there are so many ways you can draw the same thing in svg, its also some sort of orchestration and coherence test
pjs_ 2y ago
I wanna see this running in a feedback loop - show the model its output and get it to make corrections. Remember that these are basically one-shot. Very different to how you or I would solve the problem (get a circle up on the screen, have a look at it, make some changes, add some wings, tweak the dimensions, etc.). We would go through hundreds or thousands of feedback cycles before we got something half-decent -- in this situation the model only gets one attempt.
cadamsau 2y ago
Careful not to shout too loud about it or they’ll start training for it! Latest Claude does a suspiciously good job…
marcodiego 2y ago
Most humans can't correctly draw a bicycle from memory: https://www.wired.com/2016/04/can-draw-bikes-memory-definitely-cant/ https://www.wired.com/2016/04/can-draw-bikes-memory-definite...
fuegoio 2y ago
Mistral Large proposition: https://chat.mistral.ai/chat/4da427b1-e033-454d-b134-c5d1f6ed6552 https://chat.mistral.ai/chat/4da427b1-e033-454d-b134-c5d1f6e...
Mistletoe 2y ago
If only it didn't have to be svg, so glorious. https://g.co/gemini/share/56e5dfa1a598 https://g.co/gemini/share/56e5dfa1a598
skissane 2y ago
Just thinking out loud: a lot of models generate pretty poor SVG output, but part of that is because they don't get any visual feedback. But, what about this workflow: given prompt, LLM generates two SVG outputs. Both are rendered by an SVG renderer, and then we combine the two into one image, one on the left and the other on the right. We then ask a visual LLM (could be the same LLM or could be a different one) to tell us whether the left half or right half of the image is a better response to the prompt. Now we've got preferences which can be used to fine-tune the LLM using DPO. And you could iteratively repeat the process – as the LLM is fine-tuned it may produce even better outputs which then produces new preferences for further fine-tuning. Would be interesting to see what kinds of results it might produce in practice.
- codetrotter 2y ago
  > Would be interesting to see what kinds of results it might produce in practice. I would not be surprised to see that the LLM generating SVG does a UNO reverse, and makes use of little colored squares in a grid to draw a “vector image” where each of the squares represents individual pixels :p
  - skissane 2y ago
    Such a “vector image” would be very large, if you put a limit on the length of the generated SVG the LLM would be blocked from pursuing that strategy. Even without an explicit limit, the LLM’s context limit may block it: if the resulting SVG is larger than the LLM’s context, the LLM may forget what it is doing part way through generating the SVG, which is unlikely to produce a good result.
e3a8 2y ago
"Generate room with no elephant" https://imgur.com/BI61S1T https://imgur.com/BI61S1T
- Zopieux 2y ago
  Models hate negative prompts. There's a reason Stable Diffusion txt2img requires a dedicated "negative" CLIP input in addition to positive. FLUX, for all its benefits, is really annoying in this aspect as you cannot really provide negative guidance without complicated hacks [1]. Therefore, "orignal" ideas that go against popular wisdom or mainstream culture will be very hard to prompt for. This is, in my opinion, an important caveat of ML "art". ML "art" is already considered useless slop because of how much it conforms to biases built in the weights. The inability to tune down adherence to mainstreaming norms is therefore all the more problematic. [1] https://www.reddit.com/r/StableDiffusion/comments/1estj69/ https://www.reddit.com/r/StableDiffusion/comments/1estj69/
boredhedgehog 2y ago
Some of the drawings, like the one from Amazon Nova Pro, are quite fascinating as abstract artworks. It's like the idea of a bicycle without its physicality.
deleted 2y ago
[deleted]
tessellated 2y ago
> aren't any pelican on a bicycle SVG files floating around (yet) maintains website collecting SVG files of pelicans on bicycles
kayo_20211030 2y ago
I searched the comments and nobody's mentioned that none of the bicycle riders looked like pelicans. Seriously, a Pelican looks a certain way, ... more than his belly can and all that.
yowzadave 2y ago
Some friends noticed another one the other day—none of the image generators were able to generate a “glass of wine, full to the brim.” All of the images were just of regular partially-filled glasses of wine, with the accompanying text trying to assure you that the glass was indeed full to the brim.

102 points by colejohnson66 2y ago

Once again, Claude wins.

I think it's interesting to consider how humans would go about this task ("Generate an SVG of a pelican riding a bicycle"), and how well they would do if they had to output the SVG into a text box without any other tools. Considering that, I think Claude 3.5 Sonnet and GPT-4o did incredibly well, and even the others might be commended for making a valid SVG at all..

nemomarx 2y ago
depends on the human, right? I imagine an artist who specializes in SVG would do pretty well and might make it a professional logo?
- egypturnash 2y ago
  Pro artist who specializes in vector work here: I would use Adobe Illustrator to draw it, while looking at actual photos of pelicans and bicycles, and export an SVG. If it needed to have a lot of named parts I could make that happen. If I had the latest version of Illustrator then I would consider seeing how well its image generation does, but I do not because it has a lot of exciting new bugs that break my normal workflow. I believe that under the hood that works by feeding your text prompt to a bitmap image generator and running the same old autotrace on it, which results in some pretty messy and hard-to-edit shapes.

danielcorin 2y ago

More recently, `gemini-exp-1206` did quite well [1]. [1]: https://github.com/simonw/pelican-bicycle/blob/main/README.md#gemini-exp-1206 https://github.com/simonw/pelican-bicycle/blob/main/README.m...

toxik 2y ago
I feel like ”quite well” is overselling it a bit. It did maybe better.
ttul 2y ago
Gemini 1206 is the new hotness in my books. I've moved my day to day LLM needs over to Google's tab for the first time. I'm not sure what they changed, but it deserves a good look. Claude 3.5-Sonnet (New) is fantastic as well, but the 2M token context window offered by Google allows you to suck in an entire code repository and reason effectively across the whole thing. Google is catching up...

simonw 2y ago

I've been using this dumb benchmark for a few months now. More posts about it here: https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/

behnamoh 2y ago
[flagged]
- simonw 2y ago
  Sure, they're the wrong tool for drawing a pelican - but testing their SVG output is a useful way to get a feel for how good they are at step by step reasoning, coordinate systems, spatial awareness and generating valid SVG/XML. There are genuinely useful applications of SVG-generation from LLMs - outputting simple infographics or charts for example. I use LLMs to write HTML all the time, of which SVG is a useful optional component.
  - KMnO4 2y ago
    This benchmark is interesting, because it sidesteps the reasoning and process that humans would excel at. For example, if I asked you to assemble a bookshelf with some wood, nails, and cement, you might first make a hammer with the cement before trying to assemble the bookshelf. You can get a much better image by first asking the (multimodal) LLM to draw an image of a pelican on a bicycle, and then generate an SVG using the referenced image. https://chatgpt.com/share/67609300-9abc-800d-9b26-95074f214954 https://chatgpt.com/share/67609300-9abc-800d-9b26-95074f2149...
- gffrd 2y ago
  > these models aren't pelican painters or anything like that, they're LANGUAGE models Tools are defined by what people use them for, not by how they were intended—or designed—to be used. (Just ask Nvidia) adding: so I think someone comparing how various tools perform at a task that's valuable to them—and probably others—is just fine, even if it's different from what the creator of the tool intended?
2y ago

eminence32 2y ago

I like how one of them is clearly (to my eyes, at least) a person holding a gun

MarkusWandel 2y ago

I'm focusing on the bike part here because, as a bike geek, I could draw one from memory that's correct in all details. But to a non-bikie that's more difficult than you'd think. I can't find the picture gallery right now but an article about it, which links another article: https://web.archive.org/web/20240419001426/https://www.wired.com/2016/04/can-draw-bikes-memory-definitely-cant/ https://web.archive.org/web/20240419001426/https://www.wired... So the fact that the AI models screw this up so badly is understandable. Sure, they screw up in ways that humans wouldn't, such as the beak backwards in one of the pictures (pointy end toward the bird!) because they don't know or care about something every human would know: What a beak is for and what it looks like in general. Or for that matter the biodynamics of how a pelican's long, spindly legs could, in fact, work a pair pedals. But ask me to draw a pelican from memory, and have a good laugh (if you're better at it than me) because to me, they're just kind of a peripheral vision, pink abstraction, not something I focus on understanding. And that's what they are to the AI model too.

parpfish 2y ago
> ... they're just kind of a peripheral vision, pink abstraction, not something I focus on understanding are there pink pelicans, or are you thinking of flamingoes?
- MarkusWandel 2y ago
  Ha, see, even missed that part! Honestly.
  - codetrotter 2y ago
    You had me wondering at > a pelican's long, spindly legs and when you then also described it as pink, it started to make sense and I too understood that you must be thinking of flamingos :^)
larubbio 2y ago
This is the the artist you are thinking of. https://www.gianlucagimini.it/portfolio-item/velocipedia/ https://www.gianlucagimini.it/portfolio-item/velocipedia/

oatsandsugar 2y ago

It looks like no LLM has seen a pelican before at all.

eitally 2y ago

Those output examples are absolutely horrifically bad compared to what I get with a cursory request to Gemini 2.0 using Imagen3 via gemini.google.com.

toxik 2y ago
This is generating SVG data, not using an image generator.
alwa 2y ago
Was yours an SVG? I think that’s what makes Simon feel that this test is useful: the LLM has to generate functioning SVG code describing these shapes.
- simonw 2y ago
  Yeah, this test is to see if a pure LLM can output SVG that renders well. It's effectively a test of their "spatial reasoning" capabilities.
  - openrisk 2y ago
    Its sneakier than that because there are so many ways you can draw the same thing in svg, its also some sort of orchestration and coherence test

pjs_ 2y ago

I wanna see this running in a feedback loop - show the model its output and get it to make corrections. Remember that these are basically one-shot. Very different to how you or I would solve the problem (get a circle up on the screen, have a look at it, make some changes, add some wings, tweak the dimensions, etc.). We would go through hundreds or thousands of feedback cycles before we got something half-decent -- in this situation the model only gets one attempt.

cadamsau 2y ago

Careful not to shout too loud about it or they’ll start training for it! Latest Claude does a suspiciously good job…

marcodiego 2y ago

Most humans can't correctly draw a bicycle from memory: https://www.wired.com/2016/04/can-draw-bikes-memory-definitely-cant/ https://www.wired.com/2016/04/can-draw-bikes-memory-definite...

fuegoio 2y ago

Mistral Large proposition: https://chat.mistral.ai/chat/4da427b1-e033-454d-b134-c5d1f6ed6552 https://chat.mistral.ai/chat/4da427b1-e033-454d-b134-c5d1f6e...

Mistletoe 2y ago

If only it didn't have to be svg, so glorious. https://g.co/gemini/share/56e5dfa1a598 https://g.co/gemini/share/56e5dfa1a598

skissane 2y ago

Just thinking out loud: a lot of models generate pretty poor SVG output, but part of that is because they don't get any visual feedback. But, what about this workflow: given prompt, LLM generates two SVG outputs. Both are rendered by an SVG renderer, and then we combine the two into one image, one on the left and the other on the right. We then ask a visual LLM (could be the same LLM or could be a different one) to tell us whether the left half or right half of the image is a better response to the prompt. Now we've got preferences which can be used to fine-tune the LLM using DPO. And you could iteratively repeat the process – as the LLM is fine-tuned it may produce even better outputs which then produces new preferences for further fine-tuning. Would be interesting to see what kinds of results it might produce in practice.

codetrotter 2y ago
> Would be interesting to see what kinds of results it might produce in practice. I would not be surprised to see that the LLM generating SVG does a UNO reverse, and makes use of little colored squares in a grid to draw a “vector image” where each of the squares represents individual pixels :p
- skissane 2y ago
  Such a “vector image” would be very large, if you put a limit on the length of the generated SVG the LLM would be blocked from pursuing that strategy. Even without an explicit limit, the LLM’s context limit may block it: if the resulting SVG is larger than the LLM’s context, the LLM may forget what it is doing part way through generating the SVG, which is unlikely to produce a good result.

e3a8 2y ago

"Generate room with no elephant" https://imgur.com/BI61S1T https://imgur.com/BI61S1T

Zopieux 2y ago
Models hate negative prompts. There's a reason Stable Diffusion txt2img requires a dedicated "negative" CLIP input in addition to positive. FLUX, for all its benefits, is really annoying in this aspect as you cannot really provide negative guidance without complicated hacks [1]. Therefore, "orignal" ideas that go against popular wisdom or mainstream culture will be very hard to prompt for. This is, in my opinion, an important caveat of ML "art". ML "art" is already considered useless slop because of how much it conforms to biases built in the weights. The inability to tune down adherence to mainstreaming norms is therefore all the more problematic. [1] https://www.reddit.com/r/StableDiffusion/comments/1estj69/ https://www.reddit.com/r/StableDiffusion/comments/1estj69/

boredhedgehog 2y ago

Some of the drawings, like the one from Amazon Nova Pro, are quite fascinating as abstract artworks. It's like the idea of a bicycle without its physicality.

deleted 2y ago

[deleted]

tessellated 2y ago

> aren't any pelican on a bicycle SVG files floating around (yet) maintains website collecting SVG files of pelicans on bicycles

kayo_20211030 2y ago

I searched the comments and nobody's mentioned that none of the bicycle riders looked like pelicans. Seriously, a Pelican looks a certain way, ... more than his belly can and all that.

yowzadave 2y ago

Some friends noticed another one the other day—none of the image generators were able to generate a “glass of wine, full to the brim.” All of the images were just of regular partially-filled glasses of wine, with the accompanying text trying to assure you that the glass was indeed full to the brim.