4 ms·

For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://news.ycombinator.com/item?id=49289

by simonw 2mo ago

For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://news.ycombinator.com/item?id=49289112#49290012 https://news.ycombinator.com/item?id=49289112#49290012. - that turned out to be my fault, not the model. My SVG rendering software was stripping some attributes. Here's the Gemini 3.7 Flash pelicans in the fixed renderer: https://tools.simonwillison.net/markdown-svg-renderer.html#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F6779a22d5e7bb6bdf29936f1600a5259 https://tools.simonwillison.net/markdown-svg-renderer.html#u...

Einenlum 2mo ago
Damn That's very good
theplumber 2mo ago
Gemini is worse
lacoolj 2mo ago
Holy crap that is above and beyond But, not animated like the Qwen 2.4T one you showed, so minus points for that
- irthomasthomas 2mo ago
  Animation was not requested.
lwansbrough 2mo ago
Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right. It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.
- fragmede 2mo ago
  Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.
- fwipsy 2mo ago
  I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks. Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it. *Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.
  - ElectricalUnion 2mo ago
    AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.
    - fwipsy 2mo ago
      Hmm, doesn't seem to translate to an especially coherent pelican: https://xcancel.com/TeksEdge/status/2063108620842356970 https://xcancel.com/TeksEdge/status/2063108620842356970
- oarsinsync 2mo ago
  Have you seen what happens when you ask 500 humans to draw a bicycle? No pelican, nobody riding it, just the bike. https://foerstel.com/memory-bikes/ https://foerstel.com/memory-bikes/
walrus01 2mo ago
It is interesting that it went with something like: "pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican" I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.
zkmon 2mo ago
>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare. How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.
- anakaine 2mo ago
  Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other. The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all. Also: > Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.

by simonw 2mo ago

Einenlum 2mo ago

Damn That's very good

theplumber 2mo ago

Gemini is worse

lacoolj 2mo ago

Holy crap that is above and beyond But, not animated like the Qwen 2.4T one you showed, so minus points for that

irthomasthomas 2mo ago
Animation was not requested.

lwansbrough 2mo ago

Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right. It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.

fragmede 2mo ago
Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.
fwipsy 2mo ago
I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks. Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it. *Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.
- ElectricalUnion 2mo ago
  AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.
  - fwipsy 2mo ago
    Hmm, doesn't seem to translate to an especially coherent pelican: https://xcancel.com/TeksEdge/status/2063108620842356970 https://xcancel.com/TeksEdge/status/2063108620842356970
oarsinsync 2mo ago
Have you seen what happens when you ask 500 humans to draw a bicycle? No pelican, nobody riding it, just the bike. https://foerstel.com/memory-bikes/ https://foerstel.com/memory-bikes/

walrus01 2mo ago

It is interesting that it went with something like: "pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican" I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.

zkmon 2mo ago

>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare. How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.

anakaine 2mo ago
Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other. The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all. Also: > Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.