6 ms·

Absolutely the best pelican I've seen from a model that runs on my laptop: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.c

by simonw 2mo ago

Absolutely the best pelican I've seen from a model that runs on my laptop: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2Ffc909bea4fecf752c7bf9bad0e9dbf2a https://tools.simonwillison.net/markdown-svg-renderer#url=ht... Bicycle is the right shape. Pelican beak is excellent. Nice background. Most importantly, the pelican has one leg on each side of the bicycle - that's very rare. (No chain on this bicycle though - in the reasoning trace it says "already chainstay... skip chain detail; maybe a small chainring.") I ran that on an M5 Max MacBook Pro using LM Studio and their 17GB GGUF: https://lmstudio.ai/models/qwen3.8 https://lmstudio.ai/models/qwen3.8 It took 21 minutes(!) and used 22,276 reasoning tokens to produce 3,223 tokens of output. (For the "they're training on your benchmark now" crowd, all of that cheating didn't prevent it from spending 20 minutes thinking about the task first! You can see the reasoning trace in the link I shared.) For comparison, here's one I got from qwen3.8-2.4t-a95b on OpenRouter, which is pleasingly animated: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F557016f0895b2abb4b9957caec781734 https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

simonw 2mo ago
For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://news.ycombinator.com/item?id=49289112#49290012 https://news.ycombinator.com/item?id=49289112#49290012. - that turned out to be my fault, not the model. My SVG rendering software was stripping some attributes. Here's the Gemini 3.7 Flash pelicans in the fixed renderer: https://tools.simonwillison.net/markdown-svg-renderer.html#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F6779a22d5e7bb6bdf29936f1600a5259 https://tools.simonwillison.net/markdown-svg-renderer.html#u...
- Einenlum 2mo ago
  Damn That's very good
- theplumber 2mo ago
  Gemini is worse
- lacoolj 2mo ago
  Holy crap that is above and beyond But, not animated like the Qwen 2.4T one you showed, so minus points for that
  - irthomasthomas 2mo ago
    Animation was not requested.
- lwansbrough 2mo ago
  Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right. It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.
  - fragmede 2mo ago
    Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.
  - fwipsy 2mo ago
    I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks. Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it. *Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.
    - ElectricalUnion 2mo ago
      AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.
      - fwipsy 2mo ago
        Hmm, doesn't seem to translate to an especially coherent pelican: https://xcancel.com/TeksEdge/status/2063108620842356970 https://xcancel.com/TeksEdge/status/2063108620842356970
- walrus01 2mo ago
  It is interesting that it went with something like: "pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican" I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.
- zkmon 2mo ago
  >> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare. How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.
  - anakaine 2mo ago
    Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other. The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all. Also: > Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.
walrus01 2mo ago
What's the result if you ask it for a "pelican equipment case" ? I've been trying the anti-bicycle pelican on some LLMs and the results are much more varied than the bicycle prompt. Some have very different ideas of pelicans (you'll get a small case with one DSLR camera in it, or a long rifle case, etc). You'll also get cases that are isometric view, or flat plane view from the front, or open or closed.
- simonw 2mo ago
  It did a pretty outstanding job on that: https://tools.simonwillison.net/markdown-svg-renderer.html#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F137c124beae157953e60c48d1c710cef https://tools.simonwillison.net/markdown-svg-renderer.html#u... This took 56 minutes! - but it didn't have exclusive use of LM Studio, I ran a few other test prompts while it was working. llm -m lmstudio/qwen/qwen3.8-27b 'Generate an SVG of a pelican equipment case'
  - walrus01 2mo ago
    That is pretty good! It's interesting it chose orange, because that's actually a somewhat rare color for pelican cases. I would say 95% of the ones going around the world right now are either black or FDE (flat dark earth) tan colored. There are some orange ones, for sure, if you google image search "orange pelican case", people use them for emergency/first aid supplies or electronics. But only the smallest ones are available in orange. It is also worth noting that it shaded the pelican case like it was a product photo taken in a studio, there's a virtual light source above and to the top left corner of it, then shading deeper color of orange as the body area progresses towards the lower right corner.
    - pbhjpbhj 2mo ago
      What are the chances that the colouring of actual pelicans is bleeding in to the prompt. Seems it could be that?
      - walrus01 2mo ago
geniium 2mo ago
I think the model got trained only on pelican to get there
- walrus01 2mo ago
  We should start asking it for an Albatross instead.
lacoolj 2mo ago
Totally agree. I actually didn't know you already got to it and I ran it on mine. It even added a scarf lol https://imgur.com/a/IgYKnmk https://imgur.com/a/IgYKnmk Took almost 90 minutes on my Threadripper Pro, and feels well worth it. 28k tokens from the Q8 model
- HanClinto 2mo ago
  And yours got the chain!
mococa 2mo ago
See, like my comment before. We don’t need the biggest/highest/frotiest model.
distrill 2mo ago
this sort of test surely only works when it's a brand new concept/prompt. like the exact same models trained newer data will surely get better at putting this together over time
- notnullorvoid 2mo ago
  Try it out with your own prompt, I suspect you'll be surprised.
notnullorvoid 2mo ago
I've done a few variations, I've been impressed with all of them. My favourite so far has been "Generate an SVG of a turtle flying a kite", result: https://imgur.com/a/bdKJPV4 https://imgur.com/a/bdKJPV4. Some will say conflating flying and flying a kite is bad, but I like it.
sahadi 2mo ago
what type of laptop runs something like that? using mac or egpu?
gerdesj 2mo ago
I was just updating a Proxmox cluster from v8 to v9 and had time on my hands and 15-30 mins after the models appeared for general download I had an initial pelican effort in .svg format. I switch off reasoning by default and for starters I ran 3.8 the same way as 3.6 on vllm on a DGX Spark box. Open web UI prompted. My word! It looks rather decent. Much better than previous efforts.
wincy 2mo ago
Not too bad, on my 5090 I got ~160tokens/sec and it took 3m20s to generate a pelican using Ninfer which someone else mentioned. I don't know why I didn't get any sweet sweet seagulls in my background, even on xhigh reasoning effort. https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632afdb5 https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632a...
redox99 2mo ago
It's definitely trained on generating SVGs. Not that they trained on the pelican prompt itself, but probably a more general RL step that has it generate SVG and a stronger vision model judges it. It's up to you if that counts as benchmaxxed.
- nl 2mo ago
  Generation of SVGs is (obviously) a generally useful task.
kamranjon 2mo ago
Have you thought about running a second tier of the Pelican benchmark where you see which model makes best pelican on lowest or no reasoning settings? I think that'd be pretty interesting and might help highlight which models have a baseline capability - even within those I'd imagine the token usage would vary wildly and might give some indications on verbosity as well.
beltsazar 2mo ago
Using a better quant Q6_K (Unsloth's) compared to your quant Q4_K_M (LM Studio's) yields this: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fdanieljl%2F0a942c0a6499d42be099117efe9a76cc https://tools.simonwillison.net/markdown-svg-renderer#url=ht... Chains exist. Red scarf is proactively added. ("Maybe a scarf blowing in the wind for charm!) No hands/wings, though. Generated 30.2k tokens in total and took 52 mins on M5 Pro in low power mode (it will possibly take less than half of that in auto energy mode).
frumiousirc 2mo ago
> Bicycle is the right shape. Pelican beak is excellent. Nice background. Relative to other results I agree. But on an absolute measure, there is not a single element in the current bicycle that is real-world accurate and many elements are omitted or non-physical (eg, the transparent seat tube top, entire lack of a head tube). Consider a series of followup benchmarks. With a fresh context of the LLM under test, ask it to generate a list of findings for how the pelican-on-a-bicycle SVG that was produced is inaccurate compared what the real world scene might appear, accepting for the limitations of SVG as a medium. Then, feed back the list of findings to the original context for a second try. The benchmark can stop here by humans looking at the result and forming their own conclusion. Next phase is to repeat the analysis phase using the 2nd context to determine what findings were satisfied and what new inaccuracies are found. These two differences can form a second benchmark. Last phase is to iterate with the goal to drive the number of findings to zero.
maldie 2mo ago
Even with "low" thinking effort and Q4 it makes reasonable pelican, only 4k tokens and half a minute on 4090: https://imgur.com/a/yd5rNCp https://imgur.com/a/yd5rNCp
KerrAvon 2mo ago
If I'm doing the math right, that's like 17 t/s? I haven't played with Qwen 3.8 yet, but that seems really slow for a 27B on an M5 Max with sufficient RAM to hold it in memory.
latentsea 2mo ago
I asked it to make an animated svg of a pelican doing tricks on a bmx bike and it did. I then asked it to turn that into a browser game. This model fucks.
nullbio 2mo ago
Ask it to do a zebra on a jetski to escape the benchmax.

by simonw 2mo ago

simonw 2mo ago

For anyone who followed yesterday's Gemini 3.7 Flash pelican which rendered in Safari but not in Firefox or Chrome... https://news.ycombinator.com/item?id=49289112#49290012 https://news.ycombinator.com/item?id=49289112#49290012. - that turned out to be my fault, not the model. My SVG rendering software was stripping some attributes. Here's the Gemini 3.7 Flash pelicans in the fixed renderer: https://tools.simonwillison.net/markdown-svg-renderer.html#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F6779a22d5e7bb6bdf29936f1600a5259 https://tools.simonwillison.net/markdown-svg-renderer.html#u...

Einenlum 2mo ago
Damn That's very good
theplumber 2mo ago
Gemini is worse
lacoolj 2mo ago
Holy crap that is above and beyond But, not animated like the Qwen 2.4T one you showed, so minus points for that
- irthomasthomas 2mo ago
  Animation was not requested.
lwansbrough 2mo ago
Perhaps this isn’t a new observation but the problem with LLMs is very clear with these. It’s a nice microcosm. The LLM will draw a fish companion (unprompted!) with a nice gradient but won’t get the pelican’s feet right. It’s obviously a problem of fundamental understanding and demonstrates that reasoning is more “directionless rigour”.
- fragmede 2mo ago
  Or a matter of having limited time but spending it thinking about the wrong thing. Which, not to anthropomorphize it, but it's not like humans don't do that, or at least I do.
- fwipsy 2mo ago
  I'd like to see a human create a better pelican SVG without being able to look at the results. LLMs are language models, not (natively) vision models*. That's why it's a good benchmark: it engages LLM's logical reasoning in a way that we can check visually. The fact that they make errors which can be spotted visually, doesn't prove that LLMs are so far behind humans on language/logic tasks. Besides which, "fundamental understanding" isn't a binary. Humans can also have a deep understanding of a subject and still make errors. (I do, anyways.) I do not think "fundamental understanding" is itself well enough understood to say definitively that LLMs do or do not have it. *Some models have multimodal capabilities but these are separate weights, I believe it's only engaged in processing images. From the reasoning trace, the model never rendered the result -- only instructed the user to do so. I welcome correction if I'm wrong, I only have a surface-level understanding here.
  - ElectricalUnion 2mo ago
    AFAIK the only well-known LLM with no "conventional" multimodal encoders required for audio and images is Gemma 4 12B.
    - fwipsy 2mo ago
      Hmm, doesn't seem to translate to an especially coherent pelican: https://xcancel.com/TeksEdge/status/2063108620842356970 https://xcancel.com/TeksEdge/status/2063108620842356970
walrus01 2mo ago
It is interesting that it went with something like: "pelican = ocean bird = nautical theme = this means I should put a captain's hat on the cartoon Pelican" I think in some sense it understood the absurdity of a pelican riding a bicycle as intentionally nonsensical, so it went with a cartoon theme for the design and the shading. Worked out really well.
zkmon 2mo ago
>> Most importantly, the pelican has one leg on each side of the bicycle - that's very rare. How can you say "each side"? I don't see any z-ordering between bicycle frame and legs.
- anakaine 2mo ago
  Because, if you look at the image you can very clearly see that the frame and relative components partially occlude one leg and not the other. The model is presumably aiming for an acceptable visual representation, not trying to produce z-ordered components and a lost of any other random requirement people might come up with. The task is to show a pelican riding a bike, not produce a technical design that is layer order correct, after all. Also: > Near leg: from (300,245) to (356,408): thigh+shin as a single slightly bent line: M300,245 C 310,320 330,370 354,406. Stroke #F2953F width 12, linecap round. Far leg: from (320,250) to (404,452): M320,250 C 350,330 385,410 402,448. Stroke slightly darker #E0802F (behind, drawn before near leg but after bike? Pelican legs are in front of frame? Pelican is drawn after bike, so legs overlap the frame. The far leg ideally would be behind the frame, but in flat cartoon this is acceptable — or draw the far leg before the pelican body but after bike; overlaps the red frame.

walrus01 2mo ago

What's the result if you ask it for a "pelican equipment case" ? I've been trying the anti-bicycle pelican on some LLMs and the results are much more varied than the bicycle prompt. Some have very different ideas of pelicans (you'll get a small case with one DSLR camera in it, or a long rifle case, etc). You'll also get cases that are isometric view, or flat plane view from the front, or open or closed.

simonw 2mo ago
It did a pretty outstanding job on that: https://tools.simonwillison.net/markdown-svg-renderer.html#url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F137c124beae157953e60c48d1c710cef https://tools.simonwillison.net/markdown-svg-renderer.html#u... This took 56 minutes! - but it didn't have exclusive use of LM Studio, I ran a few other test prompts while it was working. llm -m lmstudio/qwen/qwen3.8-27b 'Generate an SVG of a pelican equipment case'
- walrus01 2mo ago
  That is pretty good! It's interesting it chose orange, because that's actually a somewhat rare color for pelican cases. I would say 95% of the ones going around the world right now are either black or FDE (flat dark earth) tan colored. There are some orange ones, for sure, if you google image search "orange pelican case", people use them for emergency/first aid supplies or electronics. But only the smallest ones are available in orange. It is also worth noting that it shaded the pelican case like it was a product photo taken in a studio, there's a virtual light source above and to the top left corner of it, then shading deeper color of orange as the body area progresses towards the lower right corner.
  - pbhjpbhj 2mo ago
    What are the chances that the colouring of actual pelicans is bleeding in to the prompt. Seems it could be that?
    - walrus01 2mo ago

geniium 2mo ago

I think the model got trained only on pelican to get there

walrus01 2mo ago
We should start asking it for an Albatross instead.

lacoolj 2mo ago

Totally agree. I actually didn't know you already got to it and I ran it on mine. It even added a scarf lol https://imgur.com/a/IgYKnmk https://imgur.com/a/IgYKnmk Took almost 90 minutes on my Threadripper Pro, and feels well worth it. 28k tokens from the Q8 model

HanClinto 2mo ago
And yours got the chain!

mococa 2mo ago

See, like my comment before. We don’t need the biggest/highest/frotiest model.

distrill 2mo ago

this sort of test surely only works when it's a brand new concept/prompt. like the exact same models trained newer data will surely get better at putting this together over time

notnullorvoid 2mo ago
Try it out with your own prompt, I suspect you'll be surprised.

notnullorvoid 2mo ago

I've done a few variations, I've been impressed with all of them. My favourite so far has been "Generate an SVG of a turtle flying a kite", result: https://imgur.com/a/bdKJPV4 https://imgur.com/a/bdKJPV4. Some will say conflating flying and flying a kite is bad, but I like it.

sahadi 2mo ago

what type of laptop runs something like that? using mac or egpu?

gerdesj 2mo ago

I was just updating a Proxmox cluster from v8 to v9 and had time on my hands and 15-30 mins after the models appeared for general download I had an initial pelican effort in .svg format. I switch off reasoning by default and for starters I ran 3.8 the same way as 3.6 on vllm on a DGX Spark box. Open web UI prompted. My word! It looks rather decent. Much better than previous efforts.

wincy 2mo ago

Not too bad, on my 5090 I got ~160tokens/sec and it took 3m20s to generate a pelican using Ninfer which someone else mentioned. I don't know why I didn't get any sweet sweet seagulls in my background, even on xhigh reasoning effort. https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632afdb5 https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632a...

redox99 2mo ago

It's definitely trained on generating SVGs. Not that they trained on the pelican prompt itself, but probably a more general RL step that has it generate SVG and a stronger vision model judges it. It's up to you if that counts as benchmaxxed.

nl 2mo ago
Generation of SVGs is (obviously) a generally useful task.

kamranjon 2mo ago

Have you thought about running a second tier of the Pelican benchmark where you see which model makes best pelican on lowest or no reasoning settings? I think that'd be pretty interesting and might help highlight which models have a baseline capability - even within those I'd imagine the token usage would vary wildly and might give some indications on verbosity as well.

beltsazar 2mo ago

Using a better quant Q6_K (Unsloth's) compared to your quant Q4_K_M (LM Studio's) yields this: https://tools.simonwillison.net/markdown-svg-renderer#url=https%3A%2F%2Fgist.github.com%2Fdanieljl%2F0a942c0a6499d42be099117efe9a76cc https://tools.simonwillison.net/markdown-svg-renderer#url=ht... Chains exist. Red scarf is proactively added. ("Maybe a scarf blowing in the wind for charm!) No hands/wings, though. Generated 30.2k tokens in total and took 52 mins on M5 Pro in low power mode (it will possibly take less than half of that in auto energy mode).

frumiousirc 2mo ago

> Bicycle is the right shape. Pelican beak is excellent. Nice background. Relative to other results I agree. But on an absolute measure, there is not a single element in the current bicycle that is real-world accurate and many elements are omitted or non-physical (eg, the transparent seat tube top, entire lack of a head tube). Consider a series of followup benchmarks. With a fresh context of the LLM under test, ask it to generate a list of findings for how the pelican-on-a-bicycle SVG that was produced is inaccurate compared what the real world scene might appear, accepting for the limitations of SVG as a medium. Then, feed back the list of findings to the original context for a second try. The benchmark can stop here by humans looking at the result and forming their own conclusion. Next phase is to repeat the analysis phase using the 2nd context to determine what findings were satisfied and what new inaccuracies are found. These two differences can form a second benchmark. Last phase is to iterate with the goal to drive the number of findings to zero.

maldie 2mo ago

Even with "low" thinking effort and Q4 it makes reasonable pelican, only 4k tokens and half a minute on 4090: https://imgur.com/a/yd5rNCp https://imgur.com/a/yd5rNCp

KerrAvon 2mo ago

If I'm doing the math right, that's like 17 t/s? I haven't played with Qwen 3.8 yet, but that seems really slow for a 27B on an M5 Max with sufficient RAM to hold it in memory.

latentsea 2mo ago

I asked it to make an animated svg of a pelican doing tricks on a bmx bike and it did. I then asked it to turn that into a browser game. This model fucks.

nullbio 2mo ago

Ask it to do a zebra on a jetski to escape the benchmax.