4 ms·

This is a very good pelican. I'm really looking forward to trying out Gemini 3 myself. https://x.com/cannn064/status/1978779247930953885 https://x.com/cannn064/

by simonw 1y ago

This is a very good pelican. I'm really looking forward to trying out Gemini 3 myself. https://x.com/cannn064/status/1978779247930953885 https://x.com/cannn064/status/1978779247930953885

jacquesm 1y ago
That's good? Looks like complete crap to me.
- recallingmemory 1y ago
  Have you seen the current SVG art that LLMs generate? It's pretty comical what they output.
- OtherShrezzing 1y ago
  I like the pelican riding a bike test, but my standards for what’s “good” seem higher than generally expected by others. The models can generate hyper realistic renders of pelicans riding bikes in png format. They also have perfect knowledge of the SVG spec, and comprehensive knowledge of most human creative artistic endeavours. They should be able to produce astonishing results for the request. I don’t want to see a chunky icon-styled vector graphic. I want to see one of these models meticulously paint what is unambiguously a pelican riding what is unambiguously a bicycle, to a quality on-par with Michelangelo, using the SVG standard as a medium. And I don’t just want it to define individual pixels. I want brush strokes building up a layered and textured birds wing.
  - scrollaway 1y ago
    It’s not true agi until it can recreate the emotional state of Van Gogh when he cut his ear and express the pain through the brush, in svg format.
  - paintbox 1y ago
    >I like the pelican riding a bike test, but my standards for what’s “good” seem higher than generally expected by others. If you train for your first marathon, is your goal to run it under 2h? We are all looking forward to perfect results, but our standards are reasonable. We know what the results were last month, and judge the improvement velocity. Nobody thinks that's a good SVG of a pelican riding a bike - on it's own. But it's a lot better compared to all the other LLM-generated SVGs of a pelican riding a bike. We judge relative results - you judge absolute results. Confusion ensues.
    - OtherShrezzing 1y ago
      I think you’re missing the criticism I’m making. The models already have the capacity both to create hyper-real imagery, and they have mastery of the SVG medium. These two capabilities are the entire recipe a human would need to produce what I’ve described. To use your marathon metaphor, they have the body of Kipchoge in his absolute prime, and are failing to qualify for a local fun-run.
  - fkyoureadthedoc 1y ago
    But you're never going to get that out of the prompt that is being used to generate these Pelicans. You're judging it on something that's not even being attempted.
- simonw 1y ago
  Here's my collection from the past year. It's definitely better than any of these! https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/
  - jacquesm 1y ago
    Ok, so we're in the dancing pig stage now. We appreciate that the pig can dance, not how well it dances.
    - afro88 1y ago
      It's quite literally the opposite. Simon is tracking how well the "pig" dances as each model gets better (or worse) at it
- jstanley 1y ago
  I was confused too at first. This is an SVG generated by an LLM - it's not from an image model. How well do you reckon you could draw a pelican on a bicycle by typing out an SVG file blind?
  - aabhay 1y ago
    I mean how well do you reckon you can denoise a jpg by hand until its a piece of art? That way of thinking isn’t helpful to understanding AI IMO
    - int_19h 1y ago
      In this case it is actually relevant. The ability to draw a pelican on a bicycle correctly depends a great deal on understanding not only what both look like in general, but on the spatial relationships between the various objects and their parts. Models that can draw this kind of thing better also tend to be better at tasks that require understanding of how things go together and interact in 3D space.
      - bgwalter 1y ago
        How do we know it's not just a mashup of existing pictures? All generated pelicans on bikes look somewhat cartoonish and use historical or artsy bikes. This is training material from 2015: https://www.behance.net/gallery/29122113/Pelican-on-bikes-with-his-children https://www.behance.net/gallery/29122113/Pelican-on-bikes-wi... There are other such images. Not an image model? How do we know that they don't convert all images to svg and train an LLM on it? How do we know that they do not cheat on this benchmark and route the query to an image model first?
        jstanley 1y ago
        "it's not impressive because they might have cheated" isn't a great argument.
        1y ago
__mharrison__ 1y ago
Benchmark is (finally) broken!
- machiaweliczny 1y ago
  Still doesn't understand physics as in that cover should be over the wheel, which should be easy if it used 2D space reasoning
ionwake 1y ago
holy smokes, i wasnt expecting the equivalent of a piece of art
torginus 1y ago
What about other animals, like a giraffe on a snowmobile?
deleted 1y ago
[deleted]

by simonw 1y ago

This is a very good pelican. I'm really looking forward to trying out Gemini 3 myself. https://x.com/cannn064/status/1978779247930953885 https://x.com/cannn064/status/1978779247930953885

jacquesm 1y ago

That's good? Looks like complete crap to me.

recallingmemory 1y ago
Have you seen the current SVG art that LLMs generate? It's pretty comical what they output.
OtherShrezzing 1y ago
I like the pelican riding a bike test, but my standards for what’s “good” seem higher than generally expected by others. The models can generate hyper realistic renders of pelicans riding bikes in png format. They also have perfect knowledge of the SVG spec, and comprehensive knowledge of most human creative artistic endeavours. They should be able to produce astonishing results for the request. I don’t want to see a chunky icon-styled vector graphic. I want to see one of these models meticulously paint what is unambiguously a pelican riding what is unambiguously a bicycle, to a quality on-par with Michelangelo, using the SVG standard as a medium. And I don’t just want it to define individual pixels. I want brush strokes building up a layered and textured birds wing.
- scrollaway 1y ago
  It’s not true agi until it can recreate the emotional state of Van Gogh when he cut his ear and express the pain through the brush, in svg format.
- paintbox 1y ago
  >I like the pelican riding a bike test, but my standards for what’s “good” seem higher than generally expected by others. If you train for your first marathon, is your goal to run it under 2h? We are all looking forward to perfect results, but our standards are reasonable. We know what the results were last month, and judge the improvement velocity. Nobody thinks that's a good SVG of a pelican riding a bike - on it's own. But it's a lot better compared to all the other LLM-generated SVGs of a pelican riding a bike. We judge relative results - you judge absolute results. Confusion ensues.
  - OtherShrezzing 1y ago
    I think you’re missing the criticism I’m making. The models already have the capacity both to create hyper-real imagery, and they have mastery of the SVG medium. These two capabilities are the entire recipe a human would need to produce what I’ve described. To use your marathon metaphor, they have the body of Kipchoge in his absolute prime, and are failing to qualify for a local fun-run.
- fkyoureadthedoc 1y ago
  But you're never going to get that out of the prompt that is being used to generate these Pelicans. You're judging it on something that's not even being attempted.
simonw 1y ago
Here's my collection from the past year. It's definitely better than any of these! https://simonwillison.net/tags/pelican-riding-a-bicycle/ https://simonwillison.net/tags/pelican-riding-a-bicycle/
- jacquesm 1y ago
  Ok, so we're in the dancing pig stage now. We appreciate that the pig can dance, not how well it dances.
  - afro88 1y ago
    It's quite literally the opposite. Simon is tracking how well the "pig" dances as each model gets better (or worse) at it
jstanley 1y ago
I was confused too at first. This is an SVG generated by an LLM - it's not from an image model. How well do you reckon you could draw a pelican on a bicycle by typing out an SVG file blind?
- aabhay 1y ago
  I mean how well do you reckon you can denoise a jpg by hand until its a piece of art? That way of thinking isn’t helpful to understanding AI IMO
  - int_19h 1y ago
    In this case it is actually relevant. The ability to draw a pelican on a bicycle correctly depends a great deal on understanding not only what both look like in general, but on the spatial relationships between the various objects and their parts. Models that can draw this kind of thing better also tend to be better at tasks that require understanding of how things go together and interact in 3D space.
    - bgwalter 1y ago
      How do we know it's not just a mashup of existing pictures? All generated pelicans on bikes look somewhat cartoonish and use historical or artsy bikes. This is training material from 2015: https://www.behance.net/gallery/29122113/Pelican-on-bikes-with-his-children https://www.behance.net/gallery/29122113/Pelican-on-bikes-wi... There are other such images. Not an image model? How do we know that they don't convert all images to svg and train an LLM on it? How do we know that they do not cheat on this benchmark and route the query to an image model first?
      - jstanley 1y ago
        "it's not impressive because they might have cheated" isn't a great argument.
        1y ago

__mharrison__ 1y ago

Benchmark is (finally) broken!

machiaweliczny 1y ago
Still doesn't understand physics as in that cover should be over the wheel, which should be easy if it used 2D space reasoning

ionwake 1y ago

holy smokes, i wasnt expecting the equivalent of a piece of art

torginus 1y ago

What about other animals, like a giraffe on a snowmobile?

deleted 1y ago

[deleted]