5 ms·

After doing some testing, Imagen 4 doesn't score any higher than Imagen 3 on my comparison chart, approximately ~60% prompt adherence accuracy. https://genai-s

After doing some testing, Imagen 4 doesn't score any higher than Imagen 3 on my comparison chart, approximately ~60% prompt adherence accuracy. https://genai-showdown.specr.net https://genai-showdown.specr.net

Onavo 1y ago
How do companies like https://icon.com https://icon.com do their image Gen if the existing SOTA for prompt adherence is so poor?
- peab 1y ago
  fine tuning and prompt techniques can go a long way. That + cherrypicking results
- yorwba 1y ago
  People who generate images for ads probably don't often need strict prompt adherence, just a random backdrop to slap a picture of their product on top of. The kind of thing they'd have used a stock image library for before. Also "create static + video ads that are 0-99% complete" suggests the performance is hit or miss.
  - AsmodiusVI 1y ago
    Exactly this. It just helps the foundation which doesn’t need specific details in most cases.
- htrp 1y ago
  multishot generation with discriminators
xixixao 1y ago
Awesome showcase! Fun descriptions. Are there similar sites?
- vunderba 1y ago
  Thanks! There are definitely other GenAI image comparison sites out there - but I found that the majority of them were more concerned with visual fidelity which IMHO is a less challenging problem than prompt adherence. This is probably one of the better known benchmarks but when I see Midjourney 7 and Imagen3 within spitting distance of each other it makes me question what kind of metrics they are using. https://artificialanalysis.ai/text-to-image https://artificialanalysis.ai/text-to-image
peab 1y ago
great website!
snug 1y ago
How do you determine how many attempts are made before the results are failing?
- mcphage 1y ago
  It's listed in Purple to the right of the model name.
  - zamadatix 1y ago
    I think they're asking how the number to stop at was determined, not what the number stopped at was. My guess as to determining whether it's 64 attempts to a pass for one and 5 attempts to a fail for another is simply "whether or not the author felt there was a chance random variance would result in a pass with a few more tries based on the initial 5ish". I.e. a bit subjective, as is the overall grading in the end anyways.
    - vunderba 1y ago
      That's exactly what it was. It's hard to define a discrete rubric for grading at an inherently qualitative level. Usually more attempts means that it seemed like the model had the "potential" to get across the finish line so I gave it more opportunities. If there's only a few attempts and ends in a failure, there's a pretty good chance that I could sort of tell that the model had ZERO chance.
woolion 1y ago
The winning image entry for "The Yarrctic Circle" by OpenAI 4o doesn't actually wields a cutlass. It's very aesthetically pleasing, even though it's so wrong in all fundamental aspects (perspective is nonsensical and anatomy is messed up, with one leg 150% longer than the other, ...). It's a very interesting resource to map some of the limits of existing models.
- NoahZuniga 1y ago
  I can't find the image you're talking about. Link pls?
- echelon 1y ago
  Google Flow is remarkable as video editing UX, but Imagen 4 doesn't really stand out amongst its image gen peers. I want to interrupt all of this hype over Imagen 4 to talk about the totally slept on Tencent Hunyuan Image 2.0 that stealthily launched last Friday. It's absolutely remarkable and features: - millisecond generation times - real time image-to-image drawing capabilities - visual instructivity (eg. you can circle regions, draw arrows, and write prompts addressing them.) - incredible prompt adherence and quality Nothing else on the market has these properties in quite this combination, so it's rather unique. Release Tweet: https://x.com/TencentHunyuan/status/1923263203825549457 https://x.com/TencentHunyuan/status/1923263203825549457 Tencent Hunyuan had a bunch of model releases all wrapped up in a product that they call "Hunyuan Game", but the Hunyuan Image 2.0 real time drawing canvas is the real star of it all. It's basically a faster, higher quality Krea: https://x.com/TencentHunyuan/status/1924713242150273424 https://x.com/TencentHunyuan/status/1924713242150273424 More real time canvas samples: https://youtu.be/tVgT42iI31c?si=WEuvie-fIDaGk2J6&t=141 https://youtu.be/tVgT42iI31c?si=WEuvie-fIDaGk2J6&t=141 (I haven't found any other videos on the internet apart from these two.) You can see how this is an incredible illustration tool. If they were to open source this, this would immediately become the top image generation model over Flux, Imagen 4, etc. At this point, really only gpt-image-1 stands apart as having godlike instructivity, but it's on the other end of the [real time <--> instructive] spectrum. A total creative image tool kit might just be gpt-image-1 and Hunyuan Image 2.0. The other models are degenerate cases. More image samples: https://x.com/Gdgtify/status/1923374102653317545 https://x.com/Gdgtify/status/1923374102653317545 If anyone from Tencent or the Hunyuan team is reading this: PLEASE, PLEASE, PLEASE OPEN SOURCE THIS. (PLEASE!!)
strongpigeon 1y ago
How can you tell you're using Imagen 4 and not Imagen 3? Gemini seems unable to tell me which model it's using. Are you using Vertex AI?
- EGreg 1y ago
  Tell me you’re using Imagen 3 without telling me you’re using Imagen 4… or something
- sidibe 1y ago
  Well they've labelled it 3/4 so I'm guessing they can't but you can use 4 it in whisk
- vunderba 1y ago
  I used Whisk. The model listing shows 3/4 because testing against Imagen 4 did not result in a measurable increase in accuracy from Imagen 3. https://labs.google/fx/tools/whisk https://labs.google/fx/tools/whisk
tintor 1y ago
More difficult examples: - wine glass that is full to the edge with wine (ie. not half full) - wrist watch not showing V (hands at 10 and 2 o'clock) - 9 step IKEA shelf assembly instruction diagram - any kind of gymnastics / sport acro
- viraptor 1y ago
  What's the reason to test the "not showing ..."? I've never seen anyone make that kind of request in real life. They ask for what they actually want instead. You'd ask for a clock showing 3:25 rather than "not 10:10". I mean, it's a fun edge case, but I'm practice - does it matter?
  - fragmede 1y ago
    > I mean, it's a fun edge case, but I'm practice - does it matter? *in practice, not I'm practice. (I swear I have a point, I'm not being needlessly pedantic.) In English, in images, mistakes stick out. Thus negative prompts are used a lot for iterative image generation. Even when you're working with a human graphics designer, you may not know what exactly you want, but you know that you don't want (some aspect of) the image in front of you. Ie: "Not that", for varying values of "that".
    - viraptor 1y ago
      > Thus negative prompts are used a lot for iterative image generation. Are they still? The negative keywords were popular in SD era. The negative prompt was popular with later models in advanced tools. But modern iterations look different - the models capable of editing are perfectly fine with processing the previous image with a prompt "remove the elephant" or "make the clock show a different time". Are the negative parts in the initial prompt still actually used in iteration?
  - tintor 1y ago
    Problem is that watchmakers always set the watches to show V with clock hands when they market their watches. This causes a very strong bias in image generation models, making it very difficult for them to generate watch that shows any other time, even if user requests it.
tintor 1y ago
Hands in Winning entry in "Not the Bees" are very unlike any driver. I wouldn't count it as a pass.
- vunderba 1y ago
  I hate to say it but I feel like as a result of staring at so many equivalents of Tyrone Rugen since the dark ages of Stable Diffusion 1.5 - I literally DID NOT EVEN notice that until you called it out. The training data in my wetware has been corrupted.
mcphage 1y ago
> "A dolphin is using its fluke to discipline a mermaid by paddling it across the backside." Hmm.
zamadatix 1y ago
I love the writing style in this.
bigmadshoe 1y ago
I'm curious why you decide to declare victory after one successful attempt, but try many times for unsuccessful models. Are you trying to measure whether a model _can_ get it right, or whether it frequently _does_ get it right? I feel like success rate is a better metric here, or at least a fixed number of trials with some success rate threshold to determine model success.
- ipnon 1y ago
  It is indicative of marginal improvements instead of new breakthroughs. iPhone 1 was a paradigm shift. iPhone 10 was essentially iPhone 9 with tweaks. As an AI optimist I would be disappointed to find we are already seeing diminishing returns on R&D.
  - sethaurus 1y ago
    At the risk of unbearable pedantry, there's never been an iPhone 9. (There was never a 2 either; there was kind of a 3, although it was really called the 3G.)
    - deleted 1y ago
      [deleted]
  - Uehreka 1y ago
    Not only did the iPhone 9 never exist, the iPhone X was a huge paradigm shift in design and capabilities. That was the phone that introduced edge-to-edge OLED screens to the iPhone line, as well as the IR camera that enabled FaceID and the first generation of Portrait mode. I know it well since it also introduced the ability for developers to build facial motion capture apps that would’ve previously required expensive pro hardware and allowed people like me to build live facial motion capture effects for theatre. Sorry to dunk so hard, but your example of technology stagnating is actually an example of breakthrough technological innovation deep into a product’s lifecycle: the very thing you were trying to say doesn’t happen.
    - 1y ago
mvdtnz 1y ago
The website is broken
- vunderba 1y ago
  That's unusual - I don't see anything in the logs and perf tests / website speed tests show everything is good. Maybe Cloudflare had a hiccup.
andybak 1y ago
Side note. It's my understanding that being a pith helmet is pretty orthogonal to having a spike. Plenty of helmets with spikes aren't pith helmets and plenty of pith helmets don't have spikes. Not sure if this affects your results or not but I resist chiming in!
- andybak 1y ago
  Also "Hippity Hop" is a Space Hopper! Wikipedia agrees with me: https://en.wikipedia.org/wiki/Space_hopper https://en.wikipedia.org/wiki/Space_hopper :) I wonder how much the commonality or frequency of names for things affects image generation? My hunch is that it it roughly correlates and you'd get better results for terms with more hits in the training data. I'd probably use Google image search as a rough proxy for this.
  - vunderba 1y ago
    well I'm about a billion years late to this but going to reply anyway. It's not mentioned but internally I give each model several "iterations" of the prompts themselves. This included using hippity hop, space hopper, and even a physical description of the toy itself. But it's a good call out!
anton-c 1y ago
They failed but man those snakes are cool. Awesome website!

Onavo 1y ago

How do companies like https://icon.com https://icon.com do their image Gen if the existing SOTA for prompt adherence is so poor?

peab 1y ago
fine tuning and prompt techniques can go a long way. That + cherrypicking results
yorwba 1y ago
People who generate images for ads probably don't often need strict prompt adherence, just a random backdrop to slap a picture of their product on top of. The kind of thing they'd have used a stock image library for before. Also "create static + video ads that are 0-99% complete" suggests the performance is hit or miss.
- AsmodiusVI 1y ago
  Exactly this. It just helps the foundation which doesn’t need specific details in most cases.
htrp 1y ago
multishot generation with discriminators

xixixao 1y ago

Awesome showcase! Fun descriptions. Are there similar sites?

vunderba 1y ago
Thanks! There are definitely other GenAI image comparison sites out there - but I found that the majority of them were more concerned with visual fidelity which IMHO is a less challenging problem than prompt adherence. This is probably one of the better known benchmarks but when I see Midjourney 7 and Imagen3 within spitting distance of each other it makes me question what kind of metrics they are using. https://artificialanalysis.ai/text-to-image https://artificialanalysis.ai/text-to-image

peab 1y ago

great website!

snug 1y ago

How do you determine how many attempts are made before the results are failing?

mcphage 1y ago
It's listed in Purple to the right of the model name.
- zamadatix 1y ago
  I think they're asking how the number to stop at was determined, not what the number stopped at was. My guess as to determining whether it's 64 attempts to a pass for one and 5 attempts to a fail for another is simply "whether or not the author felt there was a chance random variance would result in a pass with a few more tries based on the initial 5ish". I.e. a bit subjective, as is the overall grading in the end anyways.
  - vunderba 1y ago
    That's exactly what it was. It's hard to define a discrete rubric for grading at an inherently qualitative level. Usually more attempts means that it seemed like the model had the "potential" to get across the finish line so I gave it more opportunities. If there's only a few attempts and ends in a failure, there's a pretty good chance that I could sort of tell that the model had ZERO chance.

woolion 1y ago

The winning image entry for "The Yarrctic Circle" by OpenAI 4o doesn't actually wields a cutlass. It's very aesthetically pleasing, even though it's so wrong in all fundamental aspects (perspective is nonsensical and anatomy is messed up, with one leg 150% longer than the other, ...). It's a very interesting resource to map some of the limits of existing models.

NoahZuniga 1y ago
I can't find the image you're talking about. Link pls?
echelon 1y ago
Google Flow is remarkable as video editing UX, but Imagen 4 doesn't really stand out amongst its image gen peers. I want to interrupt all of this hype over Imagen 4 to talk about the totally slept on Tencent Hunyuan Image 2.0 that stealthily launched last Friday. It's absolutely remarkable and features: - millisecond generation times - real time image-to-image drawing capabilities - visual instructivity (eg. you can circle regions, draw arrows, and write prompts addressing them.) - incredible prompt adherence and quality Nothing else on the market has these properties in quite this combination, so it's rather unique. Release Tweet: https://x.com/TencentHunyuan/status/1923263203825549457 https://x.com/TencentHunyuan/status/1923263203825549457 Tencent Hunyuan had a bunch of model releases all wrapped up in a product that they call "Hunyuan Game", but the Hunyuan Image 2.0 real time drawing canvas is the real star of it all. It's basically a faster, higher quality Krea: https://x.com/TencentHunyuan/status/1924713242150273424 https://x.com/TencentHunyuan/status/1924713242150273424 More real time canvas samples: https://youtu.be/tVgT42iI31c?si=WEuvie-fIDaGk2J6&t=141 https://youtu.be/tVgT42iI31c?si=WEuvie-fIDaGk2J6&t=141 (I haven't found any other videos on the internet apart from these two.) You can see how this is an incredible illustration tool. If they were to open source this, this would immediately become the top image generation model over Flux, Imagen 4, etc. At this point, really only gpt-image-1 stands apart as having godlike instructivity, but it's on the other end of the [real time <--> instructive] spectrum. A total creative image tool kit might just be gpt-image-1 and Hunyuan Image 2.0. The other models are degenerate cases. More image samples: https://x.com/Gdgtify/status/1923374102653317545 https://x.com/Gdgtify/status/1923374102653317545 If anyone from Tencent or the Hunyuan team is reading this: PLEASE, PLEASE, PLEASE OPEN SOURCE THIS. (PLEASE!!)

strongpigeon 1y ago

How can you tell you're using Imagen 4 and not Imagen 3? Gemini seems unable to tell me which model it's using. Are you using Vertex AI?

EGreg 1y ago
Tell me you’re using Imagen 3 without telling me you’re using Imagen 4… or something
sidibe 1y ago
Well they've labelled it 3/4 so I'm guessing they can't but you can use 4 it in whisk
vunderba 1y ago
I used Whisk. The model listing shows 3/4 because testing against Imagen 4 did not result in a measurable increase in accuracy from Imagen 3. https://labs.google/fx/tools/whisk https://labs.google/fx/tools/whisk

tintor 1y ago

More difficult examples: - wine glass that is full to the edge with wine (ie. not half full) - wrist watch not showing V (hands at 10 and 2 o'clock) - 9 step IKEA shelf assembly instruction diagram - any kind of gymnastics / sport acro

viraptor 1y ago
What's the reason to test the "not showing ..."? I've never seen anyone make that kind of request in real life. They ask for what they actually want instead. You'd ask for a clock showing 3:25 rather than "not 10:10". I mean, it's a fun edge case, but I'm practice - does it matter?
- fragmede 1y ago
  > I mean, it's a fun edge case, but I'm practice - does it matter? *in practice, not I'm practice. (I swear I have a point, I'm not being needlessly pedantic.) In English, in images, mistakes stick out. Thus negative prompts are used a lot for iterative image generation. Even when you're working with a human graphics designer, you may not know what exactly you want, but you know that you don't want (some aspect of) the image in front of you. Ie: "Not that", for varying values of "that".
  - viraptor 1y ago
    > Thus negative prompts are used a lot for iterative image generation. Are they still? The negative keywords were popular in SD era. The negative prompt was popular with later models in advanced tools. But modern iterations look different - the models capable of editing are perfectly fine with processing the previous image with a prompt "remove the elephant" or "make the clock show a different time". Are the negative parts in the initial prompt still actually used in iteration?
- tintor 1y ago
  Problem is that watchmakers always set the watches to show V with clock hands when they market their watches. This causes a very strong bias in image generation models, making it very difficult for them to generate watch that shows any other time, even if user requests it.

tintor 1y ago

Hands in Winning entry in "Not the Bees" are very unlike any driver. I wouldn't count it as a pass.

vunderba 1y ago
I hate to say it but I feel like as a result of staring at so many equivalents of Tyrone Rugen since the dark ages of Stable Diffusion 1.5 - I literally DID NOT EVEN notice that until you called it out. The training data in my wetware has been corrupted.

mcphage 1y ago

> "A dolphin is using its fluke to discipline a mermaid by paddling it across the backside." Hmm.

zamadatix 1y ago

I love the writing style in this.

bigmadshoe 1y ago

I'm curious why you decide to declare victory after one successful attempt, but try many times for unsuccessful models. Are you trying to measure whether a model _can_ get it right, or whether it frequently _does_ get it right? I feel like success rate is a better metric here, or at least a fixed number of trials with some success rate threshold to determine model success.

ipnon 1y ago
It is indicative of marginal improvements instead of new breakthroughs. iPhone 1 was a paradigm shift. iPhone 10 was essentially iPhone 9 with tweaks. As an AI optimist I would be disappointed to find we are already seeing diminishing returns on R&D.
- sethaurus 1y ago
  At the risk of unbearable pedantry, there's never been an iPhone 9. (There was never a 2 either; there was kind of a 3, although it was really called the 3G.)
  - deleted 1y ago
    [deleted]
- Uehreka 1y ago
  Not only did the iPhone 9 never exist, the iPhone X was a huge paradigm shift in design and capabilities. That was the phone that introduced edge-to-edge OLED screens to the iPhone line, as well as the IR camera that enabled FaceID and the first generation of Portrait mode. I know it well since it also introduced the ability for developers to build facial motion capture apps that would’ve previously required expensive pro hardware and allowed people like me to build live facial motion capture effects for theatre. Sorry to dunk so hard, but your example of technology stagnating is actually an example of breakthrough technological innovation deep into a product’s lifecycle: the very thing you were trying to say doesn’t happen.
  - 1y ago

mvdtnz 1y ago

The website is broken

vunderba 1y ago
That's unusual - I don't see anything in the logs and perf tests / website speed tests show everything is good. Maybe Cloudflare had a hiccup.

andybak 1y ago

Side note. It's my understanding that being a pith helmet is pretty orthogonal to having a spike. Plenty of helmets with spikes aren't pith helmets and plenty of pith helmets don't have spikes. Not sure if this affects your results or not but I resist chiming in!

andybak 1y ago
Also "Hippity Hop" is a Space Hopper! Wikipedia agrees with me: https://en.wikipedia.org/wiki/Space_hopper https://en.wikipedia.org/wiki/Space_hopper :) I wonder how much the commonality or frequency of names for things affects image generation? My hunch is that it it roughly correlates and you'd get better results for terms with more hits in the training data. I'd probably use Google image search as a rough proxy for this.
- vunderba 1y ago
  well I'm about a billion years late to this but going to reply anyway. It's not mentioned but internally I give each model several "iterations" of the prompts themselves. This included using hippity hop, space hopper, and even a physical description of the toy itself. But it's a good call out!

anton-c 1y ago

They failed but man those snakes are cool. Awesome website!