4 ms·

Improving image generation with better captions [pdf]

126 points by alievk 3y ago

alievk 3y ago
Core takeaways: - major improvement is from detailed image captioning - they trained an image captioning model to produce short and detailed captions - they use T5 text encoder - they use GPT-4 to "upsample" short user prompts - they train a U-net decoder and distill it to 2 denoising steps - text rendering is still unreliable, they believe the model has hard time mapping word tokens into letters in image
- GaggiX 3y ago
  They have only distilled the diffusion model used as a replacement for the VAE decoder, not the main diffusion model.
  - alievk 3y ago
    So the latent to pixel space conversion is made with another diffusion model? This is kind of refiner used in SDXL?
    - GaggiX 3y ago
      >So the latent to pixel space conversion is made with another diffusion model? Yup >This is kind of refiner used in SDXL? The refiner only works on the latent.
vunderba 3y ago
It's pretty unreal how much better DALL-E 3 is at adhering to prompt accuracy versus its largest for-profit competitor, Midjourney. Quality-wise I think MJ still has the edge, but if it takes 2000 v-rolls to get there (assuming you can at all), then I'd say MJ has a steep hill to climb.
- astrange 3y ago
  If you haven't tried Ideogram or DeepFloyd, those are better yet at the specific case of "writing verbatim text in your prompt". To the point Ideogram's trending page is entirely taken over by images of Latina women's names with bling effects and Disney characters last I looked. DALLE3 is definitely amazingly higher quality but I still feel it's kind of… useless. It's too hard to control in conversation form, because it's not a multimodal LLM but rather just works by rewriting text prompts. ChatGPT doesn't really know what dalle can and can't do, the actual dalle model still frequently fights you and just generates whatever it wants, when it generates prompts it sometimes leaves out or writes conflicting details, and it writes every prompt in "friendly harmless AI voice" so there's superfluous hype adjectives. This is odd because GPT4V actually is a multimodal model, and asking it to describe images as text works really well IME. Common failure modes I saw playing with it: * if you set something in eg "France", ChatGPT will rewrite it to be diverse and inclusive by literally saying "diverse people" a lot, but then also adds a lot of super-stereotypical details, and then dalle ignores half of it. So you get sometimes-diverse French people who are all wearing striped shirts and berets in front of the Eiffel Tower. * You can have a conversation with it and tell it to edit images, which it does by writing new prompts, but it's dumb and thinks the prompts are part of the conversation. So a second round prompt will sometimes leave out all the details and just say "The girl from before" and produce a useless image. * Composition is usually boring (very center-aligned and symmetrical) and if you try to control it by using words like "camera", it will put cameras in the picture half the time. * Typical AI failure modes: "anime" generates a kind of commercial/fanart style that doesn't exist rather than frames from a TV show. You can't generate things if they're hard to explain in English, like "a keyboard whose keys are house keys". But it's really good at putting things in Minecraft.
  - thom 3y ago
    I recognise some of what you're saying here. I've used DALL-E 3 in anger recently and while the results have really impressed me, every time I've tried to actually tweak and improve something I've ended up getting further and further away from what I want.
  - gzer0 3y ago
gzer0 3y ago
> During testing, we have noticed that this capability is unreliable as words are have missing or extra characters. We suspect this may have to do with the T5 text encoder we used: when the model encounters text in a prompt, it actually sees tokens that represent whole words and must map those to letters in an image. Ah, interesting remark regarding text rendering.
- ronsor 3y ago
  This isn't a new problem. A Google paper from late 2022 mentioned this fact; it went away when they used a byte/character-level text encoder (vs. tokenizing text first).
  - mycall 3y ago
    It still is an amazing state machine.
  - ollin 3y ago
    "Character-Aware Models Improve Visual Text Rendering" https://arxiv.org/abs/2212.10562 https://arxiv.org/abs/2212.10562 for anyone curious
  - tobr 3y ago
    This is an aside, but in general, aren’t tokens a pretty bad hack to improve context length? I don’t know enough about the theory to say, but it seems like it should make all kinds of things much more fragile, like how well it can understand misspellings, or languages that don’t map well to the tokens available.
liuliu 3y ago
Just want to call out this is fresh from OpenAI that actually talks about data preparation, their models (to some extents) and some negative learnings. Still not as complete as what's from their peers (Stability AI, Meta, adept.ai etc). Probably it just shows their core focus is on LLM or there is a change of heart?
amanzi 3y ago
I re-generated some of these images using the same prompts. It's truly amazing how good some of the artwork is. Most AI-generated images I've seen look good on the surface, but when you look closer there are obvious blemishes and weird things going on. But I've generated a bunch of images in Dall-E 3 and have been amazed at the results. e.g. my version of the "fierce garden gnome warrior" in widescreen: https://1drv.ms/i/s!Avl8TQnojpIQrsRCFFRQQDpNcyNKNw?e=VAZfVe https://1drv.ms/i/s!Avl8TQnojpIQrsRCFFRQQDpNcyNKNw?e=VAZfVe
totetsu 3y ago
I have to say that the GPT4 Dalle plugin is very good at making what you ask for. I was a bit disappointed to lose access to the new model directly through the dalle interface, because i enjoyed pushing its limits to make wierd images, But its amazing to be able to do things like, say "hey lets come up with a yaml schema to describe an outfit, and then make me some images of that" and get very very consistent results. When I was writing prompts myself I felt I was more just jumping into the soup image vector space.
visarga 3y ago
When I used GPT4-Image to get a full detailed description of an image, I was sure it was going to be used as a method for generating training data.
- cocoflunchy 3y ago
  I just tried this and wow, I'm impressed. Input / output: https://imgur.com/a/nTSkJUK https://imgur.com/a/nTSkJUK
  - danielbln 3y ago
    The images seem to be broken.
aargh_aargh 3y ago
Here you can try part of the process yourself (image to text): https://huggingface.co/spaces/fffiloni/CLIP-Interrogator-2 https://huggingface.co/spaces/fffiloni/CLIP-Interrogator-2
- Jackson__ 3y ago
  Well, not quite. The linked hf space uses the ViT-H-14 OpenCLIP model, which was trained on the Laion-2B dataset[0], which I'd categorize as fitting the reports description of "noisy and inaccurate image captions" perfectly. [0] https://laion.ai/blog/large-openclip/ https://laion.ai/blog/large-openclip/
  - aargh_aargh 3y ago
    I see, so they relabelled a smaller (more representative) subset for the captioner manually but more diligently, then used that to relabel their large set (analogous to LAION) with more descriptive captions, then trained DALL-E 3 on that. By the way, in the case of CLIP-Interrogator-2, I wonder how they came up with their wordlists (included in the repo). Are they just all unique terms from OpenCLIP?
forgingahead 3y ago
Nice to see this - also, looks like one of the primary authors is jbetker, who built the best open source TTS model that I've seen yet (TorToiSe): https://nonint.com/2023/09/23/dall-e-3/ https://nonint.com/2023/09/23/dall-e-3/ https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts
- danielbln 3y ago
  Dang, they went far. Good for them.
throwawy18231 3y ago
The first image they put is a furry. They know that the biggest target audience is furries commissioning images to artists
devinprater 3y ago
See? Accessibility improves everything, even LLM's. Captions/Alt-text are great, even for this. :)

126 points by alievk 3y ago

Core takeaways: - major improvement is from detailed image captioning - they trained an image captioning model to produce short and detailed captions - they use T5 text encoder - they use GPT-4 to "upsample" short user prompts - they train a U-net decoder and distill it to 2 denoising steps - text rendering is still unreliable, they believe the model has hard time mapping word tokens into letters in image

GaggiX 3y ago
They have only distilled the diffusion model used as a replacement for the VAE decoder, not the main diffusion model.
- alievk 3y ago
  So the latent to pixel space conversion is made with another diffusion model? This is kind of refiner used in SDXL?
  - GaggiX 3y ago
    >So the latent to pixel space conversion is made with another diffusion model? Yup >This is kind of refiner used in SDXL? The refiner only works on the latent.

vunderba 3y ago

It's pretty unreal how much better DALL-E 3 is at adhering to prompt accuracy versus its largest for-profit competitor, Midjourney. Quality-wise I think MJ still has the edge, but if it takes 2000 v-rolls to get there (assuming you can at all), then I'd say MJ has a steep hill to climb.

astrange 3y ago
If you haven't tried Ideogram or DeepFloyd, those are better yet at the specific case of "writing verbatim text in your prompt". To the point Ideogram's trending page is entirely taken over by images of Latina women's names with bling effects and Disney characters last I looked. DALLE3 is definitely amazingly higher quality but I still feel it's kind of… useless. It's too hard to control in conversation form, because it's not a multimodal LLM but rather just works by rewriting text prompts. ChatGPT doesn't really know what dalle can and can't do, the actual dalle model still frequently fights you and just generates whatever it wants, when it generates prompts it sometimes leaves out or writes conflicting details, and it writes every prompt in "friendly harmless AI voice" so there's superfluous hype adjectives. This is odd because GPT4V actually is a multimodal model, and asking it to describe images as text works really well IME. Common failure modes I saw playing with it: * if you set something in eg "France", ChatGPT will rewrite it to be diverse and inclusive by literally saying "diverse people" a lot, but then also adds a lot of super-stereotypical details, and then dalle ignores half of it. So you get sometimes-diverse French people who are all wearing striped shirts and berets in front of the Eiffel Tower. * You can have a conversation with it and tell it to edit images, which it does by writing new prompts, but it's dumb and thinks the prompts are part of the conversation. So a second round prompt will sometimes leave out all the details and just say "The girl from before" and produce a useless image. * Composition is usually boring (very center-aligned and symmetrical) and if you try to control it by using words like "camera", it will put cameras in the picture half the time. * Typical AI failure modes: "anime" generates a kind of commercial/fanart style that doesn't exist rather than frames from a TV show. You can't generate things if they're hard to explain in English, like "a keyboard whose keys are house keys". But it's really good at putting things in Minecraft.
- thom 3y ago
  I recognise some of what you're saying here. I've used DALL-E 3 in anger recently and while the results have really impressed me, every time I've tried to actually tweak and improve something I've ended up getting further and further away from what I want.
- gzer0 3y ago

gzer0 3y ago

> During testing, we have noticed that this capability is unreliable as words are have missing or extra characters. We suspect this may have to do with the T5 text encoder we used: when the model encounters text in a prompt, it actually sees tokens that represent whole words and must map those to letters in an image. Ah, interesting remark regarding text rendering.

ronsor 3y ago
This isn't a new problem. A Google paper from late 2022 mentioned this fact; it went away when they used a byte/character-level text encoder (vs. tokenizing text first).
- mycall 3y ago
  It still is an amazing state machine.
- ollin 3y ago
  "Character-Aware Models Improve Visual Text Rendering" https://arxiv.org/abs/2212.10562 https://arxiv.org/abs/2212.10562 for anyone curious
- tobr 3y ago
  This is an aside, but in general, aren’t tokens a pretty bad hack to improve context length? I don’t know enough about the theory to say, but it seems like it should make all kinds of things much more fragile, like how well it can understand misspellings, or languages that don’t map well to the tokens available.

liuliu 3y ago

Just want to call out this is fresh from OpenAI that actually talks about data preparation, their models (to some extents) and some negative learnings. Still not as complete as what's from their peers (Stability AI, Meta, adept.ai etc). Probably it just shows their core focus is on LLM or there is a change of heart?

amanzi 3y ago

I re-generated some of these images using the same prompts. It's truly amazing how good some of the artwork is. Most AI-generated images I've seen look good on the surface, but when you look closer there are obvious blemishes and weird things going on. But I've generated a bunch of images in Dall-E 3 and have been amazed at the results. e.g. my version of the "fierce garden gnome warrior" in widescreen: https://1drv.ms/i/s!Avl8TQnojpIQrsRCFFRQQDpNcyNKNw?e=VAZfVe https://1drv.ms/i/s!Avl8TQnojpIQrsRCFFRQQDpNcyNKNw?e=VAZfVe

totetsu 3y ago

I have to say that the GPT4 Dalle plugin is very good at making what you ask for. I was a bit disappointed to lose access to the new model directly through the dalle interface, because i enjoyed pushing its limits to make wierd images, But its amazing to be able to do things like, say "hey lets come up with a yaml schema to describe an outfit, and then make me some images of that" and get very very consistent results. When I was writing prompts myself I felt I was more just jumping into the soup image vector space.

visarga 3y ago

When I used GPT4-Image to get a full detailed description of an image, I was sure it was going to be used as a method for generating training data.

cocoflunchy 3y ago
I just tried this and wow, I'm impressed. Input / output: https://imgur.com/a/nTSkJUK https://imgur.com/a/nTSkJUK
- danielbln 3y ago
  The images seem to be broken.

aargh_aargh 3y ago

Here you can try part of the process yourself (image to text): https://huggingface.co/spaces/fffiloni/CLIP-Interrogator-2 https://huggingface.co/spaces/fffiloni/CLIP-Interrogator-2

Jackson__ 3y ago
Well, not quite. The linked hf space uses the ViT-H-14 OpenCLIP model, which was trained on the Laion-2B dataset[0], which I'd categorize as fitting the reports description of "noisy and inaccurate image captions" perfectly. [0] https://laion.ai/blog/large-openclip/ https://laion.ai/blog/large-openclip/
- aargh_aargh 3y ago
  I see, so they relabelled a smaller (more representative) subset for the captioner manually but more diligently, then used that to relabel their large set (analogous to LAION) with more descriptive captions, then trained DALL-E 3 on that. By the way, in the case of CLIP-Interrogator-2, I wonder how they came up with their wordlists (included in the repo). Are they just all unique terms from OpenCLIP?

forgingahead 3y ago

Nice to see this - also, looks like one of the primary authors is jbetker, who built the best open source TTS model that I've seen yet (TorToiSe): https://nonint.com/2023/09/23/dall-e-3/ https://nonint.com/2023/09/23/dall-e-3/ https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts

danielbln 3y ago
Dang, they went far. Good for them.

throwawy18231 3y ago

The first image they put is a furry. They know that the biggest target audience is furries commissioning images to artists

devinprater 3y ago

See? Accessibility improves everything, even LLM's. Captions/Alt-text are great, even for this. :)