9 ms·

Expressive text-to-image generation with rich text

89 points by plurby 3y ago

gorenb 3y ago
my god, i think midjourney and dalle should do this now
minimaxir 3y ago
A relatively functionally similar approach is prompt term weighting with libraries such as compel: https://github.com/damian0815/compel https://github.com/damian0815/compel Prompt weighting alone can fix undesired aspects of an output, especially with SDXL and its dual text encoders.
- jbhuang0604 3y ago
  We compared with three other prompt term weighting (aka font size) methods: - Prompt-to-Prompt - Parenthesis - Repeating Our results show that Prompt-to-Prompt would generate artifacts and other heuristic methods are not very effective. You can see some examples in Figure 21, 22, 23. https://arxiv.org/pdf/2304.06720.pdf https://arxiv.org/pdf/2304.06720.pdf
Der_Einzige 3y ago
I LOVE this. All of the techniques that they are showing have already existed for awhile in places like Automatic1111/ComfyUI or its extensions (i.e. regional prompting, attention weights). Having it connect so seamlessly with rich text is awesome and is a cool UI trick that might make normies notice it. Also, related, but NLP is extremely undertooled on the prompt engineering side. Most of the techniques here would work just fine on any LLM. If you don't believe me, read this: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...
- jbhuang0604 3y ago
  Thanks! BTW, we recently implemented the Automatic1111 extension so you can install and use it directly with A1111. https://github.com/songweige/sd-webui-rich-text https://github.com/songweige/sd-webui-rich-text
LASR 3y ago
How well does this work with LLMs? Anyone tried this? I am curious about the references and footnotes approach the most.
- simbolit 3y ago
  The prompt input fields of all LLMs I know are plaintext only.
simbolit 3y ago
I looked at this, and thought about it, and then I waited for an hour, and now I looked at it again, and I can't help but think this is useless. We can already weigh parts of prompts, we can already specify colors or styles for parts of the images. And even if we could not, none of this needs rich text. In the beginning I even think their comparisons are dishonest. They compare "plaintext" prompts with "rich text" prompts, but the rich text prompts contain more information. What? Like, seriously, who is surprised the following two prompts give different images? (1) "A girl with long hair sitting in a cafe, by a table with coffee on it, best quality, ultra detailed, dynamic pose." (2) "A girl with long [Richtext:orange] hair sitting in a cafe, by a table with coffee on it, best quality, ultra detailed, dynamic pose. [Footnote:The ceramic coffee cup with intricate design, a dance of earthy browns and delicate gold accents. The dark, velvety latte is in it.]" the worst part is "Font style indicates the styles of local regions". In the comparison with other methods section they actually have to specify in parentheses what each font means style-wise, because nobody knows and (let's be frank) nobody wants to learn. So why not just use these plaintext parentheses in the prompt? I really stopped myself from immediately posting my (rather negative) opinion, but after over an hour, it hasn't changed. As far as i can see, this isn't useful, rich text prompts are a gimmick.
- EL_Loco 3y ago
  I had the same thought. The gothic church one, for example. Why wouldn't I just write "A pink gothic church in the sunset" instead of writing "A gothic church" and then having to do the extra steps to turn the word "church" into pink? Of course, I'm very ignorant of the uses of such tech, so there's probably some usefulness in this.
  - 90-00-09 3y ago
    The value I see is in constructing more complex prompts. Agree with your example but could see myself using this feature for prompts with multiple objects/aspects that require specific details. Probably not much different from inlining all details, just a nice separation of concerns: you can describe the high level requirement first, and then add and tweak individual details.
90-00-09 3y ago
I like this idea. It could be handy to be able to focus on individual descriptions in complex prompts. Is this then mostly a "UI" feature that is being translated to a traditional prompt? (As a side note: using decorative typefaces was an unconvincing example.)
- jbhuang0604 3y ago
  The UI part is basically a way to organize the user's intention. In the backend, we develop method for extracting "token maps" (i.e., which spatial regions correspond to specific words) and use region-based diffusion to achieve these localized editing results. The second half of the video provides an overview of the method. https://www.youtube.com/watch?v=ihDbAUh0LXk https://www.youtube.com/watch?v=ihDbAUh0LXk
  - 90-00-09 3y ago
    TIL about region-based diffusion. Thanks for the context!
pugworthy 3y ago
I would love to experiment with the idea of font interpretation. People can and do anthropomorphize fonts, but then they have names with meanings which might or might not be useful. For example, I'm wondering if a prompt written in Comic Sans should be turned into a comic-style illustration, or does it come out as a simplistic and childish drawing? Is a gothic font meant to imply a style of architecture, old Germanic peoples, or goth music and style? See also https://design.tutsplus.com/articles/the-psychology-of-fonts--cms-34943 https://design.tutsplus.com/articles/the-psychology-of-fonts...
- simbolit 3y ago
  if i understand it correctly, this is an a-priori mapping from font to style, the ml model doesn't interpret the font. I might be wrong tho, the paper isn't super clear.
  - jbhuang0604 3y ago
    Yes, the particular font family was just a "label". We don't really interpret what that font style would look like.
atleastoptimal 3y ago
This is very cool, but it's gimmicky. All of the rich text could simply be a modifier before or after the word (such as an adjective or phrase). Given most LLM work is plain text, this benefit isn't as neatly transferable as prompt engineering.
- jbhuang0604 3y ago
  Thanks for the comment! Our method is model-agnostic. It can be easily adapted to any LLM (aka the text-encode) and any text-to-image models. For example, the method was originally tested in Stable Diffusion 1.4. But we can easily apply it to Stable Diffusion-XL (or any finetuned model like ANIMAGINE-XL) even though the new model has a different text encoder and U-Net weights.
- MrNeon 3y ago
  If you specify the color of an object in a Stable Diffusion prompt the odds of that color appearing somewhere else explode. Tools like Regional Prompter help a lot with this, and I think the 'region specification by prompt' mode works similarly to Rich Text prompting. Not even dall-e 3 is free from color/style leaking to other parts of the image yet.
PixelForg 3y ago
I'm impressed by the pixel art generation, will definitely try it.
- jbhuang0604 3y ago
  Thank you!
littlestymaar 3y ago
While I don't think the rich text thing is particularly useful, I'm very impressed by the approach, especially how it manages to change the resulting image in a way you can control (that is, without regenerating the whole thing and ending up with something with random undesirable changes). The stability of the overall image during local changes makes me think that maybe this could be a key to video generation (because the biggest problem with existing diffusion-based approach for video is their instability from frame to frame).

89 points by plurby 3y ago

my god, i think midjourney and dalle should do this now

A relatively functionally similar approach is prompt term weighting with libraries such as compel: https://github.com/damian0815/compel https://github.com/damian0815/compel Prompt weighting alone can fix undesired aspects of an output, especially with SDXL and its dual text encoders.

jbhuang0604 3y ago
We compared with three other prompt term weighting (aka font size) methods: - Prompt-to-Prompt - Parenthesis - Repeating Our results show that Prompt-to-Prompt would generate artifacts and other heuristic methods are not very effective. You can see some examples in Figure 21, 22, 23. https://arxiv.org/pdf/2304.06720.pdf https://arxiv.org/pdf/2304.06720.pdf

Der_Einzige 3y ago

I LOVE this. All of the techniques that they are showing have already existed for awhile in places like Automatic1111/ComfyUI or its extensions (i.e. regional prompting, attention weights). Having it connect so seamlessly with rich text is awesome and is a cool UI trick that might make normies notice it. Also, related, but NLP is extremely undertooled on the prompt engineering side. Most of the techniques here would work just fine on any LLM. If you don't believe me, read this: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865ca4bb328eb58faf https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

jbhuang0604 3y ago
Thanks! BTW, we recently implemented the Automatic1111 extension so you can install and use it directly with A1111. https://github.com/songweige/sd-webui-rich-text https://github.com/songweige/sd-webui-rich-text

LASR 3y ago

How well does this work with LLMs? Anyone tried this? I am curious about the references and footnotes approach the most.

simbolit 3y ago
The prompt input fields of all LLMs I know are plaintext only.

simbolit 3y ago

I looked at this, and thought about it, and then I waited for an hour, and now I looked at it again, and I can't help but think this is useless. We can already weigh parts of prompts, we can already specify colors or styles for parts of the images. And even if we could not, none of this needs rich text. In the beginning I even think their comparisons are dishonest. They compare "plaintext" prompts with "rich text" prompts, but the rich text prompts contain more information. What? Like, seriously, who is surprised the following two prompts give different images? (1) "A girl with long hair sitting in a cafe, by a table with coffee on it, best quality, ultra detailed, dynamic pose." (2) "A girl with long [Richtext:orange] hair sitting in a cafe, by a table with coffee on it, best quality, ultra detailed, dynamic pose. [Footnote:The ceramic coffee cup with intricate design, a dance of earthy browns and delicate gold accents. The dark, velvety latte is in it.]" the worst part is "Font style indicates the styles of local regions". In the comparison with other methods section they actually have to specify in parentheses what each font means style-wise, because nobody knows and (let's be frank) nobody wants to learn. So why not just use these plaintext parentheses in the prompt? I really stopped myself from immediately posting my (rather negative) opinion, but after over an hour, it hasn't changed. As far as i can see, this isn't useful, rich text prompts are a gimmick.

EL_Loco 3y ago
I had the same thought. The gothic church one, for example. Why wouldn't I just write "A pink gothic church in the sunset" instead of writing "A gothic church" and then having to do the extra steps to turn the word "church" into pink? Of course, I'm very ignorant of the uses of such tech, so there's probably some usefulness in this.
- 90-00-09 3y ago
  The value I see is in constructing more complex prompts. Agree with your example but could see myself using this feature for prompts with multiple objects/aspects that require specific details. Probably not much different from inlining all details, just a nice separation of concerns: you can describe the high level requirement first, and then add and tweak individual details.

90-00-09 3y ago

I like this idea. It could be handy to be able to focus on individual descriptions in complex prompts. Is this then mostly a "UI" feature that is being translated to a traditional prompt? (As a side note: using decorative typefaces was an unconvincing example.)

jbhuang0604 3y ago
The UI part is basically a way to organize the user's intention. In the backend, we develop method for extracting "token maps" (i.e., which spatial regions correspond to specific words) and use region-based diffusion to achieve these localized editing results. The second half of the video provides an overview of the method. https://www.youtube.com/watch?v=ihDbAUh0LXk https://www.youtube.com/watch?v=ihDbAUh0LXk
- 90-00-09 3y ago
  TIL about region-based diffusion. Thanks for the context!

pugworthy 3y ago

I would love to experiment with the idea of font interpretation. People can and do anthropomorphize fonts, but then they have names with meanings which might or might not be useful. For example, I'm wondering if a prompt written in Comic Sans should be turned into a comic-style illustration, or does it come out as a simplistic and childish drawing? Is a gothic font meant to imply a style of architecture, old Germanic peoples, or goth music and style? See also https://design.tutsplus.com/articles/the-psychology-of-fonts--cms-34943 https://design.tutsplus.com/articles/the-psychology-of-fonts...

simbolit 3y ago
if i understand it correctly, this is an a-priori mapping from font to style, the ml model doesn't interpret the font. I might be wrong tho, the paper isn't super clear.
- jbhuang0604 3y ago
  Yes, the particular font family was just a "label". We don't really interpret what that font style would look like.

atleastoptimal 3y ago

This is very cool, but it's gimmicky. All of the rich text could simply be a modifier before or after the word (such as an adjective or phrase). Given most LLM work is plain text, this benefit isn't as neatly transferable as prompt engineering.

jbhuang0604 3y ago
Thanks for the comment! Our method is model-agnostic. It can be easily adapted to any LLM (aka the text-encode) and any text-to-image models. For example, the method was originally tested in Stable Diffusion 1.4. But we can easily apply it to Stable Diffusion-XL (or any finetuned model like ANIMAGINE-XL) even though the new model has a different text encoder and U-Net weights.
MrNeon 3y ago
If you specify the color of an object in a Stable Diffusion prompt the odds of that color appearing somewhere else explode. Tools like Regional Prompter help a lot with this, and I think the 'region specification by prompt' mode works similarly to Rich Text prompting. Not even dall-e 3 is free from color/style leaking to other parts of the image yet.

PixelForg 3y ago

I'm impressed by the pixel art generation, will definitely try it.

jbhuang0604 3y ago
Thank you!

littlestymaar 3y ago

While I don't think the rich text thing is particularly useful, I'm very impressed by the approach, especially how it manages to change the resulting image in a way you can control (that is, without regenerating the whole thing and ending up with something with random undesirable changes). The stability of the overall image during local changes makes me think that maybe this could be a key to video generation (because the biggest problem with existing diffusion-based approach for video is their instability from frame to frame).