4 ms·

I've been following the image generation field for a couple months now and while the answer to the title is "yes for most things" it is easily fixed. Use a bett

by nodja 4y ago

I've been following the image generation field for a couple months now and while the answer to the title is "yes for most things" it is easily fixed. Use a better text encoder. My favorite anecdote for showing how having a text encoder that actually understands the world is important to image generation is when querying for "barack obama" on a model trained on a dataset that has never seen Barack Obama the model somehow generates images of random black men wearing suits[1]. This is, in my non-expert opinion, a clear indication that the model's knowledge of the world is leaking through to the image generator. So if my understanding is right, as long as a concept can be represented properly in the text embeddings of a model, the image generation will be able to use that. If my anecdote doesn't convince you, consider that one of google's findings on the Imagen paper was that increasing the size of the text encoder had a much bigger effect on not only the quality of the image, but also have the image follow the prompt correctly, including having the image generator being able to spell words. I think the next big step in the text to image generation field, aside from the current efforts to optimize the diffusion models, will be to train an efficient text encoder that can generate high quality embeddings. [1] Results of querying "barack obama" to an early version of cene555's imagen reproduction effort. https://i.imgur.com/oUo3QdF.png https://i.imgur.com/oUo3QdF.png

snek_case 4y ago
> when querying for "barack obama" on a model trained on a dataset that has never seen Barack Obama the model somehow generates images of random black men wearing suits[1]. This is, in my non-expert opinion, a clear indication that the model's knowledge of the world is leaking through to the image generator. That's super interesting. It's not just black men in suits either. It's older black men, with the American flag in the background who look like they might be speaking. Clearly the model has a pretty in-depth knowledge of the context surrounding Barack Obama. I would say the image generation model is also doing a pretty great job at stitching those concepts together in a way that's coherent. It's not a random jumble. It's kind of what you would expect if you asked a human artist to draw a black American president.

by nodja 4y ago

snek_case 4y ago

> when querying for "barack obama" on a model trained on a dataset that has never seen Barack Obama the model somehow generates images of random black men wearing suits[1]. This is, in my non-expert opinion, a clear indication that the model's knowledge of the world is leaking through to the image generator. That's super interesting. It's not just black men in suits either. It's older black men, with the American flag in the background who look like they might be speaking. Clearly the model has a pretty in-depth knowledge of the context surrounding Barack Obama. I would say the image generation model is also doing a pretty great job at stitching those concepts together in a way that's coherent. It's not a random jumble. It's kind of what you would expect if you asked a human artist to draw a black American president.