4 ms·

I wonder how many elements are copied verbatim from the input images. Obviously, it's all synthesized from existing images, but some of these have regions that

by skupig 5y ago

I wonder how many elements are copied verbatim from the input images. Obviously, it's all synthesized from existing images, but some of these have regions that are particularly suspect to me: https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bcfa5ac8da01b78b31003075b27173a0d4049fc03b60370c.png/w_1024 https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bcf... https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5c84b38dfe49730d29c3be81f744c7f4e333a046381f1e1a.png/w_1024 https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5c8... DALL-E definitely does not encode the mechanics of cloth and yet there are regions of perfectly consistent geometry and lighting in the clothing wrinkles. I have a strong suspicion once these models start getting into mainstream hands it's not going to be uncommon for people to find the input photos that were copied. What do we do when a popular image from DALL-E turns out to be an artist's existing creative work with a style transfer applied?

recuter 5y ago
The AI maximalists simply refuse to entertain this possibility. The most interesting thing about DALL-E is its understanding of the prompt, objects, concepts and scenes. Which has improved significantly but is still very far from perfect. The rest is a rorschach test that abstracts the flaws away. I'd be almost as impressed with it if it spewed out stick figures. In fact I wish it had an unstylized mode, so one can more clearly see and think about the way it composites things.
- simonh 5y ago
  Quite, focusing on the artistic merit or quality of the visual elements is missing the point. These are collages. What's interesting is the comprehension and interpretation of the prompts, the way the meaning of the visual elements is understood, and that they are composed in semantically appropriate ways. Having said that the way the visual elements are integrated together without painfully obvious seams or blurring is impressive.
  - recuter 5y ago
    Many of the images, which are not as cherry picked as the ones promoted when DALL-E 2 was announced, exhibit obvious seams when looking closely. Sometimes it works out and sometimes it doesn't. A lot of the ones that work out actually seem OKish because the obvious flaws could be considered forms of "artistic expression". In art there are no mistakes, only happy accidents.
- axg11 5y ago
  You're discounting the possibility that this is _all_ that intelligence is: understanding and composition. Arguably, DALL-E has already surpassed the average person's ability to create images. Sure, there are many creatives that could be a better job. The average person can't create anything that comes close though.
  - recuter 5y ago
    Yes. Yes, I am.
    - edanm 5y ago
      Ok, I'll bite. Why are you discounting that possibility?
Veedrac 5y ago
> DALL-E definitely does not encode the mechanics of cloth And nor do you, so how are you so sure at a glance that it's consistent? It's probably only perceptually convincing. Betting that it must be cheating because it's good is classic AI denial.
- hoseja 5y ago
  But you do, at least a little imperfect bit.
  - Veedrac 5y ago
    Not in a relevant sense. When you see cloth, they don't look right or wrong based on whether you ran a detailed mental simulation, but based on how the patterns conform to the object. Plausibility is much more important than precision. There is not much reason to expect DALL-E 2 should not to be able to encode that same rule of thumb.
quirino 5y ago
Doesn't this fall into the same category as DALL-E begin able to create reflections in water and smooth surfaces to the elements it adds to images? (Look specifically at the Flamingo examples on their website) I'm sure it doesn't encode the mechanics of reflections either, but considering all the other stuff it can do, I'm wouldn't be surprised if it could learn how those work. But I'm also really interested in the possibility of finding obvious traces of specific input images on the output. The fact that "The Emperor of the Galaxy, byzantine mosaic" generates something strongly resembling Jesus may be a sign of this. Maybe someone with more AI knowledge is able to explain how possible/likely that is.
TuringTest 5y ago
I think the words examples provide insight into how the image generation works at several levels. You can clearly see [1] how the glyphs are created by approximately merging several letters in one place to create a new symbol, while maintaining the overall structure of words and paragraphs within a well-composed layout in the page. I would say that the merit of the technique is not in generating new images, but in correctly associating the images and the concepts they represent, at the right level (art styles at the low-level pixel representation, and objects as recognizable high-level features). I agree that it will be common to recognize where some reused elements come from, in special for small concept domains or art styles; this image generation is largely making clever collages. However, at a sufficiently fine-grained level, it is not unlike how most human artists create art. [1] https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/855638801ffcab0188609438f8c5e20aae809d060078a505.png/w_1024 https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/855...
axg11 5y ago
I'm really confused about why you think this would even be a valid criticism. If I asked you to draw clothing, do you calculate cloth dynamics? No. You would immediately look for either a real life or image example, then you would draw using that as inspiration. Even if DALL-E was copying certain segments of an identically in order to compose new images, it is a huge advance. On top of that, much work has been done on GANs and other generative algorithms to investigate how similar output images are to training images.
popcube 5y ago
Starting a very long debate about whether human just another DALL-E or fighting on whether model just copy and change old art by some calculations... finally it just no one care, how can you catch so many people?

by skupig 5y ago

recuter 5y ago

The AI maximalists simply refuse to entertain this possibility. The most interesting thing about DALL-E is its understanding of the prompt, objects, concepts and scenes. Which has improved significantly but is still very far from perfect. The rest is a rorschach test that abstracts the flaws away. I'd be almost as impressed with it if it spewed out stick figures. In fact I wish it had an unstylized mode, so one can more clearly see and think about the way it composites things.

simonh 5y ago
Quite, focusing on the artistic merit or quality of the visual elements is missing the point. These are collages. What's interesting is the comprehension and interpretation of the prompts, the way the meaning of the visual elements is understood, and that they are composed in semantically appropriate ways. Having said that the way the visual elements are integrated together without painfully obvious seams or blurring is impressive.
- recuter 5y ago
  Many of the images, which are not as cherry picked as the ones promoted when DALL-E 2 was announced, exhibit obvious seams when looking closely. Sometimes it works out and sometimes it doesn't. A lot of the ones that work out actually seem OKish because the obvious flaws could be considered forms of "artistic expression". In art there are no mistakes, only happy accidents.
axg11 5y ago
You're discounting the possibility that this is _all_ that intelligence is: understanding and composition. Arguably, DALL-E has already surpassed the average person's ability to create images. Sure, there are many creatives that could be a better job. The average person can't create anything that comes close though.
- recuter 5y ago
  Yes. Yes, I am.
  - edanm 5y ago
    Ok, I'll bite. Why are you discounting that possibility?

Veedrac 5y ago

> DALL-E definitely does not encode the mechanics of cloth And nor do you, so how are you so sure at a glance that it's consistent? It's probably only perceptually convincing. Betting that it must be cheating because it's good is classic AI denial.

hoseja 5y ago
But you do, at least a little imperfect bit.
- Veedrac 5y ago
  Not in a relevant sense. When you see cloth, they don't look right or wrong based on whether you ran a detailed mental simulation, but based on how the patterns conform to the object. Plausibility is much more important than precision. There is not much reason to expect DALL-E 2 should not to be able to encode that same rule of thumb.

quirino 5y ago

Doesn't this fall into the same category as DALL-E begin able to create reflections in water and smooth surfaces to the elements it adds to images? (Look specifically at the Flamingo examples on their website) I'm sure it doesn't encode the mechanics of reflections either, but considering all the other stuff it can do, I'm wouldn't be surprised if it could learn how those work. But I'm also really interested in the possibility of finding obvious traces of specific input images on the output. The fact that "The Emperor of the Galaxy, byzantine mosaic" generates something strongly resembling Jesus may be a sign of this. Maybe someone with more AI knowledge is able to explain how possible/likely that is.

TuringTest 5y ago

I think the words examples provide insight into how the image generation works at several levels. You can clearly see [1] how the glyphs are created by approximately merging several letters in one place to create a new symbol, while maintaining the overall structure of words and paragraphs within a well-composed layout in the page. I would say that the merit of the technique is not in generating new images, but in correctly associating the images and the concepts they represent, at the right level (art styles at the low-level pixel representation, and objects as recognizable high-level features). I agree that it will be common to recognize where some reused elements come from, in special for small concept domains or art styles; this image generation is largely making clever collages. However, at a sufficiently fine-grained level, it is not unlike how most human artists create art. [1] https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/855638801ffcab0188609438f8c5e20aae809d060078a505.png/w_1024 https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/855...

axg11 5y ago

I'm really confused about why you think this would even be a valid criticism. If I asked you to draw clothing, do you calculate cloth dynamics? No. You would immediately look for either a real life or image example, then you would draw using that as inspiration. Even if DALL-E was copying certain segments of an identically in order to compose new images, it is a huge advance. On top of that, much work has been done on GANs and other generative algorithms to investigate how similar output images are to training images.

popcube 5y ago

Starting a very long debate about whether human just another DALL-E or fighting on whether model just copy and change old art by some calculations... finally it just no one care, how can you catch so many people?