3 ms·

I'd also say, every of these images would fail a reverse test (i.e., asking a person to describe the image and what it represents.) The task is not just about

I'd also say, every of these images would fail a reverse test (i.e., asking a person to describe the image and what it represents.) The task is not just about generating an image that may somehow be in accordance with the prompt, but also to generate a significant image. [Edit] The equivalent to a Turing test for compositional images would be something like this: have as set of 100 images with their respective prompts, some generated by an AI, some by a human graphic designer / artist; let the test person pick the images that were generated by a computer. Mind that this would not only involve the problem of compositionality per se, but also a meaningful and/or artistic composition of the image itself. Is someone attempting to express what is given in the prompt?

m00x 4y ago
They actually use the reverse test to train the generator, and to score which image is most relevant to the prompt from the many images given by the generator. Dall-E does this using the OpenAI CLIP model. You can see the mini version here using this exact logic https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-Mini-Explained--Vmlldzo4NjIxODA https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-Mini-E...
- masswerk 4y ago
  What I'm aiming at is about what is shown by an image, not what is in an image. Take for example the images for "A digital art picture of a robot child riding a llama with a bell on its tail through a desert", which Scott Alexander counts as a win. The first image actually shows a merry llama, a robot, which is unmistakably a robot child, riding the llama and it's clearly a desert scene. If we forget for a moment about the missing bell, it's probably the best picture. But it is also very blunt in composition. I can't imagine why anybody should have made this image. Maybe, if somebody approached a designer, like, "See, we have this wooden toy cube and need an illustration for this face of the cube. What about a cute picture of a robot child riding a llama with a bell on its tail through a desert?" – But, at closer inspection, there's something sinister going on: it's rather the llama that is leading the robot child by a rein, not the other way round. – I mean, this is meant to be a cute toy! And where is the bell? We need to talk about that contract again… The second image is undisclosed, so we can't really say anything about this. The third image is rather special. The llama seems to be robotic as well, the robot, which is – again – clearly a child, seems to be not only riding the llama, but both appear somehow integrated into a single unit, which cumulates in the robot child's face screen. There is an eerie feeling about this image. The fact, that the bell seems to be attached to the rein as some kind of link between the llama and its rider doesn't exactly help. (There's also a conic extrusion at the back of the llama, but I'd rather interpret this as part of the llama, and it's not attached to its tail.) The composition in its flat side view produces a tension focusing towards the left side of the frame, on something, which is not shown, but apparently a vital part of the story. While I might notice the mountain in the background, I'd probably forget to write home about the scene being set in the desert. But I would note that we're missing context to understand this image and what may be shown by it. The fourth image, finally, is clearly Star Wars, robot edition. However, no bell. ("A robot child riding a llama with a bell on its tail through a desert" – "Ah, you mean Star Wars!") I'm not even sure which of these images Alexander did pick as a winner. And I would describe neither image by the prompt, nor would I dare to imagine that a human had chosen these exact means to show what is described in the prompt. Having said that, thanks for the link to the DALL-E Mini paper!
t_mann 4y ago
I bet most humans would fail this test on images that everybody agrees are adequate portrayals. Answering a short query with an image is a highly non-injective mapping, you simply don't know what aspects of the scene were specifically asked for in the query, and which ones were filled in by the artist / AI. Eg, the queries "opening of a medieval theme park", "announcement of a witch trial", "king charles proclamation" might all be reasonably answered by similar images containing a small crowd and a speaker in a medieval-looking setting, even though they're not meant to refer to the same time periods or settings at all.
- masswerk 4y ago
  Mind that the test is meant to include the prompt/query. E.g., take one of Alexander's winners, the robot in the cathedral: a human would probably answer to the prompt by making the cathedral part of the subject, by investing some effort in pointing out that this is indeed a cathedral, instead of just conforming to the query by showing some ambigue bits in the background, which may or may not represent parts of a cathedral. The quest of the machine is still "create an image by allotting the elements provided in the query", not "compose an image showing this and that as the subject" – and there's a significant difference. Closing this gap would require the AI to form a concept of what is given as a subject in its entirety and then constructing a plausible scene around this, by a meaningful placement of the subject, which is clearly beyond the state of art. I admit that there is a certain appeal to those images, for their distinctive dreamlike quality, as there's often a specific tension in the rather blunt composition and an apparent subject, which seems to be beyond what is actually depicted, as if this was just a casually picked specimen from a series of illustrations for a broader story line. But I'd bet that we're going to have been seeing too much of this soon, in order to be still amazed.

m00x 4y ago

They actually use the reverse test to train the generator, and to score which image is most relevant to the prompt from the many images given by the generator. Dall-E does this using the OpenAI CLIP model. You can see the mini version here using this exact logic https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-Mini-Explained--Vmlldzo4NjIxODA https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-Mini-E...

masswerk 4y ago
What I'm aiming at is about what is shown by an image, not what is in an image. Take for example the images for "A digital art picture of a robot child riding a llama with a bell on its tail through a desert", which Scott Alexander counts as a win. The first image actually shows a merry llama, a robot, which is unmistakably a robot child, riding the llama and it's clearly a desert scene. If we forget for a moment about the missing bell, it's probably the best picture. But it is also very blunt in composition. I can't imagine why anybody should have made this image. Maybe, if somebody approached a designer, like, "See, we have this wooden toy cube and need an illustration for this face of the cube. What about a cute picture of a robot child riding a llama with a bell on its tail through a desert?" – But, at closer inspection, there's something sinister going on: it's rather the llama that is leading the robot child by a rein, not the other way round. – I mean, this is meant to be a cute toy! And where is the bell? We need to talk about that contract again… The second image is undisclosed, so we can't really say anything about this. The third image is rather special. The llama seems to be robotic as well, the robot, which is – again – clearly a child, seems to be not only riding the llama, but both appear somehow integrated into a single unit, which cumulates in the robot child's face screen. There is an eerie feeling about this image. The fact, that the bell seems to be attached to the rein as some kind of link between the llama and its rider doesn't exactly help. (There's also a conic extrusion at the back of the llama, but I'd rather interpret this as part of the llama, and it's not attached to its tail.) The composition in its flat side view produces a tension focusing towards the left side of the frame, on something, which is not shown, but apparently a vital part of the story. While I might notice the mountain in the background, I'd probably forget to write home about the scene being set in the desert. But I would note that we're missing context to understand this image and what may be shown by it. The fourth image, finally, is clearly Star Wars, robot edition. However, no bell. ("A robot child riding a llama with a bell on its tail through a desert" – "Ah, you mean Star Wars!") I'm not even sure which of these images Alexander did pick as a winner. And I would describe neither image by the prompt, nor would I dare to imagine that a human had chosen these exact means to show what is described in the prompt. Having said that, thanks for the link to the DALL-E Mini paper!

t_mann 4y ago

I bet most humans would fail this test on images that everybody agrees are adequate portrayals. Answering a short query with an image is a highly non-injective mapping, you simply don't know what aspects of the scene were specifically asked for in the query, and which ones were filled in by the artist / AI. Eg, the queries "opening of a medieval theme park", "announcement of a witch trial", "king charles proclamation" might all be reasonably answered by similar images containing a small crowd and a speaker in a medieval-looking setting, even though they're not meant to refer to the same time periods or settings at all.

masswerk 4y ago
Mind that the test is meant to include the prompt/query. E.g., take one of Alexander's winners, the robot in the cathedral: a human would probably answer to the prompt by making the cathedral part of the subject, by investing some effort in pointing out that this is indeed a cathedral, instead of just conforming to the query by showing some ambigue bits in the background, which may or may not represent parts of a cathedral. The quest of the machine is still "create an image by allotting the elements provided in the query", not "compose an image showing this and that as the subject" – and there's a significant difference. Closing this gap would require the AI to form a concept of what is given as a subject in its entirety and then constructing a plausible scene around this, by a meaningful placement of the subject, which is clearly beyond the state of art. I admit that there is a certain appeal to those images, for their distinctive dreamlike quality, as there's often a specific tension in the rather blunt composition and an apparent subject, which seems to be beyond what is actually depicted, as if this was just a casually picked specimen from a series of illustrations for a broader story line. But I'd bet that we're going to have been seeing too much of this soon, in order to be still amazed.