5 ms·

I don't think it is actually painting at all but I need to read the paper carefully. I think it is using a free text query to select the best possible clipart

I don't think it is actually painting at all but I need to read the paper carefully. I think it is using a free text query to select the best possible clipart from a big library and blends it together. Still very interesting and useful. It would be extremely impressive if the "Kuala dunking a basketball" had a puddle on the court in which it was reflected correctly, that would be mind blowing.

Imnimo 5y ago
This is actual image generation - the 'decoder' takes as input a latent code (representing the encoding of the text query), and synthesizes an image. It's not compositing or querying a reference library. The only time that real images enter the process is during training - after that, it's just the network weights.
- recuter 5y ago
  It is compositing as final step. I understand that the Kuala it is compositing may have been a previously un-existent Kuala that it synthesized from a library of previously tagged Kuala images... that's cool, but what is the difference really from just plucking one of the pre-existing Kualas into the scene? The difference is just that it makes the compositing easier. If you don't have a pre-existing image that would match the shadows and angles you can hallucinate a new Kuala that does. Neat trick. But I bet if I threw the poor marsupial at a basket net it would look really differently than the original clipart of it climbing some tree in a slow and relaxed manner. See what I mean? Maybe Dall-E 2 can make it strike a new pose. The limb positions could be altered. But the facial expression? And if the basketball background has wind blowing leaves in one direction the Kuala fur won't match, it will look like the training set fur. The puddle won't reflect it. 'etc. This thing doesn't understand what a Kuala is like a 3-yr old. It understands the text "Kuala" is associated with that tagged collection of pixel blobs and can conjure up similar blobs unto new backgrounds - but it can't paint me a new type of Kuala that it hasn't seen before. It just looks that way.
  - andybak 5y ago
    > It is compositing as final step. I might be misinterpeting your use of "compositing" here (and my own technical knowledge is fairly shallow) but I don't think there's any compositing of elements generally in AI image generation. (unless Dall-E 2 changes this. I haven't read the paper yet)
    - recuter 5y ago
      https://cdn.openai.com/papers/dall-e-2.pdf https://cdn.openai.com/papers/dall-e-2.pdf > Given an image x, we can obtain its CLIP image embedding zi and then use our decoder to “invert” zi, producing new images that we call variations of our input. .. It is also possible to combine two images for variations. To do so, we perform spherical interpolation of their CLIP embeddings zi and zj to obtain intermediate zθ = slerp(zi, zj , θ), and produce variations of zθ by passing it through the decoder. From the limitations section: > We find that the reconstructions mix up objects and attributes.
      - andybak 5y ago
        I'm not sure that's "compositing" except in the most abstract sense? But maybe that's the sense in which you mean it. I'd argue that at no point is there a representation of a "teddy bear" and "a background" that map closely to their visual representation - that are combined. (I'm aware I'm being imprecise so give me some leeway here)
        astrange 5y ago
        This model's predecessor could do image editing with some help: https://arxiv.org/pdf/2112.10741.pdf https://arxiv.org/pdf/2112.10741.pdf so it could distinguish individual objects from backgrounds. Other ML models can definitely do that; it's called "panoptic segmentation".
        recuter 5y ago
        Thank you! Fascinating, I didn't know about panoptic segmentation - that makes things much more interesting. It really needs to expose the whole pipeline to become truly useful.

Imnimo 5y ago

This is actual image generation - the 'decoder' takes as input a latent code (representing the encoding of the text query), and synthesizes an image. It's not compositing or querying a reference library. The only time that real images enter the process is during training - after that, it's just the network weights.

recuter 5y ago
It is compositing as final step. I understand that the Kuala it is compositing may have been a previously un-existent Kuala that it synthesized from a library of previously tagged Kuala images... that's cool, but what is the difference really from just plucking one of the pre-existing Kualas into the scene? The difference is just that it makes the compositing easier. If you don't have a pre-existing image that would match the shadows and angles you can hallucinate a new Kuala that does. Neat trick. But I bet if I threw the poor marsupial at a basket net it would look really differently than the original clipart of it climbing some tree in a slow and relaxed manner. See what I mean? Maybe Dall-E 2 can make it strike a new pose. The limb positions could be altered. But the facial expression? And if the basketball background has wind blowing leaves in one direction the Kuala fur won't match, it will look like the training set fur. The puddle won't reflect it. 'etc. This thing doesn't understand what a Kuala is like a 3-yr old. It understands the text "Kuala" is associated with that tagged collection of pixel blobs and can conjure up similar blobs unto new backgrounds - but it can't paint me a new type of Kuala that it hasn't seen before. It just looks that way.
- andybak 5y ago
  > It is compositing as final step. I might be misinterpeting your use of "compositing" here (and my own technical knowledge is fairly shallow) but I don't think there's any compositing of elements generally in AI image generation. (unless Dall-E 2 changes this. I haven't read the paper yet)
  - recuter 5y ago
    https://cdn.openai.com/papers/dall-e-2.pdf https://cdn.openai.com/papers/dall-e-2.pdf > Given an image x, we can obtain its CLIP image embedding zi and then use our decoder to “invert” zi, producing new images that we call variations of our input. .. It is also possible to combine two images for variations. To do so, we perform spherical interpolation of their CLIP embeddings zi and zj to obtain intermediate zθ = slerp(zi, zj , θ), and produce variations of zθ by passing it through the decoder. From the limitations section: > We find that the reconstructions mix up objects and attributes.
    - andybak 5y ago
      I'm not sure that's "compositing" except in the most abstract sense? But maybe that's the sense in which you mean it. I'd argue that at no point is there a representation of a "teddy bear" and "a background" that map closely to their visual representation - that are combined. (I'm aware I'm being imprecise so give me some leeway here)
      - astrange 5y ago
        This model's predecessor could do image editing with some help: https://arxiv.org/pdf/2112.10741.pdf https://arxiv.org/pdf/2112.10741.pdf so it could distinguish individual objects from backgrounds. Other ML models can definitely do that; it's called "panoptic segmentation".
        recuter 5y ago
        Thank you! Fascinating, I didn't know about panoptic segmentation - that makes things much more interesting. It really needs to expose the whole pipeline to become truly useful.