10 ms·

What always bother me with this stuff is, well, you say one approach is more sensible than the other because the images happen to come out more pleasing. But t

What always bother me with this stuff is, well, you say one approach is more sensible than the other because the images happen to come out more pleasing. But there's no real rhyme or reason, it is a sort of alchemy. Is text encoding strictly worse or is it an artifact of the implementation? And if it is strictly worse, which is probably the case, why specifically? What is actually going on here? I can't argue that their results are not visually pleasing. But I'm not sure what one can really infer from all of this once the excitement washes over you. Blending photos together in a scene in photoshop is not a difficult task. It is nuanced and tedious but not hard, any pixel slinger will tell you. An app that accepts a smattering of photos and stitches them together nicely can be coded up any number of ways. This is a fantastic and time saving photoshop plugin. But what do we have really? "Kuala dunking basketball" needs to "understand" the separate items and select from the image library hoops and a Kuala where the angles and shadows roughly match. Very interesting, potentially useful. But if doesn't spit up exactly what you want can't edit it further. I think the next step has got to be that it conjures up a 3d scene in Unreal or blender so you can zoom in and around convincingly for further tweaks. Not a flat image.

aaron695 5y ago
ehsankia 5y ago
> But there's no real rhyme or reason, it is a sort of alchemy. Is there a rhyme or reason as to why picasso decided to paint like that? Yes these networks are hard to reason about, but so are real human brains.
woko 5y ago
> Very interesting, potentially useful. But if doesn't spit up exactly what you want can't edit it further. DALL-E 2 spits as many outputs as you want. Then you choose the one you prefer.
Imnimo 5y ago
Yeah, I mean you're right that ultimately the proof is in the pudding. But I do think we could have guessed that this sort of approach would be better (at least at a high level - I'm not claiming I could have predicted all the technical details!). The previous approaches were sort of the best that people could do without access to the training data and resources - you had a pretrained CLIP encoder that could tell you how well a text caption and an image matched, and you had a pretrained image generator (GAN, diffusion model, whatever), and it was just a matter of trying to force the generator to output something that CLIP thought looked like the caption. You'd basically do gradient ascent to make the image look more and more and more like the text prompt (all the while trying to balance the need to still look like a realistic image). Just from an algorithm aesthetics perspective, it was very much a duct tape and chicken wire approach. The analogy I would give is if you gave a three-year-old some paints, and they made an image and showed it to you, and you had to say, "this looks like a little like a sunset" or "this looks a lot like a sunset". They would keep going back and adjusting their painting, and you'd keep giving feedback, and eventually you'd get something that looks like a sunset. But it'd be better, if you could manage it, to just teach the three-year-old how to paint, rather than have this brute force process. Obviously the real challenge here is "well how do you teach a three-year-old how to paint?" - and I think you're right that that question still has a lot of alchemy to it.
- recuter 5y ago
  I don't think it is actually painting at all but I need to read the paper carefully. I think it is using a free text query to select the best possible clipart from a big library and blends it together. Still very interesting and useful. It would be extremely impressive if the "Kuala dunking a basketball" had a puddle on the court in which it was reflected correctly, that would be mind blowing.
  - Imnimo 5y ago
    This is actual image generation - the 'decoder' takes as input a latent code (representing the encoding of the text query), and synthesizes an image. It's not compositing or querying a reference library. The only time that real images enter the process is during training - after that, it's just the network weights.
mrandish 5y ago
> This is a fantastic and time saving photoshop plugin. But what do we have really? Stock photography sales are in the many billions of dollars per year and custom commissioned photography is larger still. That's a pretty seriously sized ready-made market. > But if doesn't spit up exactly what you want can't edit it further. I suspect there's a big startup opportunity in pioneering an easy-to-use interface allowing users to provide fast iterative feedback to the model - including positional and relational constraints ("put this thing over there"). Perhaps even more valuable would be easy yet granular ways to unconstrain the model. For example, "keep the basketball hoop like that but make the basketball an unexpected color and have the panda's right paw doing something pandas don't do that human hands often do."
- johnwheeler 5y ago
  Or as a precursor to Meta Horizon build a 3D world with speech https://www.fastcompany.com/90725035/metaverse-horizon-worlds-meta-quest-2-builder-bot https://www.fastcompany.com/90725035/metaverse-horizon-world...
- recuter 5y ago
  Certainly food for thought. Say I'm looking for photography of real events and places, like a royal weeding or a volcano erupting does this help me? Of specific places and architectural features? Of a protest? You're suggesting clipart on steroids: https://thispersondoesnotexist.com https://thispersondoesnotexist.com I think if I was istockphoto.com I'd be a little worried, but that is microstock photography. I'm not sure that is worth billions. In fact I know it isn't. Besides once this tech is wildly available if anything it devalues this sort of thing further closer to $0. It would probably augment existing processes rather than replace them completely. If you are doing a photoshoot for a banana stand with a human model with characteristics x,y,z you're still going to get a human from an agency or craigslist to pose. If suddenly the client informs you that they needed human a,b,c instead maybe one of these forthcoming tools will let you swap that out faster. You'd upload your photoshoot and an example or two of the type of human model you wished you had retroactively and it would fix it up faster than an intern. Cool.
  - 5y ago
rileyphone 5y ago
It would be interesting to see more attempts to “reverse engineer” ML models like in https://distill.pub/2020/circuits/curve-circuits https://distill.pub/2020/circuits/curve-circuits - maybe even with a ML model of its own?
moyix 5y ago
> But if doesn't spit up exactly what you want can't edit it further. Why? You can tweak the prompt, change parameters, or even use the actual "edit" capability that they demo in the post.
- recuter 5y ago
  Maybe I am misunderstanding but if you start tweaking the prompt you'll end with something completely different. The "edit" capability, as far as I can tell please correct me if I got confused, is picking your favorite out of the generated variations. I would like to "lock" the scene and add instructions like "throw in a reflection".
  - Jack000 5y ago
    This is exactly what they demo - they lock a scene and add a flamingo in three different locations. In another one they lock the scene and add a corgi.
    - recuter 5y ago
      Not quite, it looks like this: - Provide an existing image - Provide a text prompt ("flamingo") - Select from X variations the new image that looks best to you - It does the equivalent of a google image search on your "flamingo" prompt - It picks the most blend-able ones as a basis to a new synthetic flamingo - It superimposes the result on your image Very cool don't get me wrong. Now I want to tweak this new floating flamingo I picked further, or have that Corgi in the museum maybe sink into the little couch a bit as it has weight in the real world. Can't. You'd have to start over with the prompt or use this as the new base image maybe. The example with furniture placement in an empty room is also very interesting. You could describe the kind of couch you want and where you want it and it will throw you decent options. But say I want the purple one in the middle of the room that it gave me as an option, but rotated a little bit. It would generate a completely new purple couch. Maybe it will even look pretty similar but not exactly the same. See what I mean?
      - 5y ago
qq66 5y ago
I think deep learning is better thought of as "science" than "engineering." Right now we're in the stage of the Greeks and Arabs where we know "if we do this then that happens." It will be awhile before we have a coherent model of it, and I don't think we will ever solve all of its mysteries.
- uoaei 5y ago
  We are getting closer with variational methods and kernel methods to achieving a more holistic framework for understanding machine learning (incl. traditional deep learning) training and inference. There is a deep unity in the fundamentals of machine learning, formed into a cohesive whole by applying the analytical techniques of statistical mechanics and Bayesian probability theory.
tracyhenry 5y ago
mahastore 5y ago
I wish there was something available in open source that has similar functions i.e sensible amalgamation of pictures based on some text.
- astrange 5y ago
  https://github.com/borisdayma/dalle-mini https://github.com/borisdayma/dalle-mini https://rudalle.ru/en/ https://rudalle.ru/en/ Smaller reproductions of the original research.

aaron695 5y ago

ehsankia 5y ago

> But there's no real rhyme or reason, it is a sort of alchemy. Is there a rhyme or reason as to why picasso decided to paint like that? Yes these networks are hard to reason about, but so are real human brains.

woko 5y ago

> Very interesting, potentially useful. But if doesn't spit up exactly what you want can't edit it further. DALL-E 2 spits as many outputs as you want. Then you choose the one you prefer.

Imnimo 5y ago

Yeah, I mean you're right that ultimately the proof is in the pudding. But I do think we could have guessed that this sort of approach would be better (at least at a high level - I'm not claiming I could have predicted all the technical details!). The previous approaches were sort of the best that people could do without access to the training data and resources - you had a pretrained CLIP encoder that could tell you how well a text caption and an image matched, and you had a pretrained image generator (GAN, diffusion model, whatever), and it was just a matter of trying to force the generator to output something that CLIP thought looked like the caption. You'd basically do gradient ascent to make the image look more and more and more like the text prompt (all the while trying to balance the need to still look like a realistic image). Just from an algorithm aesthetics perspective, it was very much a duct tape and chicken wire approach. The analogy I would give is if you gave a three-year-old some paints, and they made an image and showed it to you, and you had to say, "this looks like a little like a sunset" or "this looks a lot like a sunset". They would keep going back and adjusting their painting, and you'd keep giving feedback, and eventually you'd get something that looks like a sunset. But it'd be better, if you could manage it, to just teach the three-year-old how to paint, rather than have this brute force process. Obviously the real challenge here is "well how do you teach a three-year-old how to paint?" - and I think you're right that that question still has a lot of alchemy to it.

recuter 5y ago
I don't think it is actually painting at all but I need to read the paper carefully. I think it is using a free text query to select the best possible clipart from a big library and blends it together. Still very interesting and useful. It would be extremely impressive if the "Kuala dunking a basketball" had a puddle on the court in which it was reflected correctly, that would be mind blowing.
- Imnimo 5y ago
  This is actual image generation - the 'decoder' takes as input a latent code (representing the encoding of the text query), and synthesizes an image. It's not compositing or querying a reference library. The only time that real images enter the process is during training - after that, it's just the network weights.

mrandish 5y ago

> This is a fantastic and time saving photoshop plugin. But what do we have really? Stock photography sales are in the many billions of dollars per year and custom commissioned photography is larger still. That's a pretty seriously sized ready-made market. > But if doesn't spit up exactly what you want can't edit it further. I suspect there's a big startup opportunity in pioneering an easy-to-use interface allowing users to provide fast iterative feedback to the model - including positional and relational constraints ("put this thing over there"). Perhaps even more valuable would be easy yet granular ways to unconstrain the model. For example, "keep the basketball hoop like that but make the basketball an unexpected color and have the panda's right paw doing something pandas don't do that human hands often do."

johnwheeler 5y ago
Or as a precursor to Meta Horizon build a 3D world with speech https://www.fastcompany.com/90725035/metaverse-horizon-worlds-meta-quest-2-builder-bot https://www.fastcompany.com/90725035/metaverse-horizon-world...
recuter 5y ago
Certainly food for thought. Say I'm looking for photography of real events and places, like a royal weeding or a volcano erupting does this help me? Of specific places and architectural features? Of a protest? You're suggesting clipart on steroids: https://thispersondoesnotexist.com https://thispersondoesnotexist.com I think if I was istockphoto.com I'd be a little worried, but that is microstock photography. I'm not sure that is worth billions. In fact I know it isn't. Besides once this tech is wildly available if anything it devalues this sort of thing further closer to $0. It would probably augment existing processes rather than replace them completely. If you are doing a photoshoot for a banana stand with a human model with characteristics x,y,z you're still going to get a human from an agency or craigslist to pose. If suddenly the client informs you that they needed human a,b,c instead maybe one of these forthcoming tools will let you swap that out faster. You'd upload your photoshoot and an example or two of the type of human model you wished you had retroactively and it would fix it up faster than an intern. Cool.
- 5y ago

rileyphone 5y ago

It would be interesting to see more attempts to “reverse engineer” ML models like in https://distill.pub/2020/circuits/curve-circuits https://distill.pub/2020/circuits/curve-circuits - maybe even with a ML model of its own?

moyix 5y ago

> But if doesn't spit up exactly what you want can't edit it further. Why? You can tweak the prompt, change parameters, or even use the actual "edit" capability that they demo in the post.

recuter 5y ago
Maybe I am misunderstanding but if you start tweaking the prompt you'll end with something completely different. The "edit" capability, as far as I can tell please correct me if I got confused, is picking your favorite out of the generated variations. I would like to "lock" the scene and add instructions like "throw in a reflection".
- Jack000 5y ago
  This is exactly what they demo - they lock a scene and add a flamingo in three different locations. In another one they lock the scene and add a corgi.
  - recuter 5y ago
    Not quite, it looks like this: - Provide an existing image - Provide a text prompt ("flamingo") - Select from X variations the new image that looks best to you - It does the equivalent of a google image search on your "flamingo" prompt - It picks the most blend-able ones as a basis to a new synthetic flamingo - It superimposes the result on your image Very cool don't get me wrong. Now I want to tweak this new floating flamingo I picked further, or have that Corgi in the museum maybe sink into the little couch a bit as it has weight in the real world. Can't. You'd have to start over with the prompt or use this as the new base image maybe. The example with furniture placement in an empty room is also very interesting. You could describe the kind of couch you want and where you want it and it will throw you decent options. But say I want the purple one in the middle of the room that it gave me as an option, but rotated a little bit. It would generate a completely new purple couch. Maybe it will even look pretty similar but not exactly the same. See what I mean?
    - 5y ago

qq66 5y ago

I think deep learning is better thought of as "science" than "engineering." Right now we're in the stage of the Greeks and Arabs where we know "if we do this then that happens." It will be awhile before we have a coherent model of it, and I don't think we will ever solve all of its mysteries.

uoaei 5y ago
We are getting closer with variational methods and kernel methods to achieving a more holistic framework for understanding machine learning (incl. traditional deep learning) training and inference. There is a deep unity in the fundamentals of machine learning, formed into a cohesive whole by applying the analytical techniques of statistical mechanics and Bayesian probability theory.

tracyhenry 5y ago

mahastore 5y ago

I wish there was something available in open source that has similar functions i.e sensible amalgamation of pictures based on some text.

astrange 5y ago
https://github.com/borisdayma/dalle-mini https://github.com/borisdayma/dalle-mini https://rudalle.ru/en/ https://rudalle.ru/en/ Smaller reproductions of the original research.