16 ms·

In the video towards the bottom of the page, there are two birds (blue jays), but in the background there are two identical buildings (which look a lot like the

In the video towards the bottom of the page, there are two birds (blue jays), but in the background there are two identical buildings (which look a lot like the CN Tower). CN Tower is the main landmark of Toronto, whose baseball team happens to be the Blue Jays. It's located near the main sportsball stadium downtown. I vaguely understand how text-to-image works, and so it makes sense that the vector space for "blue jays" would be near "toronto" or "cn tower". The improvements in scale and speed (image -> now video) are impressive, but given how incredibly able the image generation models are, they simultaneously feel crippled and limited by their lack of editing / iteration ability. Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close.

appplication 3y ago
I don’t spend a lot of time keeping up with the space, but I could have sworn I’ve seen a demo that allowed you to iterate in the way you’re suggesting. Maybe someone else can link it.
- accrual 3y ago
  It's not exactly like GP described (e.g. move bike to the left) but there is a more advanced SD technique called inpainting [0] that allows you to manually recompose parts of the image, e.g. to fix bad eyes and hands. [0] https://stable-diffusion-art.com/inpainting_basics/ https://stable-diffusion-art.com/inpainting_basics/
- ssalka 3y ago
  My guess is you're thinking of InstructPix2Pix[1], with prompts like "make the sky green" or "replace the fruits with cake" [1] https://github.com/timothybrooks/instruct-pix2pix https://github.com/timothybrooks/instruct-pix2pix
  - appplication 3y ago
    This is exactly it!
- tjoff 3y ago
  Emu-Edit is the closest I've seen. https://emu-edit.metademolab.com/ https://emu-edit.metademolab.com/ https://ai.meta.com/blog/emu-text-to-video-generation-image-editing-research/ https://ai.meta.com/blog/emu-text-to-video-generation-image-...
kshacker 3y ago
Assuming we can post links, you mean this video: https://youtu.be/G7mihAy691g?si=o2KCmR2Uh_97UQ0N https://youtu.be/G7mihAy691g?si=o2KCmR2Uh_97UQ0N Also, maybe you can't edit post facto, but when you give prompts, would you not be able to say : two blue jays but no CN tower
- FrozenTuna 3y ago
  Yes, its called a negative prompt. Idk if txt2video has it, but both llms and stable-diffusion have it so I'd assume its good to go.
  - nottheengineer 3y ago
    Haven't implemented negative prompts yet, but from what I can tell it's as simple as substracting from the prompt in embedding space.
FrozenTuna 3y ago
Not exactly what you're asking for, but AnimateDiff has introduced creating gifs to SD. Still takes quite a bit of tweaking IME.
xianshou 3y ago
Emu edit should be exactly what you're looking for: https://ai.meta.com/blog/emu-text-to-video-generation-image-editing-research/ https://ai.meta.com/blog/emu-text-to-video-generation-image-...
- smcleod 3y ago
  It doesn’t look like the code for that is available anywhere though?
TacticalCoder 3y ago
> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. I feel like we're close too, but for another reason. For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer can immediately spot that. However I'm willing to bet that we'll soon have something much better: you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. And the scene shall be sent into Blender and you'll click on a button and have an actual rendering made by Blender, with correct lighting. Wanna move that bicycle? Move it in the 3D scene exactly where you want. That is coming. And for audio it's the same: why generate an audio file when soon models shall be able to generate the various tracks, with all the instruments and whatnots, allowing to create the audio file? That is coming too.
- p1esk 3y ago
  Are you working on all that?
  - cptaj 3y ago
    Probably not. But there does seem to be a clear path to it. The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format. You need a whole metric fuckton of these. After that, you train your model to produce those 3d scene files instead of image bitmaps. You can do this for a lot of other tasks. These general purpose models can learn anything that you can usefully represent in data. I can imagine AGI being, at least in part, a large set of these purpose trained models. Heck, maybe our brains work this way. When we learn to throw a ball, we train a model in a subset of our brain to do just this and then this model is called on by our general consciousness when needed. Sorry, I'm just rambling here but its very exciting stuff.
    - 3y ago
treesciencebot 3y ago
Have you seen fal.ai/dynamic where you can perform image to image synthesis (basically editing an existing image with the help of diffusion process) using LCMs to provide a real time UI?
filterfiber 3y ago
> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. Emu can do that. The bluejay/toronto thing may be addressable later (I suspect via more detailed annotations a la dalle3) - these current video models are highly focused on figuring out temporal coherence
JoshTriplett 3y ago
I also wonder if the model takes capitalization into account. Capitalized "Blue Jays" seems more likely to reference the sports team; the birds would be lowercase.
psunavy03 3y ago
> sportsball This is not the flex you think it is. You don't have to like sports, but snarking on people who do doesn't make you intellectual, it just makes you come across as a douchebag, no different than a sports fan making fun of "D&D nerds" or something.
- chaps 3y ago
  Ah, Mr. Kettle, I see you've met my friend, Mr. Pot!
- Zetaphor 3y ago
  This has become a colloquial term for describing all sports, not the insult you're perceiving it to be. Rather than projecting your own hangups and calling people names, try instead assuming that they're not trying to offend you personally and are just using common vernacular.
  - achileas 3y ago
    If only there was an existing way to refer to sports generally! And OP was referring to a specific sport (baseball), not sports generally.
    - btbuildem 3y ago
      The Rogers Centre hosts baseball, football, and basketball games - so in this case "sportsball" was just a shorthand for all these ball sports.
- deleted 3y ago
  [deleted]
- jojobas 3y ago
  Would you get incensed by "petrolhead", "greenfingers" or "trekkie"? Is that what you choose to be emotional about?
- callalex 3y ago
  You’re really not helping the “sports fans are combative thugs” stereotype by going off on an insult tirade over an innocent word.
amoshebb 3y ago
I wonder what other odd connections are made due to city-name almost certainly being the most common word next to sportsball-name. Do the parameters think that Jazz musicians are mormon? Padres often surf? Wizards like the Lincoln Memorial?
ProfessorZoom 3y ago
that sounds like v0 by vercel, you can iterate just like you asked, to combine that type of iteration with video would be really awesome
jackmott 3y ago
[dead]
01100011 3y ago
I recently tried to generate clip art for a presentation using GPT-4/DALL-E 3. I found it could handle some updates but the output generally varied wildly as I tried to refine the image. For instance, I'd have a cartoon character checking its watch and also wearing a pocket watch. Trying to remove the pocket watch resulted in an entirely new cartoon with little stylistic continuity to the first. Also, I originally tried to get the 3 characters in the image to be generated simultaneously, but eventually gave up as DALL-E had a hard time understanding how I wanted them positioned relative to each other. I just generated 3 separate characters and positioned them in the same image using Gimp.
- btbuildem 3y ago
  Yes that's exactly what I'm referring to! It feels as if there is no context continuity between the attempts.
stevage 3y ago
I wondered similarly whether the astronaut's weird gait was because it was kind of "moonwalking" on the moon.
achileas 3y ago
> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. Nearly all of the available models have this, even the highly commercialized ones like in Adobe Firefly and Canva, it’s called inpainting in most tools.
- btbuildem 3y ago
  I think that's more "inpainting" where the existing software solution uses AI to accelerate certain image editing tasks. I was looking for whole-image manipulation at the "conceptual" level.
  - achileas 3y ago
    They have this. Inpainting is just a subset of the image-to-image workflow and you don't have to provide a region if you want to do whole-image manipulation.
dsmmcken 3y ago
Adobe is doing some great work here in my opinion in terms of building AI tools that make sense for artist workflows. This "sneak peak" demo from the recent Adobe Max conference is pretty much exactly what you described, actually better because you can just click on an object in the image and drag it. See video: https://www.adobe.com/max/2023/sessions/project-stardust-gs6-11.html https://www.adobe.com/max/2023/sessions/project-stardust-gs6...
- btbuildem 3y ago
  Right, that's embedded directly into the existing workflow. Looks like a very powerful feature indeed.
- deleted 3y ago
  [deleted]
- thatoneguy 3y ago
  Makes me wonder if they train their data on everything anyone has ever uploaded to Creative Cloud.
Veraslane 3y ago
[dead]
COAGULOPATH 3y ago
> they simultaneously feel crippled and limited by their lack of editing / iteration ability. Yeah. They're not "videos" so much as images that move around a bit. This doesn't really look any better than those Midjourney + RunwayML videos we had half a year ago. >Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. Google has a model called Phenaki that supposedly allows for that kind of stuff. But the public can't use it so it's hard to say how good it actually is.
zeckalpha 3y ago
I see that as a reference to the AI generated Toronto Blue Jays advertisement gone wrong that went viral earlier this year. https://www.blogto.com/sports_play/2023/06/ai-generated-toronto-blue-jays-commercial/ https://www.blogto.com/sports_play/2023/06/ai-generated-toro...
omneity 3y ago
Nice eye! As for your last question yes that exists. There are two models from Meta that do exactly this, instruction based iteration on photos, Emu Edit[0], and videos, Emu Video[1]. There's also LLaVa-interactive[2] for photos where you can even chat with the model about the current image. [0]: https://emu-edit.metademolab.com/ https://emu-edit.metademolab.com/ [1]: https://emu-video.metademolab.com/ https://emu-video.metademolab.com/ [2]: https://llava-vl.github.io/llava-interactive/ https://llava-vl.github.io/llava-interactive/

appplication 3y ago

I don’t spend a lot of time keeping up with the space, but I could have sworn I’ve seen a demo that allowed you to iterate in the way you’re suggesting. Maybe someone else can link it.

accrual 3y ago
It's not exactly like GP described (e.g. move bike to the left) but there is a more advanced SD technique called inpainting [0] that allows you to manually recompose parts of the image, e.g. to fix bad eyes and hands. [0] https://stable-diffusion-art.com/inpainting_basics/ https://stable-diffusion-art.com/inpainting_basics/
ssalka 3y ago
My guess is you're thinking of InstructPix2Pix[1], with prompts like "make the sky green" or "replace the fruits with cake" [1] https://github.com/timothybrooks/instruct-pix2pix https://github.com/timothybrooks/instruct-pix2pix
- appplication 3y ago
  This is exactly it!
tjoff 3y ago
Emu-Edit is the closest I've seen. https://emu-edit.metademolab.com/ https://emu-edit.metademolab.com/ https://ai.meta.com/blog/emu-text-to-video-generation-image-editing-research/ https://ai.meta.com/blog/emu-text-to-video-generation-image-...

kshacker 3y ago

Assuming we can post links, you mean this video: https://youtu.be/G7mihAy691g?si=o2KCmR2Uh_97UQ0N https://youtu.be/G7mihAy691g?si=o2KCmR2Uh_97UQ0N Also, maybe you can't edit post facto, but when you give prompts, would you not be able to say : two blue jays but no CN tower

FrozenTuna 3y ago
Yes, its called a negative prompt. Idk if txt2video has it, but both llms and stable-diffusion have it so I'd assume its good to go.
- nottheengineer 3y ago
  Haven't implemented negative prompts yet, but from what I can tell it's as simple as substracting from the prompt in embedding space.

FrozenTuna 3y ago

Not exactly what you're asking for, but AnimateDiff has introduced creating gifs to SD. Still takes quite a bit of tweaking IME.

xianshou 3y ago

Emu edit should be exactly what you're looking for: https://ai.meta.com/blog/emu-text-to-video-generation-image-editing-research/ https://ai.meta.com/blog/emu-text-to-video-generation-image-...

smcleod 3y ago
It doesn’t look like the code for that is available anywhere though?

TacticalCoder 3y ago

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. I feel like we're close too, but for another reason. For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer can immediately spot that. However I'm willing to bet that we'll soon have something much better: you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. And the scene shall be sent into Blender and you'll click on a button and have an actual rendering made by Blender, with correct lighting. Wanna move that bicycle? Move it in the 3D scene exactly where you want. That is coming. And for audio it's the same: why generate an audio file when soon models shall be able to generate the various tracks, with all the instruments and whatnots, allowing to create the audio file? That is coming too.

p1esk 3y ago
Are you working on all that?
- cptaj 3y ago
  Probably not. But there does seem to be a clear path to it. The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format. You need a whole metric fuckton of these. After that, you train your model to produce those 3d scene files instead of image bitmaps. You can do this for a lot of other tasks. These general purpose models can learn anything that you can usefully represent in data. I can imagine AGI being, at least in part, a large set of these purpose trained models. Heck, maybe our brains work this way. When we learn to throw a ball, we train a model in a subset of our brain to do just this and then this model is called on by our general consciousness when needed. Sorry, I'm just rambling here but its very exciting stuff.
  - 3y ago

treesciencebot 3y ago

Have you seen fal.ai/dynamic where you can perform image to image synthesis (basically editing an existing image with the help of diffusion process) using LCMs to provide a real time UI?

filterfiber 3y ago

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. Emu can do that. The bluejay/toronto thing may be addressable later (I suspect via more detailed annotations a la dalle3) - these current video models are highly focused on figuring out temporal coherence

JoshTriplett 3y ago

I also wonder if the model takes capitalization into account. Capitalized "Blue Jays" seems more likely to reference the sports team; the birds would be lowercase.

psunavy03 3y ago

> sportsball This is not the flex you think it is. You don't have to like sports, but snarking on people who do doesn't make you intellectual, it just makes you come across as a douchebag, no different than a sports fan making fun of "D&D nerds" or something.

chaps 3y ago
Ah, Mr. Kettle, I see you've met my friend, Mr. Pot!
Zetaphor 3y ago
This has become a colloquial term for describing all sports, not the insult you're perceiving it to be. Rather than projecting your own hangups and calling people names, try instead assuming that they're not trying to offend you personally and are just using common vernacular.
- achileas 3y ago
  If only there was an existing way to refer to sports generally! And OP was referring to a specific sport (baseball), not sports generally.
  - btbuildem 3y ago
    The Rogers Centre hosts baseball, football, and basketball games - so in this case "sportsball" was just a shorthand for all these ball sports.
deleted 3y ago
[deleted]
jojobas 3y ago
Would you get incensed by "petrolhead", "greenfingers" or "trekkie"? Is that what you choose to be emotional about?
callalex 3y ago
You’re really not helping the “sports fans are combative thugs” stereotype by going off on an insult tirade over an innocent word.

amoshebb 3y ago

I wonder what other odd connections are made due to city-name almost certainly being the most common word next to sportsball-name. Do the parameters think that Jazz musicians are mormon? Padres often surf? Wizards like the Lincoln Memorial?

ProfessorZoom 3y ago

that sounds like v0 by vercel, you can iterate just like you asked, to combine that type of iteration with video would be really awesome

jackmott 3y ago

[dead]

01100011 3y ago

I recently tried to generate clip art for a presentation using GPT-4/DALL-E 3. I found it could handle some updates but the output generally varied wildly as I tried to refine the image. For instance, I'd have a cartoon character checking its watch and also wearing a pocket watch. Trying to remove the pocket watch resulted in an entirely new cartoon with little stylistic continuity to the first. Also, I originally tried to get the 3 characters in the image to be generated simultaneously, but eventually gave up as DALL-E had a hard time understanding how I wanted them positioned relative to each other. I just generated 3 separate characters and positioned them in the same image using Gimp.

btbuildem 3y ago
Yes that's exactly what I'm referring to! It feels as if there is no context continuity between the attempts.

stevage 3y ago

I wondered similarly whether the astronaut's weird gait was because it was kind of "moonwalking" on the moon.

achileas 3y ago

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. Nearly all of the available models have this, even the highly commercialized ones like in Adobe Firefly and Canva, it’s called inpainting in most tools.

btbuildem 3y ago
I think that's more "inpainting" where the existing software solution uses AI to accelerate certain image editing tasks. I was looking for whole-image manipulation at the "conceptual" level.
- achileas 3y ago
  They have this. Inpainting is just a subset of the image-to-image workflow and you don't have to provide a region if you want to do whole-image manipulation.

dsmmcken 3y ago

Adobe is doing some great work here in my opinion in terms of building AI tools that make sense for artist workflows. This "sneak peak" demo from the recent Adobe Max conference is pretty much exactly what you described, actually better because you can just click on an object in the image and drag it. See video: https://www.adobe.com/max/2023/sessions/project-stardust-gs6-11.html https://www.adobe.com/max/2023/sessions/project-stardust-gs6...

btbuildem 3y ago
Right, that's embedded directly into the existing workflow. Looks like a very powerful feature indeed.
deleted 3y ago
[deleted]
thatoneguy 3y ago
Makes me wonder if they train their data on everything anyone has ever uploaded to Creative Cloud.

Veraslane 3y ago

[dead]

COAGULOPATH 3y ago

> they simultaneously feel crippled and limited by their lack of editing / iteration ability. Yeah. They're not "videos" so much as images that move around a bit. This doesn't really look any better than those Midjourney + RunwayML videos we had half a year ago. >Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. Google has a model called Phenaki that supposedly allows for that kind of stuff. But the public can't use it so it's hard to say how good it actually is.

zeckalpha 3y ago

I see that as a reference to the AI generated Toronto Blue Jays advertisement gone wrong that went viral earlier this year. https://www.blogto.com/sports_play/2023/06/ai-generated-toronto-blue-jays-commercial/ https://www.blogto.com/sports_play/2023/06/ai-generated-toro...

omneity 3y ago

Nice eye! As for your last question yes that exists. There are two models from Meta that do exactly this, instruction based iteration on photos, Emu Edit[0], and videos, Emu Video[1]. There's also LLaVa-interactive[2] for photos where you can even chat with the model about the current image. [0]: https://emu-edit.metademolab.com/ https://emu-edit.metademolab.com/ [1]: https://emu-video.metademolab.com/ https://emu-video.metademolab.com/ [2]: https://llava-vl.github.io/llava-interactive/ https://llava-vl.github.io/llava-interactive/