5 ms·

Here is a preliminary test for video editing using ControlNet I made: https://www.youtube.com/watch?v=u52MOA4YaGk https://www.youtube.com/watch?v=u52MOA4YaGk A

Here is a preliminary test for video editing using ControlNet I made: https://www.youtube.com/watch?v=u52MOA4YaGk https://www.youtube.com/watch?v=u52MOA4YaGk As you can see, there is still quite a bit of flicker, I'm working to reduce that. But the consistency is much better compared to, say, img2img. I'm hoping to ship a prototype this week.

meghan_rain 4y ago
S-so the girl on the right in the second half of the video is ot real...?
- refulgentis 4y ago
  really curious about this too OP, is the face generated? How do you keep temporal consistency with that?
  - synapticpaint 4y ago
    So, the video was generated by applying ControlNet to the input video frame by frame. Every inference setting is the same for every frame -- seed, prompt, CFG, steps, and sampler. The only thing that changes frame to frame is that the pose changes slightly. So actually, if SD was well behaved, you would expect the difference between adjacent frames to be small, because the change in the input is small. But SD is somewhat schizophrenic so you get this amount of flicker from even small changes in input. I also had to specify what the outfit should be (I got a lot more discrepancies when I didn't do this from the outfit changing frame to frame). You can see that the outfit changes color in the second version, I bet you can get that to be even more consistent if you specify the color in the prompt too. If you create a dreambooth model of a character, you can probably also get consistency of the face that way. In this case I didn't need to do this because I didn't care who I got, I just asked for an "average woman".
    - meghan_rain 4y ago
      Is there like a "temperature" setting you can change? And set it to 0 to produce less flickering?
      - Lerc 4y ago
        The flickering comes from the fundamental nature of the de-noising mechanism involved in the diffusion model. The ability to create multiple novel images for the same input comes from adding noise with a random seed. Currently this is more or less done every frame which is why you get the flickering. Keeping the same seed wouldn't be helpful if you want the image to move. What could be of use here is a noise transformation layer that can use the same noise for every frame but transformed to match desired motion. For video conversion you could possibly extract motion vectors from successive frames to warp the noise. I assume someone is working on this somewhere.
        synapticpaint 4y ago
        "The flickering comes from the fundamental nature of the de-noising mechanism involved in the diffusion model." -- agreed "Keeping the same seed wouldn't be helpful if you want the image to move." -- No, I'm using the same seed (and prompt). The image moves because ControlNet opens up another channel of input, in this case the pose data.
        Lerc 4y ago
        Yes but that still produces temporal aliasing because the unmoving noise is battling the moving controlnet input. I can't find it right now but there was a good example showing a gallery of one word prompts with the same seed. While the images were of different subjects you could clearly see the impact of the noise controlling layout. What was was a capital letter A in one image was a persons legs in another, but that same overall structure was visible in the same place in 90% of the images
      - shubb 4y ago
        I wonder if putting an adversary network on top would reduce the flickering. A mechanism that only accepts a frame of it is detected to be three next frame in a video of the same person, otherwise regenerate
- synapticpaint 4y ago
  Correct, it's generated by SD.
guiambros 4y ago
Haven't read the paper yet, but curious how different ControlNet is from Text2LIVE ([1], [2]). Seems it's solving the same problem with temporal consistency, no? [1] https://www.youtube.com/watch?v=8U9o5aZ2y5w https://www.youtube.com/watch?v=8U9o5aZ2y5w [2] https://text2live.github.io/ https://text2live.github.io/
- synapticpaint 4y ago
  No, ControlNet wasn't made to solve temporal consistency, it was made to add more control (hence the name) to image models. I am using it in a way that the authors may not have thought of, because the paper doesn't mention video editing.
thom 4y ago
Impressive that it doesn’t just do pose transfer but applies the correct reverse kinematics too (hand on the wall/rail etc).
- synapticpaint 4y ago
  Yes, it's asking SD for an image with some set of characteristics, and SD has some notion of what is plausible from what it saw during training.
jpeter 4y ago
Can you mask the background out?

meghan_rain 4y ago

S-so the girl on the right in the second half of the video is ot real...?

refulgentis 4y ago
really curious about this too OP, is the face generated? How do you keep temporal consistency with that?
- synapticpaint 4y ago
  So, the video was generated by applying ControlNet to the input video frame by frame. Every inference setting is the same for every frame -- seed, prompt, CFG, steps, and sampler. The only thing that changes frame to frame is that the pose changes slightly. So actually, if SD was well behaved, you would expect the difference between adjacent frames to be small, because the change in the input is small. But SD is somewhat schizophrenic so you get this amount of flicker from even small changes in input. I also had to specify what the outfit should be (I got a lot more discrepancies when I didn't do this from the outfit changing frame to frame). You can see that the outfit changes color in the second version, I bet you can get that to be even more consistent if you specify the color in the prompt too. If you create a dreambooth model of a character, you can probably also get consistency of the face that way. In this case I didn't need to do this because I didn't care who I got, I just asked for an "average woman".
  - meghan_rain 4y ago
    Is there like a "temperature" setting you can change? And set it to 0 to produce less flickering?
    - Lerc 4y ago
      The flickering comes from the fundamental nature of the de-noising mechanism involved in the diffusion model. The ability to create multiple novel images for the same input comes from adding noise with a random seed. Currently this is more or less done every frame which is why you get the flickering. Keeping the same seed wouldn't be helpful if you want the image to move. What could be of use here is a noise transformation layer that can use the same noise for every frame but transformed to match desired motion. For video conversion you could possibly extract motion vectors from successive frames to warp the noise. I assume someone is working on this somewhere.
      - synapticpaint 4y ago
        "The flickering comes from the fundamental nature of the de-noising mechanism involved in the diffusion model." -- agreed "Keeping the same seed wouldn't be helpful if you want the image to move." -- No, I'm using the same seed (and prompt). The image moves because ControlNet opens up another channel of input, in this case the pose data.
        Lerc 4y ago
        Yes but that still produces temporal aliasing because the unmoving noise is battling the moving controlnet input. I can't find it right now but there was a good example showing a gallery of one word prompts with the same seed. While the images were of different subjects you could clearly see the impact of the noise controlling layout. What was was a capital letter A in one image was a persons legs in another, but that same overall structure was visible in the same place in 90% of the images
    - shubb 4y ago
      I wonder if putting an adversary network on top would reduce the flickering. A mechanism that only accepts a frame of it is detected to be three next frame in a video of the same person, otherwise regenerate
synapticpaint 4y ago
Correct, it's generated by SD.

guiambros 4y ago

Haven't read the paper yet, but curious how different ControlNet is from Text2LIVE ([1], [2]). Seems it's solving the same problem with temporal consistency, no? [1] https://www.youtube.com/watch?v=8U9o5aZ2y5w https://www.youtube.com/watch?v=8U9o5aZ2y5w [2] https://text2live.github.io/ https://text2live.github.io/

synapticpaint 4y ago
No, ControlNet wasn't made to solve temporal consistency, it was made to add more control (hence the name) to image models. I am using it in a way that the authors may not have thought of, because the paper doesn't mention video editing.

thom 4y ago

Impressive that it doesn’t just do pose transfer but applies the correct reverse kinematics too (hand on the wall/rail etc).

synapticpaint 4y ago
Yes, it's asking SD for an image with some set of characteristics, and SD has some notion of what is plausible from what it saw during training.

jpeter 4y ago

Can you mask the background out?