5 ms·

Photorealistic Video Generation with Diffusion Models

156 points by smusamashah 3y ago

youssefabdelm 3y ago
better but... still sucks... cant wait til it works
londons_explore 3y ago
Is this another case where the architecture is trivial and obvious, but the data and compute requirements are obscene?
- j45 3y ago
  While this interests me, a lot of this stuff started obscene and became not obscene. A line of sight, however disappointing is still better than no line of sight. Maybe it will turn the open source efforts onto these things that are announced.
- VikingCoder 3y ago
  1. All the data in the entire world 2. ??? 3. Profit
- 6510 3y ago
  They did wrote: "Images in this section is not generated by our model."
thequadehunter 3y ago
No code..as usual :/
- sschueller 3y ago
  Looks like a pretty standard comfyui svd workflow. https://github.com/thecooltechguy/ComfyUI-Stable-Video-Diffusion https://github.com/thecooltechguy/ComfyUI-Stable-Video-Diffu...
  - GaggiX 3y ago
    This paper is not about SVD (Stable Video Diffusion).
  - peab 3y ago
    that's a different model, by stability AI. Right now everyone is coming out with video models, and they're all very similar. Stability, Meta, Google, Nvidia, tencent, Alibaba all have released either demos or code for Stable Diffusion based video generation. They're all fairly similar but slightly different architecture.
ur-whale 3y ago
Amazing. The fact that we can synthesize stuff like this from text makes me believe the real world obeys some laws of physics we still haven't figured out. Specifically, the fact that when we interpolate in embedding space, the interpolated point still represents something almost always physically plausible (not always though, witness the surfing cat that doesn't get wet underwater).
- thfuran 3y ago
  But the embedding space is deliberately and laboriously constructed such that it represents realistic things at as many points as possible. That's what the training is, right?
- renegade-otter 3y ago
  Eh? It's automated statistical learning. Yes, it's very cool, but it is far from AGI, let alone magic. I am playing around with SD right now - pulling sleepless nights because it's so fascinating, but I do get to see the patterns. Generating unreal, magical stuff is actually a lot of fun, but if you want it to generate something that makes sense - good luck. It is really hard. I can get Elizabeth Olsen into a Halo suit in a war zone on an alien world, but getting her to sit in a car like a normal human being has been impossible. It's always some twisted horror show. The predictive model does not know what to do with it and how it all fits. That's after you clean up all the anatomical nightmare fuel like the fingers situation. This is why most of the SD artwork you see are magical elf girls in the foreground or some unearthly monsters. A well-defined subject in the foreground (a human-ish one), it will do VERY well. If you really want to get your imagination going with every little detail being near-perfect, you will be busy. Oh, and the text - it obviously cannot do text. It's in the right place, but it looks funny. This is amazing tech, but it's not about to go sentient and kill me at night.
  - echelon 3y ago
    No no, I feel the OP is onto something. > obeys some laws of physics we still haven't figured out. That statement might have been over the top, but I think they're right with respect to the information. Spatial logic, optics, signal processing, reasoning. It's beginning to feel like there's an interesting representation that cross-cuts these spaces in a powerful, generalizable way.
plastic3169 3y ago
Seems gruel to even say this as the tech is pure magic, but sad that these videos often get that ultra processed stockfootage look. I would be more impressed to just have bland everyday videos with muted colors, shaky cam and uninteresting people doing everyday stuff.
- nuz 3y ago
  Loras can often achieve that look. No weights though but that would have likely worked to skew it in the 'amateur video' direction.
- eurekin 3y ago
  I imagine some kind of service could pop up, exactly for that. For pictures, we already have: https://magnific.ai/ https://magnific.ai/ Which seems to have been endorsed by one of the best retouchers I know of (https://www.youtube.com/@pratiknaikedu https://www.youtube.com/@pratiknaikedu)
  - jacobolus 3y ago
    What they are asking for is to have these machine-generated "photorealistic" outputs look more like real photos/videos, not to turn every normal looking photo into a lurid dream-scene monstrosity with completely made up details.
  - TaylorAlexander 3y ago
    Just a note you can run upscaling on your home desktop with Real-ESRGAN: https://github.com/xinntao/Real-ESRGAN https://github.com/xinntao/Real-ESRGAN
sschueller 3y ago
If you tweak the outputs you can get much better results but takes time and resources. You are also quite limited to the duration at this point. Here is an example I made yesterday, probably around 5 minutes of computing on my 3080ti: https://imgur.com/a/4dPv1hC https://imgur.com/a/4dPv1hC
- eurekin 3y ago
  Could you create a writeup (or record screen, or screenshot relevant config) how to reproduce that effect?
  - sschueller 3y ago
    I will do a post on my blog when I get a chance. Only got it working yesterday after a lot of trial and error mostly due to the lack of GPU power. It's pretty much the standard comfyui svd workflow with extra upscaling.
    - deleted 3y ago
      [deleted]
- yieldcrv 3y ago
  reminds me of an ad for an FKK sauna club in central europe this would be good enough for promos of those!
- nuz 3y ago
  This is about a different model than stable diffusion. You're using SVD I'm guessing, the weights here aren't released.
kleiba 3y ago
OT: re "a spaceship destroyer approaching Earth" (example in the first row of the second set) - I cannot for the life of me generate an SD image of a two-person space ship (a shuttle, but more Star Wars X-Wing than USS Enterprise shuttle) flying towards the viewer. Any helpful tips?
- gaogao 3y ago
  Sketch it out and then ControlNet img2img
einpoklum 3y ago
I like how the destroyer is the love child of the Millenium Falcon and the front of a BSG battlestar. Also can't say no to a cat video (even if it doesn't really remind of Van Gogh).
dylan604 3y ago
I was chatting with a studio manager at a post f/x house when he received a demo reel that immediately was relocated to the trash without viewing it. I called him on it about not even looking at it before rejecting it. He had me take it out of the trash to play it. Before it stared, he told me exactly what would be on the tape and the order of the clips, which is exactly what was on the tape. He said it was the fall semester with such and such teacher at the school, and every submission was exactly the same. They were just the class assignments made into a demo reel. Everyone of these generative AI demo pages feels exactly like that. Nothing new other than we've done the same thing as someone else type of releases. Now, I'm willing to say I'm jaded, so if this represents something we haven't already seen umpteengillion times already, where did they bury the lede? It's all still uncanny valley. The wheels on do not look real. The wake from the viking's vacuum does not look real. I'm not sure what the corgi is walking on, but it doesn't look like the ground.
- echelon 3y ago
  That studio manager will be out of a job in five years if he doesn't get on the bandwagon and evolve. This is the film to digital transition. It's going to be rough for a bit, then take over everything all at once. The upside for these editors is that they'll be able to be single-person studios and pursue their own vision without meddling from studios or massive capital, equipment, and personnel needs.
  - dylan604 3y ago
    Ha! I took a phone call back when I was in the shiny round disc business that said based on the prices we were quoting, that we'd be out of business in a year because Apple released DVD Studio Pro. Instead, we gained even more business from people that tried to have their projects done in DVDSP, but then ran into road blocks making it impossible to do the job. They'd then come back to us, and receive an even higher quote since now there is even less time before their deadlines. cie la vie. You say out of business. I say, mediocrity all the way down from here.
    - echelon 3y ago
system2 3y ago
I would love to see something longer than half a second, one day. This is the 50th video demo and it is still the same. Extremely short blurry gifs, always single angle and something morphing. Ai's video generation progress seems to be not going forward.

156 points by smusamashah 3y ago

youssefabdelm 3y ago

better but... still sucks... cant wait til it works

londons_explore 3y ago

Is this another case where the architecture is trivial and obvious, but the data and compute requirements are obscene?

j45 3y ago
While this interests me, a lot of this stuff started obscene and became not obscene. A line of sight, however disappointing is still better than no line of sight. Maybe it will turn the open source efforts onto these things that are announced.
VikingCoder 3y ago
1. All the data in the entire world 2. ??? 3. Profit
6510 3y ago
They did wrote: "Images in this section is not generated by our model."

thequadehunter 3y ago

No code..as usual :/

sschueller 3y ago
Looks like a pretty standard comfyui svd workflow. https://github.com/thecooltechguy/ComfyUI-Stable-Video-Diffusion https://github.com/thecooltechguy/ComfyUI-Stable-Video-Diffu...
- GaggiX 3y ago
  This paper is not about SVD (Stable Video Diffusion).
- peab 3y ago
  that's a different model, by stability AI. Right now everyone is coming out with video models, and they're all very similar. Stability, Meta, Google, Nvidia, tencent, Alibaba all have released either demos or code for Stable Diffusion based video generation. They're all fairly similar but slightly different architecture.

ur-whale 3y ago

Amazing. The fact that we can synthesize stuff like this from text makes me believe the real world obeys some laws of physics we still haven't figured out. Specifically, the fact that when we interpolate in embedding space, the interpolated point still represents something almost always physically plausible (not always though, witness the surfing cat that doesn't get wet underwater).

thfuran 3y ago
But the embedding space is deliberately and laboriously constructed such that it represents realistic things at as many points as possible. That's what the training is, right?
renegade-otter 3y ago
Eh? It's automated statistical learning. Yes, it's very cool, but it is far from AGI, let alone magic. I am playing around with SD right now - pulling sleepless nights because it's so fascinating, but I do get to see the patterns. Generating unreal, magical stuff is actually a lot of fun, but if you want it to generate something that makes sense - good luck. It is really hard. I can get Elizabeth Olsen into a Halo suit in a war zone on an alien world, but getting her to sit in a car like a normal human being has been impossible. It's always some twisted horror show. The predictive model does not know what to do with it and how it all fits. That's after you clean up all the anatomical nightmare fuel like the fingers situation. This is why most of the SD artwork you see are magical elf girls in the foreground or some unearthly monsters. A well-defined subject in the foreground (a human-ish one), it will do VERY well. If you really want to get your imagination going with every little detail being near-perfect, you will be busy. Oh, and the text - it obviously cannot do text. It's in the right place, but it looks funny. This is amazing tech, but it's not about to go sentient and kill me at night.
- echelon 3y ago
  No no, I feel the OP is onto something. > obeys some laws of physics we still haven't figured out. That statement might have been over the top, but I think they're right with respect to the information. Spatial logic, optics, signal processing, reasoning. It's beginning to feel like there's an interesting representation that cross-cuts these spaces in a powerful, generalizable way.

plastic3169 3y ago

Seems gruel to even say this as the tech is pure magic, but sad that these videos often get that ultra processed stockfootage look. I would be more impressed to just have bland everyday videos with muted colors, shaky cam and uninteresting people doing everyday stuff.

nuz 3y ago
Loras can often achieve that look. No weights though but that would have likely worked to skew it in the 'amateur video' direction.
eurekin 3y ago
I imagine some kind of service could pop up, exactly for that. For pictures, we already have: https://magnific.ai/ https://magnific.ai/ Which seems to have been endorsed by one of the best retouchers I know of (https://www.youtube.com/@pratiknaikedu https://www.youtube.com/@pratiknaikedu)
- jacobolus 3y ago
  What they are asking for is to have these machine-generated "photorealistic" outputs look more like real photos/videos, not to turn every normal looking photo into a lurid dream-scene monstrosity with completely made up details.
- TaylorAlexander 3y ago
  Just a note you can run upscaling on your home desktop with Real-ESRGAN: https://github.com/xinntao/Real-ESRGAN https://github.com/xinntao/Real-ESRGAN

sschueller 3y ago

If you tweak the outputs you can get much better results but takes time and resources. You are also quite limited to the duration at this point. Here is an example I made yesterday, probably around 5 minutes of computing on my 3080ti: https://imgur.com/a/4dPv1hC https://imgur.com/a/4dPv1hC

eurekin 3y ago
Could you create a writeup (or record screen, or screenshot relevant config) how to reproduce that effect?
- sschueller 3y ago
  I will do a post on my blog when I get a chance. Only got it working yesterday after a lot of trial and error mostly due to the lack of GPU power. It's pretty much the standard comfyui svd workflow with extra upscaling.
  - deleted 3y ago
    [deleted]
yieldcrv 3y ago
reminds me of an ad for an FKK sauna club in central europe this would be good enough for promos of those!
nuz 3y ago
This is about a different model than stable diffusion. You're using SVD I'm guessing, the weights here aren't released.

kleiba 3y ago

OT: re "a spaceship destroyer approaching Earth" (example in the first row of the second set) - I cannot for the life of me generate an SD image of a two-person space ship (a shuttle, but more Star Wars X-Wing than USS Enterprise shuttle) flying towards the viewer. Any helpful tips?

gaogao 3y ago
Sketch it out and then ControlNet img2img

einpoklum 3y ago

I like how the destroyer is the love child of the Millenium Falcon and the front of a BSG battlestar. Also can't say no to a cat video (even if it doesn't really remind of Van Gogh).

dylan604 3y ago

I was chatting with a studio manager at a post f/x house when he received a demo reel that immediately was relocated to the trash without viewing it. I called him on it about not even looking at it before rejecting it. He had me take it out of the trash to play it. Before it stared, he told me exactly what would be on the tape and the order of the clips, which is exactly what was on the tape. He said it was the fall semester with such and such teacher at the school, and every submission was exactly the same. They were just the class assignments made into a demo reel. Everyone of these generative AI demo pages feels exactly like that. Nothing new other than we've done the same thing as someone else type of releases. Now, I'm willing to say I'm jaded, so if this represents something we haven't already seen umpteengillion times already, where did they bury the lede? It's all still uncanny valley. The wheels on do not look real. The wake from the viking's vacuum does not look real. I'm not sure what the corgi is walking on, but it doesn't look like the ground.

echelon 3y ago
That studio manager will be out of a job in five years if he doesn't get on the bandwagon and evolve. This is the film to digital transition. It's going to be rough for a bit, then take over everything all at once. The upside for these editors is that they'll be able to be single-person studios and pursue their own vision without meddling from studios or massive capital, equipment, and personnel needs.
- dylan604 3y ago
  Ha! I took a phone call back when I was in the shiny round disc business that said based on the prices we were quoting, that we'd be out of business in a year because Apple released DVD Studio Pro. Instead, we gained even more business from people that tried to have their projects done in DVDSP, but then ran into road blocks making it impossible to do the job. They'd then come back to us, and receive an even higher quote since now there is even less time before their deadlines. cie la vie. You say out of business. I say, mediocrity all the way down from here.
  - echelon 3y ago

system2 3y ago

I would love to see something longer than half a second, one day. This is the 50th video demo and it is still the same. Extremely short blurry gifs, always single angle and something morphing. Ai's video generation progress seems to be not going forward.