5 ms·

Flexible diffusion modeling of long videos

65 points by thorum 4y ago

ur-whale 4y ago
There has been so much fun made of the infinite monkey theorem [1] over the years, but look where we are now ! [1] https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem
dreadlordbone 4y ago
hugged to death
SonOfLilit 4y ago
I was really hoping this was a "diffusion model" in the same sense that these guys built a reaction-diffusion based self-healing system: https://distill.pub/2020/growing-ca https://distill.pub/2020/growing-ca
- jawarner 4y ago
  The reaction diffusion / NCA models have been applied to videos recently — check it out: https://wandb.ai/johnowhitaker/nca/reports/Fun-with-Neural-Cellular-Automata--VmlldzoyMDQ5Mjg0 https://wandb.ai/johnowhitaker/nca/reports/Fun-with-Neural-C...
- jiggawatts 4y ago
  That was a fascinating article with a great mix of readability and interactive demos.
- sampo 4y ago
  Yeah, the neural network "diffusion models" are not very well named. If you have background in natural sciences, you would understand diffusion to mean, well, diffusion. Whereas there generative neural networks are about (1) blurring data by Gaussian noise, (2) teaching a NN to denoise the noised data, and finally (3) with Gaussian noise as input, let the denoiser NN to generate new data. So it's not so much about diffusion as it's about reversing the diffusion. And it's not really (smooth) diffusion, but Gaussian noise. "Denoising autoencoder" is already used for processes that reconstruct partially corrupted input. So what name to suggest for a process that reconstructs data from nothing but noise?
dane-pgp 4y ago
Now this just needs to be integrated with DALL-E/Imagen and GPT-3 (plus a text to speech engine), to create an offline version of YouTube.
- mensetmanusman 4y ago
  It’s definitely fun contemplating a future where there is nothing special on video.
  - contravariant 4y ago
    Finally a way to restrict the library of babel to (seemingly) meaningful books!
- politician 4y ago
  Great idea! It’ll need someone to train a censorship engine to complete the illusion.
  - dane-pgp 4y ago
    You'll be relieved to hear that those models already have censorship engines built in to try to prevent them generating problematic content.
    - Gigachad 4y ago
      It still doesn’t work because even the latest AI tech is unable to understand the complex rules of what problematic content is. How can you identify socially acceptable bias. For example, you’d expect it to be biased towards cars with 4 wheels vs rare 3 wheeled cars, but how does it know that bias is ok but being biased to male lawyers isn’t. And then the billion other similar situations.
      - deleted 4y ago
        [deleted]
      - 4y ago
silencedogood3 4y ago

65 points by thorum 4y ago

There has been so much fun made of the infinite monkey theorem [1] over the years, but look where we are now ! [1] https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem

dreadlordbone 4y ago

hugged to death

SonOfLilit 4y ago

I was really hoping this was a "diffusion model" in the same sense that these guys built a reaction-diffusion based self-healing system: https://distill.pub/2020/growing-ca https://distill.pub/2020/growing-ca

jawarner 4y ago
The reaction diffusion / NCA models have been applied to videos recently — check it out: https://wandb.ai/johnowhitaker/nca/reports/Fun-with-Neural-Cellular-Automata--VmlldzoyMDQ5Mjg0 https://wandb.ai/johnowhitaker/nca/reports/Fun-with-Neural-C...
jiggawatts 4y ago
That was a fascinating article with a great mix of readability and interactive demos.
sampo 4y ago
Yeah, the neural network "diffusion models" are not very well named. If you have background in natural sciences, you would understand diffusion to mean, well, diffusion. Whereas there generative neural networks are about (1) blurring data by Gaussian noise, (2) teaching a NN to denoise the noised data, and finally (3) with Gaussian noise as input, let the denoiser NN to generate new data. So it's not so much about diffusion as it's about reversing the diffusion. And it's not really (smooth) diffusion, but Gaussian noise. "Denoising autoencoder" is already used for processes that reconstruct partially corrupted input. So what name to suggest for a process that reconstructs data from nothing but noise?

dane-pgp 4y ago

Now this just needs to be integrated with DALL-E/Imagen and GPT-3 (plus a text to speech engine), to create an offline version of YouTube.

mensetmanusman 4y ago
It’s definitely fun contemplating a future where there is nothing special on video.
- contravariant 4y ago
  Finally a way to restrict the library of babel to (seemingly) meaningful books!
politician 4y ago
Great idea! It’ll need someone to train a censorship engine to complete the illusion.
- dane-pgp 4y ago
  You'll be relieved to hear that those models already have censorship engines built in to try to prevent them generating problematic content.
  - Gigachad 4y ago
    It still doesn’t work because even the latest AI tech is unable to understand the complex rules of what problematic content is. How can you identify socially acceptable bias. For example, you’d expect it to be biased towards cars with 4 wheels vs rare 3 wheeled cars, but how does it know that bias is ok but being biased to male lawyers isn’t. And then the billion other similar situations.
    - deleted 4y ago
      [deleted]
    - 4y ago

silencedogood3 4y ago