4 ms·

If we can use GANs to produce believable images of a subject and we can identify whether a video frame can exist between two others, it seems like we can produc

If we can use GANs to produce believable images of a subject and we can identify whether a video frame can exist between two others, it seems like we can produce infinitely high framerate videos that look like they were captured that way. This also means we can make slow motion video when we didn't record at a high framerate. I think with that, colorization, and possibly similar tools for audio, we might see some amazing recreations of classic performances.

danielscrubs 7y ago
Scrubs in 4k!
gwern 7y ago
Yep, you certainly can. 'In-betweening', like superresolution, has been a GAN thing for years now, because triplets of frames are a clean dataset but you also care more about perceptual plausibility than pixel error. People use in-betweening GANs to make things like 60 FPS anime. (Not entirely sure why, but they do.)
- usmannk 7y ago
  That sounds interesting. Where can I read more about people doing this? (And other fun things with GANs and/or anime)
  - zitterbewegung 7y ago
    Gwern has a website https://www.gwern.net https://www.gwern.net
  - deleted 7y ago
    [deleted]
- cycrutchfield 7y ago
  I mean, both things have been around much much longer than GANs. I think Freeman’s paper on superresolution is from the early 2000s.
- rrishi 7y ago
  This is really interesting. Any place I can get to know more about this?
- ionionionioa 7y ago
  >People use in-betweening GANs to make things like 60 FPS anime Animation seems like an especially poor fit to me, since the actual framerate is often much lower than the video's framerate. Framerate can vary between scenes and even within different parts of one scene! Typically the background is very low framerate (sometimes as low as 4 FPS), the foreground is higher framerate (typically 8-12 FPS), while pans, zooms, and 3D elements are at a full 24 FPS. Most of the additional frames from interpolation will therefore be exact duplicates of other frames. This does little to improve the smoothness of the video. It just adds in artifacts. And, since the frames between two drawings will be interpolated while frames within one drawing will be unchanged, the framerate will be inconsistent and appear as judder. Interpolation will never work for 2D animation. No way, no how. Any worthwhile system will need to modify existing frames rather than simply adding more in between the original frames. I can understand interpolation for live action (though I still dislike it), but it is absolutely god-awful for animation.
  - gwern 7y ago
    I think that's wrong: the whole point of GANs is that they're quite intelligent and good at faking outputs. I've seen interpolated/in-betweened videos (mostly but not entirely live-action), and it looks realistic to me. The reason I'm somewhat skeptical is that just because something looks realistic doesn't mean that it what is intended. It's a version of the 'zoom in, enhance, enhance' problem. It's like the _Hobbit_ problem: a GAN could perfectly well fake a 60FPS version of a 30FPS version of the _Hobbit_ such that you couldn't tell that it wasn't the actual 60FPS version that Peter Jackson shot... but the problem is that it's 60FPS and that just feels wrong for cinema. Animators, anime included, use the limitations of framerate and deliberate switches between animating 'on twos' etc, with reductions in framerates being deliberately done for action segments and sakuga and other reasons. An anime isn't simply a film which was unavoidably shot with a too-low framerate. (This is less true of superresolution: in most cases, if an anime studio could have afforded to animate at a higher resolution originally, they would have; and you're not compromising any 'artistic vision' if you use a GAN to do a good upscaling job instead of a lousy bilinear upscale built into your video player.)
    - 7y ago
antpls 7y ago
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Frame Interpolation and Enhancement : https://sites.google.com/view/wenbobao/memc-net https://sites.google.com/view/wenbobao/memc-net One of the author also worked on https://news.ycombinator.com/item?id=20188316 https://news.ycombinator.com/item?id=20188316
shostack 7y ago
Couple this then be used to reduce the storage required for phones to capture slow motion video since it can simply be done with regular video in post server side?
kachurovskiy 7y ago
This is already possible in some video editors, see e.g. DaVinci Resolve optical flow - https://youtu.be/4LsLaJq5-ow https://youtu.be/4LsLaJq5-ow
spyder 7y ago
Yes: AI Learns Video Frame Interpolation | Two Minute Papers #197 https://www.youtube.com/watch?v=T_g6S3f0Z5I https://www.youtube.com/watch?v=T_g6S3f0Z5I

danielscrubs 7y ago

Scrubs in 4k!

gwern 7y ago

Yep, you certainly can. 'In-betweening', like superresolution, has been a GAN thing for years now, because triplets of frames are a clean dataset but you also care more about perceptual plausibility than pixel error. People use in-betweening GANs to make things like 60 FPS anime. (Not entirely sure why, but they do.)

usmannk 7y ago
That sounds interesting. Where can I read more about people doing this? (And other fun things with GANs and/or anime)
- zitterbewegung 7y ago
  Gwern has a website https://www.gwern.net https://www.gwern.net
- deleted 7y ago
  [deleted]
cycrutchfield 7y ago
I mean, both things have been around much much longer than GANs. I think Freeman’s paper on superresolution is from the early 2000s.
rrishi 7y ago
This is really interesting. Any place I can get to know more about this?
ionionionioa 7y ago
>People use in-betweening GANs to make things like 60 FPS anime Animation seems like an especially poor fit to me, since the actual framerate is often much lower than the video's framerate. Framerate can vary between scenes and even within different parts of one scene! Typically the background is very low framerate (sometimes as low as 4 FPS), the foreground is higher framerate (typically 8-12 FPS), while pans, zooms, and 3D elements are at a full 24 FPS. Most of the additional frames from interpolation will therefore be exact duplicates of other frames. This does little to improve the smoothness of the video. It just adds in artifacts. And, since the frames between two drawings will be interpolated while frames within one drawing will be unchanged, the framerate will be inconsistent and appear as judder. Interpolation will never work for 2D animation. No way, no how. Any worthwhile system will need to modify existing frames rather than simply adding more in between the original frames. I can understand interpolation for live action (though I still dislike it), but it is absolutely god-awful for animation.
- gwern 7y ago
  I think that's wrong: the whole point of GANs is that they're quite intelligent and good at faking outputs. I've seen interpolated/in-betweened videos (mostly but not entirely live-action), and it looks realistic to me. The reason I'm somewhat skeptical is that just because something looks realistic doesn't mean that it what is intended. It's a version of the 'zoom in, enhance, enhance' problem. It's like the _Hobbit_ problem: a GAN could perfectly well fake a 60FPS version of a 30FPS version of the _Hobbit_ such that you couldn't tell that it wasn't the actual 60FPS version that Peter Jackson shot... but the problem is that it's 60FPS and that just feels wrong for cinema. Animators, anime included, use the limitations of framerate and deliberate switches between animating 'on twos' etc, with reductions in framerates being deliberately done for action segments and sakuga and other reasons. An anime isn't simply a film which was unavoidably shot with a too-low framerate. (This is less true of superresolution: in most cases, if an anime studio could have afforded to animate at a higher resolution originally, they would have; and you're not compromising any 'artistic vision' if you use a GAN to do a good upscaling job instead of a lousy bilinear upscale built into your video player.)
  - 7y ago

antpls 7y ago

MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Frame Interpolation and Enhancement : https://sites.google.com/view/wenbobao/memc-net https://sites.google.com/view/wenbobao/memc-net One of the author also worked on https://news.ycombinator.com/item?id=20188316 https://news.ycombinator.com/item?id=20188316

shostack 7y ago

Couple this then be used to reduce the storage required for phones to capture slow motion video since it can simply be done with regular video in post server side?

kachurovskiy 7y ago

This is already possible in some video editors, see e.g. DaVinci Resolve optical flow - https://youtu.be/4LsLaJq5-ow https://youtu.be/4LsLaJq5-ow

spyder 7y ago

Yes: AI Learns Video Frame Interpolation | Two Minute Papers #197 https://www.youtube.com/watch?v=T_g6S3f0Z5I https://www.youtube.com/watch?v=T_g6S3f0Z5I