4 ms·

> synthesizes multiple intermediate frames from two input images That's a neat use case, and definitely a good way to show off, but what about more than one im

> synthesizes multiple intermediate frames from two input images That's a neat use case, and definitely a good way to show off, but what about more than one image? The overwhelming majority of video that exists today is 30fps or lower. The overwhelming majority of displays support 60hz or more. Most high-end TVs do some realtime frame interpolation, but there is only so much an algorithm can do to fill in the blanks. It doesn't take long to see artifacts. I would be more interested to see what an ML-based approach could do with the edge cases of interpolating 30fps video than 2 frames.

zlatan28 4y ago
They can upsample FPS on videos (more than 2 frames), https://github.com/google-research/frame-interpolation#many-in-between-frames-interpolation https://github.com/google-research/frame-interpolation#many-...
summerlight 4y ago
Actually most of the video frame interpolation programs in the market uses two frames interpolation. Theoretically, you can do a better job with multiple frames but this doesn't bring much more values beside of some extreme cases.
- thomastjeffery 4y ago
  > Actually most of the video frame interpolation programs in the market uses two frames interpolation. > Theoretically, you can do a better job with multiple frames but this doesn't bring much more values beside of some extreme cases. Edge cases that require more information than is present in two frames are very common. That's why most frame interpolation methods also have an "artifact masking" feature. But what if we did use the information from surrounding frames? That would probably be too complicated for traditional frame interpolation, but that's not what we're talking about. What if we used a data set trained on the entire video file - or even a collection of similar video files - to fill in the gaps?

zlatan28 4y ago

They can upsample FPS on videos (more than 2 frames), https://github.com/google-research/frame-interpolation#many-in-between-frames-interpolation https://github.com/google-research/frame-interpolation#many-...

summerlight 4y ago

Actually most of the video frame interpolation programs in the market uses two frames interpolation. Theoretically, you can do a better job with multiple frames but this doesn't bring much more values beside of some extreme cases.

thomastjeffery 4y ago
> Actually most of the video frame interpolation programs in the market uses two frames interpolation. > Theoretically, you can do a better job with multiple frames but this doesn't bring much more values beside of some extreme cases. Edge cases that require more information than is present in two frames are very common. That's why most frame interpolation methods also have an "artifact masking" feature. But what if we did use the information from surrounding frames? That would probably be too complicated for traditional frame interpolation, but that's not what we're talking about. What if we used a data set trained on the entire video file - or even a collection of similar video files - to fill in the gaps?