4 ms·

DAIN: Depth-Aware Video Frame Interpolation

69 points by ArtWomb 7y ago

ArtWomb 7y ago
Original research: https://sites.google.com/view/wenbobao/dain https://sites.google.com/view/wenbobao/dain
Animats 7y ago
Oh, like Framefree, from 2006.[1][2] Framefree worked on harder cases, too; not just video game characters with nice sharp edges. That was a really good idea. It came from Kerner Optical, which was spun off from Lucasfilm and went bust. They had their own frameless video compression format, and there was a Firefox plug-in for playing it. Some of that technology went into systems for making 3D movies from 2D movies, and ultra slow motion for sports. I saw it up at Kerner, in Marin north of SF, about 15 years ago, and I had the plug-in and some files it could play. It's basically an automatic delaminator (the hard part) that separates an image into multiple planes. Then each plane from two images in a sequence has morph points found. The interpolation is a standard 2D morph. You can morph as slowly as you want. The business end of the operation was totally botched - there were holding companies, an offshore shell company in Jersey, something called Monolith, and some company called Quin Land in Japan. All of which seem to have sunk without a trace. Sad. I wonder if anybody paid the maintenance fees on the patents. [1] https://web.archive.org/web/20081216024454if_/http://www.framefree.com/ https://web.archive.org/web/20081216024454if_/http://www.fra... [2] https://www.youtube.com/watch?v=VBfss0AaNaU https://www.youtube.com/watch?v=VBfss0AaNaU
- jedimastert 7y ago
  > Framefree worked on harder cases, too; not just video game characters with nice sharp edges. Check out the claymation example: https://www.youtube.com/watch?v=eAUn7Nvx73s https://www.youtube.com/watch?v=eAUn7Nvx73s
  - foota 7y ago
    What's with the artifacts down the middle? Is one side 60 fps and the other not?
    - afrodc_ 7y ago
      Yes. It seems like the left is without the effect and the right shows the improvement with the technique.
      - foota 7y ago
        Ok, thanks!
sabalaba 7y ago
Very impressive work. I would love to see what this looks like when you start to drop more and more frames. I’d be curious to see how this technique starts to degrade as the input FPS goes down. Seems like it already can dramatically reduce the amount of work involved in making an animation. Very cool to see pixel art animated at 60 FPS!
Sesse__ 7y ago
There's a huge amount of research within this field. For a ranking list on a standardized test set, you can see e.g. http://vision.middlebury.edu/flow/eval/results/results-i1.php http://vision.middlebury.edu/flow/eval/results/results-i1.ph... (there are many others). DAIN is roughly in top 10. Sadly, there's next to no focus on the realtime use case, which is useful for e.g. sports. For my own project Futatabi (see https://nageru.sesse.net/ https://nageru.sesse.net/), I chose to reimplement one that's fairly obscure, DIS (Dense Inverse Search), on the GPU, so that I could get it realtime. IIRC, it's nowhere as good as DAIN is, but there's that 60 fps demand... :-)
- dharma1 7y ago
  good info! How much better are the new deep learning based methods compared to older optical flow approaches like Twixtor?
  - Sesse__ 7y ago
    I've never actually tried Twixtor myself, but assuming Twixtor hasn't just adopted the DL methods, I would assume they're significantly better for the difficult cases. Especially occlusion and non-uniform flow (think small objects that move the opposite way from their background) are really difficult cases to get right with optical flow. You'll inevitably end up with some kind of artifacts. The good news is that if you're only upsampling by 2x, half of your frames are going to be original frames, which masks away most artifacts unless they're really bad. Do note that some of the DL stuff uses optical flow internally in some form. It turns out that just hallucinating pixels directly isn't necessarily giving the best possible results, but you can do stuff like putting an optical flow layer in your network (it's differentiable), or doing optical flow first and then using that as an input to a DL network. It's a good reminder that designing your network for the actual problem gives the best results. Generally I haven't kept track of the state of the art over the last year or so; the field is just too big, and I just had to go with a method that was implementable (especially without having tons of training data ready). I tested 5–6 different ones that I thought could reasonably approach real time, chose DIS, spent a while implementing it and then called it a day.

69 points by ArtWomb 7y ago

Original research: https://sites.google.com/view/wenbobao/dain https://sites.google.com/view/wenbobao/dain

Oh, like Framefree, from 2006.[1][2] Framefree worked on harder cases, too; not just video game characters with nice sharp edges. That was a really good idea. It came from Kerner Optical, which was spun off from Lucasfilm and went bust. They had their own frameless video compression format, and there was a Firefox plug-in for playing it. Some of that technology went into systems for making 3D movies from 2D movies, and ultra slow motion for sports. I saw it up at Kerner, in Marin north of SF, about 15 years ago, and I had the plug-in and some files it could play. It's basically an automatic delaminator (the hard part) that separates an image into multiple planes. Then each plane from two images in a sequence has morph points found. The interpolation is a standard 2D morph. You can morph as slowly as you want. The business end of the operation was totally botched - there were holding companies, an offshore shell company in Jersey, something called Monolith, and some company called Quin Land in Japan. All of which seem to have sunk without a trace. Sad. I wonder if anybody paid the maintenance fees on the patents. [1] https://web.archive.org/web/20081216024454if_/http://www.framefree.com/ https://web.archive.org/web/20081216024454if_/http://www.fra... [2] https://www.youtube.com/watch?v=VBfss0AaNaU https://www.youtube.com/watch?v=VBfss0AaNaU

jedimastert 7y ago
> Framefree worked on harder cases, too; not just video game characters with nice sharp edges. Check out the claymation example: https://www.youtube.com/watch?v=eAUn7Nvx73s https://www.youtube.com/watch?v=eAUn7Nvx73s
- foota 7y ago
  What's with the artifacts down the middle? Is one side 60 fps and the other not?
  - afrodc_ 7y ago
    Yes. It seems like the left is without the effect and the right shows the improvement with the technique.
    - foota 7y ago
      Ok, thanks!

sabalaba 7y ago

Very impressive work. I would love to see what this looks like when you start to drop more and more frames. I’d be curious to see how this technique starts to degrade as the input FPS goes down. Seems like it already can dramatically reduce the amount of work involved in making an animation. Very cool to see pixel art animated at 60 FPS!

Sesse__ 7y ago

There's a huge amount of research within this field. For a ranking list on a standardized test set, you can see e.g. http://vision.middlebury.edu/flow/eval/results/results-i1.php http://vision.middlebury.edu/flow/eval/results/results-i1.ph... (there are many others). DAIN is roughly in top 10. Sadly, there's next to no focus on the realtime use case, which is useful for e.g. sports. For my own project Futatabi (see https://nageru.sesse.net/ https://nageru.sesse.net/), I chose to reimplement one that's fairly obscure, DIS (Dense Inverse Search), on the GPU, so that I could get it realtime. IIRC, it's nowhere as good as DAIN is, but there's that 60 fps demand... :-)

dharma1 7y ago
good info! How much better are the new deep learning based methods compared to older optical flow approaches like Twixtor?
- Sesse__ 7y ago
  I've never actually tried Twixtor myself, but assuming Twixtor hasn't just adopted the DL methods, I would assume they're significantly better for the difficult cases. Especially occlusion and non-uniform flow (think small objects that move the opposite way from their background) are really difficult cases to get right with optical flow. You'll inevitably end up with some kind of artifacts. The good news is that if you're only upsampling by 2x, half of your frames are going to be original frames, which masks away most artifacts unless they're really bad. Do note that some of the DL stuff uses optical flow internally in some form. It turns out that just hallucinating pixels directly isn't necessarily giving the best possible results, but you can do stuff like putting an optical flow layer in your network (it's differentiable), or doing optical flow first and then using that as an input to a DL network. It's a good reminder that designing your network for the actual problem gives the best results. Generally I haven't kept track of the state of the art over the last year or so; the field is just too big, and I just had to go with a method that was implementable (especially without having tons of training data ready). I tested 5–6 different ones that I thought could reasonably approach real time, chose DIS, spent a while implementing it and then called it a day.