7 ms·
FILM: Frame Interpolation for Large Motion
- vans554 4y agovery cool how does this compare to NVIDIA nvfruc? https://docs.nvidia.com/video-technologies/optical-flow-sdk/nvfruc-programming-guide/index.html#frame-rate-upconversion https://docs.nvidia.com/video-technologies/optical-flow-sdk/...
- summerlight 4y agoThe target application is somewhat different. FILM is more focused on high quality animation generation while nvfruc is for real-time streaming.
- vans554 4y agoYea but this is irrelevent unless you implying that FILM is better quality while nvfruc is lower quality but faster so applicable for realtime? if nvfruc is better quality + faster then the comparison is that nvfruc is better.
- summerlight 4y agonvfruc is extremely limited in its capability, basically small motion interpolation between two frames within a very short interval. FILM (and other deep learning based interpolation methods) is capable of interpolating two frames with large motion differences at a significantly larger computational cost. I don't understand why you're trying to imply nvfruc could be better at both cost and quality? Even Nvidia won't make such bold claims.
- vans554 4y agoThanks.
- modeless 4y agoSomeone must be doing this for animation, right? I want to watch all the animated classics at full frame rate. Purists may gasp but I think that it won't be long before these models can do a lot better than mere interpolation, preserving the original feel of sequences despite the increased frame rate and avoiding the "soap opera" effect.
- teraflop 4y agoThe "soap opera" effect is mainly a perceptual response to the high frame rate itself, even if there are no interpolation artifacts. That's why it has its name: soap operas are typically filmed at 60fps, with cameras that cameras one full frame every 1/60th of a second.
- AstralStorm 4y agoMore specifically, the reaction to lack of blur. You can get 60 fps upconverted footage from motion interpolation to not exhibit the effect by adding copious gaussian motion blur. The same problem (plus perceived judder due to low fps) appears if you play a video on a screen equipped with correctly tuned strobe backlight. It's one of the reasons as to why simple kernel interpolation (blurmotion) with most kernels does not exhibit this effect either.
- ubercow13 4y agoWhy would there be a lack of blur? The 24fps (say) footage will start with more blur as the shutter speed will be slower and this blur will remain when interpolated to 60fps. It would surely have a physically impossibly high amount of blur after interpolation.
- dagmx 4y agoInterpolation doesn’t change shutter angle which is what causes motion blur to have a specific aesthetic look, so when interpolated it looks wrong for the frame rate. You’re right in that it doesn’t change the amount of blur, I’m not sure why the person you’re replying to thinks that. But it does change the aesthetic of the blur. Perhaps they’re thinking of the issue with the Hobbit where the higher frame rate at a given shutter angle (180 let’s say) that shortened the blur compared to a traditional 24 fps film. They adjusted this for the second hobbit film to leave the shutter open longer (not a 360 shutter, but something like a 270) to bring back some of that feeling.
- worldofmatthew 4y agoCan we please stop using common names for new technologies? will make it harder to reference in a decade.
- zamadatix 4y agoIt'll never happen. Best to qualify when searching "FILM interpolation" or expand when introducing "Frame Interpolation for Large Motion" as was done here. That way you get the best of both worlds, friendly acronym for active use and extremely referenceable name for long term or first time use. The long form is probably what you'd need to do without common abbreviations anyways, weird acronyms just to be unique doesn't help when actively using it now or a decade later when you're trying to reference back to it and nobody knows what gLMFI1 was anyways - unless it becomes extremely popular in which case gLMFI1 is a pain to use all the time.
- prox 4y agoIt’s not a distaster but it is better to use made up names, like filmz, filmy, filmerz, filmstack .. the list goes on.
- deleted 4y ago[deleted]
- kentlyons 4y agoThey give the full proper citation for referencing the work: @inproceedings{reda2022film, title = {FILM: Frame Interpolation for Large Motion}, author = {Fitsum Reda and Janne Kontkanen and Eric Tabellion and Deqing Sun and Caroline Pantofaru and Brian Curless}, booktitle = {European Conference on Computer Vision (ECCV)}, year = {2022} }
- Hard_Space 4y agoYup. There's another framework called 'Feature-wise Linear Modulate' (FiLM), also related to video, but otherwise unrelated (note the lowercase 'i'); and thanks to Hasbro, Neural Radiance Fields (NeRF) is surprisingly difficult to search (but Hasbro was first, so NeRF not a great choice). I've seen 2-3 other examples of insane name duplication in CV research in the last couple of months.
- mzs 4y agoYikes the teeth just materialize like in some nightmare! https://film-net.github.io/static/images/000204/interpolated-2s.mp4 https://film-net.github.io/static/images/000204/interpolated...
- rob74 4y agoYeah, even in the example from the README, something seems a bit off, although I can't put my finger on it. But I wonder how the animation would look if the subject turns its head...
- Ftuuky 4y agoNone of the demos are properly working for me. Am I doing something wrong, like the photos are too large?
- snickmy 4y agoWhile I'm really impressed by the work in this paper (and codebase) I cannot stress enough that having minor as examples is definitely not a good idea on so many levels!
- exit 4y agoi appreciate that "I cannot stress enough" is somewhat idiomatic, however it strikes me as odd to announce this but then not offer least some of the specific reasons you believe this.
- Gordonjcp 4y agoWhy is that?
- snickmy 4y agothere are aspect of copyright law, and ownership of that image (this is work done inside google) that a minor, which eventually be an adult have no control. Additionally, minor images can be abused (I'm leaving at it, but you can assume that some AI breakthrough in generative imagines can be abused)
- angry_octet 4y agoOne of the immediately noticable uncanny valley effects is the eye appear to pan up, which never occurs in real life -- the eye jerks to a new direction as a step movement, called a saccade. From an ML perspective, I would have assumed they would train on high frame rate video, but it seems they didn't, they're estimating motion of pixel clumps. I don't see how it can ever do things like eyes/smiles/teeth. Useful for reconstructed pseudo slideshows in Google Photos with added vampire effect.