7 ms·
Video quality seems really good, but limitations are quite restrictive "Our model encounters challenges when processing extremely long videos (e.g. 200 frames o
by metalrain 2y ago
Video quality seems really good, but limitations are quite restrictive "Our model encounters challenges when processing extremely long videos (e.g. 200 frames or more)".
I'd say most videos in practice are longer than 200 frames, so lot more research is still needed.
- chompychop 2y agoI guess one can break videos into 200-frame chunks and process them independent of each other.
- prmoustache 2y agoAt 30fps, which is not high, that would mean chunks of less than 7 seconds. Doable but highly impractical to say the least.
- rowanG077 2y agoYou will probably have to have some overhang of time to get the state space to match enough to minimize flicker in between fragments.
- bredren 2y agoPerhaps a second pass that focuses on smoothing out the frames where the clips are joined.
- bmicraft 2y agoYou could probably mitigate this by using overlapping clips and fading between them. Pretty crude but could be close to unnoticeable, depending on how unstable the technique actually is.
- IanCal 2y ago7s is pretty alright, I've seen HLS chunks of 6 seconds, that's pretty common I think.
- _puk 2y ago6s was adopted as the "standard" by Apple [0]. For live streaming it's pretty common to see 2 or 3 seconds (reduces broadcast delay, but with some caveats). 0: https://dev.to/100mslive/introduction-to-low-latency-streaming-with-hls-27f1#:~:text=Reducing%20the%20default%20segment%20size https://dev.to/100mslive/introduction-to-low-latency-streami...
- sp332 2y agoMaybe they could do a lower framerate and then use a different AI tool to interpolate something smoother.
- littlestymaar 2y agoIt's not so much that it would be impratical (video streaming, like HLS or MPEG-Dash, requires to chunk videos in pieces of roughly this size) but you'd lose the inter-frame consistency at segments boundaries, and I suspect the resulting video would be flickering at the transition. It could work for TV or movies if done properly at the scene transition time though.
- whywhywhywhy 2y agoNot if there isn't coherency between those chunks
- anigbrowl 2y agoEasily solved, just overlap by ~40 frames and fade the upscaled last frames of chunk A into the start of chunk B before processing. Editors do tricks like this all the time.
- y04nn 2y agoAnd now you end up with 40 blurred frames for each transition.
- anigbrowl 2y ago'before processing'
- readyman 2y agoDecent editors may try that once, but they will give up right away because it will only work by coincidence.
- epolanski 2y agoThere has to be a way where you can do it intelligently in chunks and reduce noise along the chunk borders. Moreover I imagine that further research and power will do a lot, smarter, and quicker. Don't forget people had toy story-comparable games in a decade or so after it was originally rendered at 1536x922.
- drra 2y agoOr upscale every 4th frame for consistency. Upscaling in between frames should be much easier.
- deleted 2y ago[deleted]
- jasonjmcghee 2y agoStill potentially useful - predict the next k frames with a sliding window throughout the video. But idk how someone can write "extremely long videos" with a straight face when meaning seconds. Maybe "long frame sequences"
- KeplerBoy 2y agoFascinating how researchers put out amazing work and then claim that videos consisting of more than 200 frames are "extremely long". Would it kill them to say that the method works best on short videos/scenes?
- jsheard 2y agoTale as old as time, in graphics papers it's "our technique achieves realtime speeds" and then 8 pages down they clarify that they mean 30fps at 640x480 on an RTX 4090.
- anvuong 2y agoAt 24fps that's not even 10 seconds. Calling it extremely long is kinda defensive.
- lupusreal 2y ago10 seconds is what, about a dozen cuts in a modern movie? Much longer has people pulling out their phones.
- jsheard 2y ago:( "Our model encounters challenges when processing >200 frame videos" :) "Our model is proven production-ready using real-world footage from Taken 3" https://www.youtube.com/watch?v=gCKhktcbfQM https://www.youtube.com/watch?v=gCKhktcbfQM
- boogieknite 2y agoFreal. To the degree that i compulsively count seconds on shots until a show/movie has a few shots over 9 seconds then they "earn my trust" and i can let it go. Im fine
- bufferoverflow 2y agoThe average shot length in a modern movie is around 2.5 seconds (down from 12 seconds in 1930's). For animations it's around 15 seconds.
- throwup238 2y agoThe textures of objects need to maintain consistency across much larger time frames, especially at 4k where you can see the pores on someone's face in a closeup.
- vasco 2y agoI'm sure if you really want to burn money on compute you can do some smart windowing in the processing and use it on overlapping chunks and do an OK job.
- bookofjoe 2y agoThe Wright Brothers' first powered flight lasted 12 seconds Source: https://www.nasa.gov/history/115-years-ago-wright-brothers-make-history-at-kitty-hawk/#:~:text=After%20several%20unsuccessful%20attempts%2C%20on,of%206.8%20miles%20per%20hour https://www.nasa.gov/history/115-years-ago-wright-brothers-m....
- srveale 2y agoOur invention works best except for extremely long flight times of 13 seconds
- m3kw9 2y agoUnless they can predict a 2 hour movie in 200 frames.
- kazinator 2y agoBreak into chunks that overlap by, say, a second, upscale separately and then blend to reduce sudden transitions in the generated details to gradual morphing. The details changing every ten seconds or so is actually a good thing; the viewer is reminded that what they are seeing is not real, yet still enjoying a high resolution video full of high frequency content that their eyes crave.
- madduci 2y agoI think it encounters memory leaks and the usage of memory goes over the roof
- babypuncher 2y agoWell there goes my dreams of making my own Deep Space Nine remaster from DVDs.
- cryptonector 2y agoIt's good enough for "enhance, enhance, enhance" situations.
- deleted 2y ago[deleted]
- geysersam 2y agoWonder what happens if you run it piece-wise on every 200 frames. Perhaps it glitches in the interface.
- anigbrowl 2y agoIf you're using this for existing material you just cut into <=8 second chunks, no big deal. Could be an absolute boon for filmmakers, otoh a nightmare for privacy because this will be applied to surveillance footage.
- deleted 2y ago[deleted]
- kyriakos 2y agoIf I am understanding the limitations section of the paper it seems like the 200 frames depends on the scene, it may be worse or better.