2 ms·

good info! How much better are the new deep learning based methods compared to older optical flow approaches like Twixtor?

• Sesse__ 7y ago
  I've never actually tried Twixtor myself, but assuming Twixtor hasn't just adopted the DL methods, I would assume they're significantly better for the difficult cases. Especially occlusion and non-uniform flow (think small objects that move the opposite way from their background) are really difficult cases to get right with optical flow. You'll inevitably end up with some kind of artifacts. The good news is that if you're only upsampling by 2x, half of your frames are going to be original frames, which masks away most artifacts unless they're really bad. Do note that some of the DL stuff uses optical flow internally in some form. It turns out that just hallucinating pixels directly isn't necessarily giving the best possible results, but you can do stuff like putting an optical flow layer in your network (it's differentiable), or doing optical flow first and then using that as an input to a DL network. It's a good reminder that designing your network for the actual problem gives the best results. Generally I haven't kept track of the state of the art over the last year or so; the field is just too big, and I just had to go with a method that was implementable (especially without having tons of training data ready). I tested 5–6 different ones that I thought could reasonably approach real time, chose DIS, spent a while implementing it and then called it a day.