7 ms·

There's a huge amount of research within this field. For a ranking list on a standardized test set, you can see e.g. http://vision.middlebury.edu/flow/eval/resu

There's a huge amount of research within this field. For a ranking list on a standardized test set, you can see e.g. http://vision.middlebury.edu/flow/eval/results/results-i1.php http://vision.middlebury.edu/flow/eval/results/results-i1.ph... (there are many others). DAIN is roughly in top 10. Sadly, there's next to no focus on the realtime use case, which is useful for e.g. sports. For my own project Futatabi (see https://nageru.sesse.net/ https://nageru.sesse.net/), I chose to reimplement one that's fairly obscure, DIS (Dense Inverse Search), on the GPU, so that I could get it realtime. IIRC, it's nowhere as good as DAIN is, but there's that 60 fps demand... :-)

dharma1 7y ago
good info! How much better are the new deep learning based methods compared to older optical flow approaches like Twixtor?
- Sesse__ 7y ago
  I've never actually tried Twixtor myself, but assuming Twixtor hasn't just adopted the DL methods, I would assume they're significantly better for the difficult cases. Especially occlusion and non-uniform flow (think small objects that move the opposite way from their background) are really difficult cases to get right with optical flow. You'll inevitably end up with some kind of artifacts. The good news is that if you're only upsampling by 2x, half of your frames are going to be original frames, which masks away most artifacts unless they're really bad. Do note that some of the DL stuff uses optical flow internally in some form. It turns out that just hallucinating pixels directly isn't necessarily giving the best possible results, but you can do stuff like putting an optical flow layer in your network (it's differentiable), or doing optical flow first and then using that as an input to a DL network. It's a good reminder that designing your network for the actual problem gives the best results. Generally I haven't kept track of the state of the art over the last year or so; the field is just too big, and I just had to go with a method that was implementable (especially without having tons of training data ready). I tested 5–6 different ones that I thought could reasonably approach real time, chose DIS, spent a while implementing it and then called it a day.

dharma1 7y ago

good info! How much better are the new deep learning based methods compared to older optical flow approaches like Twixtor?

Sesse__ 7y ago
I've never actually tried Twixtor myself, but assuming Twixtor hasn't just adopted the DL methods, I would assume they're significantly better for the difficult cases. Especially occlusion and non-uniform flow (think small objects that move the opposite way from their background) are really difficult cases to get right with optical flow. You'll inevitably end up with some kind of artifacts. The good news is that if you're only upsampling by 2x, half of your frames are going to be original frames, which masks away most artifacts unless they're really bad. Do note that some of the DL stuff uses optical flow internally in some form. It turns out that just hallucinating pixels directly isn't necessarily giving the best possible results, but you can do stuff like putting an optical flow layer in your network (it's differentiable), or doing optical flow first and then using that as an input to a DL network. It's a good reminder that designing your network for the actual problem gives the best results. Generally I haven't kept track of the state of the art over the last year or so; the field is just too big, and I just had to go with a method that was implementable (especially without having tons of training data ready). I tested 5–6 different ones that I thought could reasonably approach real time, chose DIS, spent a while implementing it and then called it a day.