4 ms·

Does this algorithm lend itself to video? Probably would take a massive cloud of systems to correct any highdef video signal, but it would be impressive for ma

by mey 15y ago

Does this algorithm lend itself to video? Probably would take a massive cloud of systems to correct any highdef video signal, but it would be impressive for many applications (news broadcast, sports, or any live event, security, or remote robots) Granted to achieve performance on the order of near realtime dsp, it would require an impressive hardware system. Then again, when I can spend the price of a coffee and get access to a cloud of cpu's...

shabble 15y ago
Without having looked into it too much (mostly just the video), there seem to be fairly strong parallels with the motion tracking/estimation that modern video codecs do. As I understand it, their system is trying to estimate the causes of noise in the image (camera shake, badly focused or dirty lens, etc) and apply an inverse transformation to produce the original 'real' image. They can't extract detail out of nowhere, so a lot of it will likely be heuristically driven, accounting for specific sources of error and applying adaptive transforms with some sort of 'looks like what we think it should look like' cost function. Figuring out the cost function is probably one of the hardest parts here. In terms of video, you've got more data to work with, which in theory means more constraints to your solution making it potentially more accurate, but I suspect it'd also be very easy to get bogged down in the quantity of data. Combining and merging data from multiple independent sources is a different problem entirely, probably more like synthetic aperture imaging: http://vision.ucsd.edu/kriegman-grp/research/synthetic_ap_tracking/ http://vision.ucsd.edu/kriegman-grp/research/synthetic_ap_tr...
dpe82 15y ago
One of the things I did during my stint at Sony was video stabilization for their desktop nonlinear video editing software. We were hoping to not only stabilize (which is a pretty straightforward thing to do these days) but also use the motion data to inform a deconvolution to clean up motion blurred frames. Alas, my project ran out of time and we ended up licensing a third party stabilizer. But yes, most image processing techniques like this lend themselves to video. This is something I hope my startup will someday be able to integrate into our collaborative video editor...
- dpe82 15y ago
  I should clarify: many things like this lend themselves to video, but not directly. As ibuildthings pointed out you have to be very careful becuase for video the human eye is very sensitive to the sharp edge artifacts algorithms like this can produce; much more so than it is to motion blur.
  - Geee 15y ago
    Yep, I can imagine continuity problems, flicker etc. could show up in video.
mseebach 15y ago
For a while digital camcorders has had a digital image stabilisation feature build in. It seems to work like this. http://camcorders.about.com/od/camcorders101/a/optical_vs_digital_image_stabilization.htm http://camcorders.about.com/od/camcorders101/a/optical_vs_di...
ibuildthings 15y ago
Actually human eye/brain is much more forgiving to motion blur in video that in still images. The chances are that one will spot quite a number of artifacts [compression as well as blur] in a grabbed frame of an HD quality video, which is otherwise unacceptable in a photograph. In theory, you can tweak this method such that it is run in parallel frames 'almost' independently. I say 'almost' since you will have to factor in temporal coherence, such that deconvolution kernels do not vary much between subsequent frames (otherwise it will cause unwarranted flicker and jitter like effects) However, more pertinent and related problem with respect to video is that of image stabilization. And computer vision community is making some exciting strides in the front; for eg: http://web.cecs.pdx.edu/~fliu/ http://web.cecs.pdx.edu/~fliu/ .

by mey 15y ago

shabble 15y ago

Without having looked into it too much (mostly just the video), there seem to be fairly strong parallels with the motion tracking/estimation that modern video codecs do. As I understand it, their system is trying to estimate the causes of noise in the image (camera shake, badly focused or dirty lens, etc) and apply an inverse transformation to produce the original 'real' image. They can't extract detail out of nowhere, so a lot of it will likely be heuristically driven, accounting for specific sources of error and applying adaptive transforms with some sort of 'looks like what we think it should look like' cost function. Figuring out the cost function is probably one of the hardest parts here. In terms of video, you've got more data to work with, which in theory means more constraints to your solution making it potentially more accurate, but I suspect it'd also be very easy to get bogged down in the quantity of data. Combining and merging data from multiple independent sources is a different problem entirely, probably more like synthetic aperture imaging: http://vision.ucsd.edu/kriegman-grp/research/synthetic_ap_tracking/ http://vision.ucsd.edu/kriegman-grp/research/synthetic_ap_tr...

dpe82 15y ago

One of the things I did during my stint at Sony was video stabilization for their desktop nonlinear video editing software. We were hoping to not only stabilize (which is a pretty straightforward thing to do these days) but also use the motion data to inform a deconvolution to clean up motion blurred frames. Alas, my project ran out of time and we ended up licensing a third party stabilizer. But yes, most image processing techniques like this lend themselves to video. This is something I hope my startup will someday be able to integrate into our collaborative video editor...

dpe82 15y ago
I should clarify: many things like this lend themselves to video, but not directly. As ibuildthings pointed out you have to be very careful becuase for video the human eye is very sensitive to the sharp edge artifacts algorithms like this can produce; much more so than it is to motion blur.
- Geee 15y ago
  Yep, I can imagine continuity problems, flicker etc. could show up in video.

mseebach 15y ago

For a while digital camcorders has had a digital image stabilisation feature build in. It seems to work like this. http://camcorders.about.com/od/camcorders101/a/optical_vs_digital_image_stabilization.htm http://camcorders.about.com/od/camcorders101/a/optical_vs_di...

ibuildthings 15y ago

Actually human eye/brain is much more forgiving to motion blur in video that in still images. The chances are that one will spot quite a number of artifacts [compression as well as blur] in a grabbed frame of an HD quality video, which is otherwise unacceptable in a photograph. In theory, you can tweak this method such that it is run in parallel frames 'almost' independently. I say 'almost' since you will have to factor in temporal coherence, such that deconvolution kernels do not vary much between subsequent frames (otherwise it will cause unwarranted flicker and jitter like effects) However, more pertinent and related problem with respect to video is that of image stabilization. And computer vision community is making some exciting strides in the front; for eg: http://web.cecs.pdx.edu/~fliu/ http://web.cecs.pdx.edu/~fliu/ .