5 ms·

Accurate Image Alignment and Registration Using OpenCV

166 points by magamig 5y ago

BigComrade 5y ago
Affine transformations are very useful for algorithmic image rotation/translation/reflection [0]. [0] https://en.wikipedia.org/wiki/Affine_transformation https://en.wikipedia.org/wiki/Affine_transformation
vanderZwan 5y ago
Image alignment allows for some fun image manipulations like these. I think one of the coolest novel applications of image alignment in recent years must be multi-frame super-resolution. For example, the Pixel phones use it to improve low-light and zoomed in photos[0]. [0] https://sites.google.com/view/handheld-super-res/ https://sites.google.com/view/handheld-super-res/
- magamig 5y ago
  Cool! I'm planning on using image alignment for enhancing the spatial resolution of hyperspectral (HS) images, through the fusion of RGB and HS images.
  - is_true 5y ago
    You get something similar to a panchromatic image?
    - magamig 5y ago
      You can use an high-resolution panchromatic image to increase the spatial resolution of an RGB image. Here the objective is to do the apply the same idea but using an RGB and HS image, increasing the spatial resolution of the latter.
- gliptic 5y ago
  This is called drizzle (with dithering, e.g. the raw photos are taken with offsets) in the astrophotography sphere and is very effective when the images are undersampled. Although, astronomy images have the advantage of being filled with stars that are relatively easy to align very accurately.
  - vanderZwan 5y ago
    I imagine the minuscule vibrations of a shutter make it a natural fit to the already existing image stacking pipeline, correct?
    - gliptic 5y ago
      I'm not sure what you mean. Mechanical shutters have no upsides I'm aware of :). Astro cameras use electronic shutters. EDIT: The dithering, or offsetting, is done by moving the telescope slightly between exposures. During exposures you want it to track the object you're photographing as accurately as possible. I imagine vibrations are a detriment even in the Pixel phone processing. It would be better if the phone could move instantly in slightly different directions and take steady raw photos in each.
      - vanderZwan 5y ago
        Ah, I was thinking of amateur astrophotography - although these days you can just set the mechanical shutter in many consumer DSLRs to stay open too I think.
        galangalalgol 5y ago
        I think maybe you were considering the vibrations a source of noise for compressed sensing? For CS to work well you generally want to know the convolved noise. So a super accurate accelerometer might be able to measure random vibrations and do even better when they are present than absent. Not sure if phone accelerometers are that good.
    - xhkkffbf 5y ago
- amelius 5y ago
  Does this mean that objects that are on a trajectory out of the view have less resolution?
- legulere 5y ago
  I wonder if you could use alignment of multiple samples for better vectorizing of letters of old books. You could create very nice vector only pdfs.
  - Syzygies 5y ago
    Yes! For a two-fer, take a pair of cell phone photos of the open book, compute a 3D model, and flatten it as if the book had been sliced and scanned. Now match up all the letter E's in the entire book for image enhancement. Vectorization is nontrivial. Most academic journal articles are now poorly scanned, and it would be nice to simply improve the scans.
shrx 5y ago
The biggest issue in real life use cases is lens distortion - these alignment algorithms usually don't correct barrel, pin-cushion or more complex lens distortions, making the alignment imperfect.
- nullc 5y ago
  Any of the standard panorama software can solve for a standard simple lens distortion model, it just needs a few more alignment points to do so.
- zwieback 5y ago
  OpenCV, which is what the OP is using, has higher order camera calibration. There are functions to calibrate based on checkerboards or dot grids and undistort images using those calibrations. Sometimes it's better to match first, then undistort, though.
thebigman433 5y ago
I currently working on an optical respiratory monitor and one of the challenges was image alignment/registration for an optical and a thermal image with wildly different resolutions, fovs. We ended up doing this by manually calibrating the images in matlab using a custom script and logging out a transformation matrix that we could then multiply the optical image by to get matching pixels in the thermal image. Really fun project, but definitely a tedious thing to do, especially when its only a small part of the overall project. The project in the link also looks very cool, just not quite right for us as we probably wouldnt consistently have enough landmarks to map.
- versatran01 5y ago
  Direct methods could be useful in your case. https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf
  - SCUSKU 5y ago
    At work I had to make a custom image registration pipeline, that uses only 2 degrees of freedom, so just x,y translation. OpenCV did not have anything that did this, but a python library called Kornia does this well. https://kornia-tutorials.readthedocs.io/en/latest/image_registration.html https://kornia-tutorials.readthedocs.io/en/latest/image_regi...
    - hikarudo 5y ago
      The opencv_contrib repo does have a module, called "reg", for direct alignment.
    - zalo 5y ago
      Actually base-OpenCV has a great function for this: `cv2.findTransformECC()`: https://learnopencv.com/image-alignment-ecc-in-opencv-c-python/ https://learnopencv.com/image-alignment-ecc-in-opencv-c-pyth... It can do dense translation, translation + rotation, Affine, and Homography alignment; I've used it in the past to do sub-pixel Aruco/AprilTag alignment (and I'd probably also use it for astrophotography).
rgarrett88 5y ago
I use a similar algorithm at work to detect if pages match a template and then align the image to OCR from mapped fields. Not perfect but pretty effective.
orena 5y ago
Turboreg is also good: Paper: http://bigwww.epfl.ch/publications/thevenaz9801.html http://bigwww.epfl.ch/publications/thevenaz9801.html Python implementation: https://pypi.org/project/pystackreg/ https://pypi.org/project/pystackreg/
- amelius 5y ago
  Would this work with images which are out of focus?
  - orena 5y ago
    Probably yes, it works on 2p Neuro images, which are of course not InFocus
account42 5y ago
Note that this is exactly what panorama software like Hugin [0] does - it even comes with a convenient command-line tool for this simple use case (aligning a stack of images that are mostly overlapping): https://wiki.panotools.org/Align_image_stack https://wiki.panotools.org/Align_image_stack [0] http://hugin.sourceforge.net/ http://hugin.sourceforge.net/
auraham 5y ago
This is the kind of post I love to find in HN: an interesting topic and code. Does anyone know what is the theme of the blog? I am interested in how the references are created. I suppose it is based on something like RST.

166 points by magamig 5y ago

Affine transformations are very useful for algorithmic image rotation/translation/reflection [0]. [0] https://en.wikipedia.org/wiki/Affine_transformation https://en.wikipedia.org/wiki/Affine_transformation

vanderZwan 5y ago

Image alignment allows for some fun image manipulations like these. I think one of the coolest novel applications of image alignment in recent years must be multi-frame super-resolution. For example, the Pixel phones use it to improve low-light and zoomed in photos[0]. [0] https://sites.google.com/view/handheld-super-res/ https://sites.google.com/view/handheld-super-res/

magamig 5y ago
Cool! I'm planning on using image alignment for enhancing the spatial resolution of hyperspectral (HS) images, through the fusion of RGB and HS images.
- is_true 5y ago
  You get something similar to a panchromatic image?
  - magamig 5y ago
    You can use an high-resolution panchromatic image to increase the spatial resolution of an RGB image. Here the objective is to do the apply the same idea but using an RGB and HS image, increasing the spatial resolution of the latter.
gliptic 5y ago
This is called drizzle (with dithering, e.g. the raw photos are taken with offsets) in the astrophotography sphere and is very effective when the images are undersampled. Although, astronomy images have the advantage of being filled with stars that are relatively easy to align very accurately.
- vanderZwan 5y ago
  I imagine the minuscule vibrations of a shutter make it a natural fit to the already existing image stacking pipeline, correct?
  - gliptic 5y ago
    I'm not sure what you mean. Mechanical shutters have no upsides I'm aware of :). Astro cameras use electronic shutters. EDIT: The dithering, or offsetting, is done by moving the telescope slightly between exposures. During exposures you want it to track the object you're photographing as accurately as possible. I imagine vibrations are a detriment even in the Pixel phone processing. It would be better if the phone could move instantly in slightly different directions and take steady raw photos in each.
    - vanderZwan 5y ago
      Ah, I was thinking of amateur astrophotography - although these days you can just set the mechanical shutter in many consumer DSLRs to stay open too I think.
      - galangalalgol 5y ago
        I think maybe you were considering the vibrations a source of noise for compressed sensing? For CS to work well you generally want to know the convolved noise. So a super accurate accelerometer might be able to measure random vibrations and do even better when they are present than absent. Not sure if phone accelerometers are that good.
  - xhkkffbf 5y ago
amelius 5y ago
Does this mean that objects that are on a trajectory out of the view have less resolution?
legulere 5y ago
I wonder if you could use alignment of multiple samples for better vectorizing of letters of old books. You could create very nice vector only pdfs.
- Syzygies 5y ago
  Yes! For a two-fer, take a pair of cell phone photos of the open book, compute a 3D model, and flatten it as if the book had been sliced and scanned. Now match up all the letter E's in the entire book for image enhancement. Vectorization is nontrivial. Most academic journal articles are now poorly scanned, and it would be nice to simply improve the scans.

shrx 5y ago

The biggest issue in real life use cases is lens distortion - these alignment algorithms usually don't correct barrel, pin-cushion or more complex lens distortions, making the alignment imperfect.

nullc 5y ago
Any of the standard panorama software can solve for a standard simple lens distortion model, it just needs a few more alignment points to do so.
zwieback 5y ago
OpenCV, which is what the OP is using, has higher order camera calibration. There are functions to calibrate based on checkerboards or dot grids and undistort images using those calibrations. Sometimes it's better to match first, then undistort, though.

thebigman433 5y ago

I currently working on an optical respiratory monitor and one of the challenges was image alignment/registration for an optical and a thermal image with wildly different resolutions, fovs. We ended up doing this by manually calibrating the images in matlab using a custom script and logging out a transformation matrix that we could then multiply the optical image by to get matching pixels in the thermal image. Really fun project, but definitely a tedious thing to do, especially when its only a small part of the overall project. The project in the link also looks very cool, just not quite right for us as we probably wouldnt consistently have enough landmarks to map.

versatran01 5y ago
Direct methods could be useful in your case. https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf
- SCUSKU 5y ago
  At work I had to make a custom image registration pipeline, that uses only 2 degrees of freedom, so just x,y translation. OpenCV did not have anything that did this, but a python library called Kornia does this well. https://kornia-tutorials.readthedocs.io/en/latest/image_registration.html https://kornia-tutorials.readthedocs.io/en/latest/image_regi...
  - hikarudo 5y ago
    The opencv_contrib repo does have a module, called "reg", for direct alignment.
  - zalo 5y ago
    Actually base-OpenCV has a great function for this: `cv2.findTransformECC()`: https://learnopencv.com/image-alignment-ecc-in-opencv-c-python/ https://learnopencv.com/image-alignment-ecc-in-opencv-c-pyth... It can do dense translation, translation + rotation, Affine, and Homography alignment; I've used it in the past to do sub-pixel Aruco/AprilTag alignment (and I'd probably also use it for astrophotography).

rgarrett88 5y ago

I use a similar algorithm at work to detect if pages match a template and then align the image to OCR from mapped fields. Not perfect but pretty effective.

orena 5y ago

Turboreg is also good: Paper: http://bigwww.epfl.ch/publications/thevenaz9801.html http://bigwww.epfl.ch/publications/thevenaz9801.html Python implementation: https://pypi.org/project/pystackreg/ https://pypi.org/project/pystackreg/

amelius 5y ago
Would this work with images which are out of focus?
- orena 5y ago
  Probably yes, it works on 2p Neuro images, which are of course not InFocus

account42 5y ago

Note that this is exactly what panorama software like Hugin [0] does - it even comes with a convenient command-line tool for this simple use case (aligning a stack of images that are mostly overlapping): https://wiki.panotools.org/Align_image_stack https://wiki.panotools.org/Align_image_stack [0] http://hugin.sourceforge.net/ http://hugin.sourceforge.net/

auraham 5y ago

This is the kind of post I love to find in HN: an interesting topic and code. Does anyone know what is the theme of the blog? I am interested in how the references are created. I suppose it is based on something like RST.