3 ms·

I currently working on an optical respiratory monitor and one of the challenges was image alignment/registration for an optical and a thermal image with wildly

I currently working on an optical respiratory monitor and one of the challenges was image alignment/registration for an optical and a thermal image with wildly different resolutions, fovs. We ended up doing this by manually calibrating the images in matlab using a custom script and logging out a transformation matrix that we could then multiply the optical image by to get matching pixels in the thermal image. Really fun project, but definitely a tedious thing to do, especially when its only a small part of the overall project. The project in the link also looks very cool, just not quite right for us as we probably wouldnt consistently have enough landmarks to map.

versatran01 5y ago
Direct methods could be useful in your case. https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf
- SCUSKU 5y ago
  At work I had to make a custom image registration pipeline, that uses only 2 degrees of freedom, so just x,y translation. OpenCV did not have anything that did this, but a python library called Kornia does this well. https://kornia-tutorials.readthedocs.io/en/latest/image_registration.html https://kornia-tutorials.readthedocs.io/en/latest/image_regi...
  - hikarudo 5y ago
    The opencv_contrib repo does have a module, called "reg", for direct alignment.
  - zalo 5y ago
    Actually base-OpenCV has a great function for this: `cv2.findTransformECC()`: https://learnopencv.com/image-alignment-ecc-in-opencv-c-python/ https://learnopencv.com/image-alignment-ecc-in-opencv-c-pyth... It can do dense translation, translation + rotation, Affine, and Homography alignment; I've used it in the past to do sub-pixel Aruco/AprilTag alignment (and I'd probably also use it for astrophotography).
magamig 5y ago
Maybe you could use this for when there are "enough" features, and keep using the manual method for the remaining situations. I'd have to take a look at the images to have a better idea of what you are dealing with.
- thebigman433 5y ago
  The main issue is that the thermal camera we're using is only a few hundred pixels tall/wide, so using it for feature detection was rough when we did try that for something. For our purposes, with cameras on a solid mount, the hardcoded transformation matrix is good enough, but I think if you had better cameras and more time/more compute power, something like this could be better. A bigger issue is that the project is running on a Pi, so just getting the image alignment running along with our face detection wouldve also been a pretty tall task if we want to stay at around ~5fps.

versatran01 5y ago

Direct methods could be useful in your case. https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf https://pages.cs.wisc.edu/~dyer/ai-qual/irani-visalg00.pdf

SCUSKU 5y ago
At work I had to make a custom image registration pipeline, that uses only 2 degrees of freedom, so just x,y translation. OpenCV did not have anything that did this, but a python library called Kornia does this well. https://kornia-tutorials.readthedocs.io/en/latest/image_registration.html https://kornia-tutorials.readthedocs.io/en/latest/image_regi...
- hikarudo 5y ago
  The opencv_contrib repo does have a module, called "reg", for direct alignment.
- zalo 5y ago
  Actually base-OpenCV has a great function for this: `cv2.findTransformECC()`: https://learnopencv.com/image-alignment-ecc-in-opencv-c-python/ https://learnopencv.com/image-alignment-ecc-in-opencv-c-pyth... It can do dense translation, translation + rotation, Affine, and Homography alignment; I've used it in the past to do sub-pixel Aruco/AprilTag alignment (and I'd probably also use it for astrophotography).

magamig 5y ago

Maybe you could use this for when there are "enough" features, and keep using the manual method for the remaining situations. I'd have to take a look at the images to have a better idea of what you are dealing with.

thebigman433 5y ago
The main issue is that the thermal camera we're using is only a few hundred pixels tall/wide, so using it for feature detection was rough when we did try that for something. For our purposes, with cameras on a solid mount, the hardcoded transformation matrix is good enough, but I think if you had better cameras and more time/more compute power, something like this could be better. A bigger issue is that the project is running on a Pi, so just getting the image alignment running along with our face detection wouldve also been a pretty tall task if we want to stay at around ~5fps.