8 ms·

Calibrating cameras is still important; it's only "mostly irrelevant in 2024" if you don't care about accuracy. Incidentally, tools like opencv and their ilk ar

by dima55 2y ago

Calibrating cameras is still important; it's only "mostly irrelevant in 2024" if you don't care about accuracy. Incidentally, tools like opencv and their ilk are also what you use if you don't care about accuracy. Modern tools like mrcal (https://mrcal.secretsauce.net/tour.html https://mrcal.secretsauce.net/tour.html) are essential if you're trying to do long-range stereo or use wide lenses or have calibration instability or any number of other ever-present issues.

deleted 2y ago
[deleted]
seventytwo 2y ago
What makes this better than OpenCV?
- dima55 2y ago
  Uncertainty propagation. Richer models that fit better. Lots of feedback and metrics and visualization to evaluate the quality of the solve. Flexibility of the tool. Documentation.
  - midjji 2y ago
    Richer models that fit better are often a trap, in particular for beginners. Use the simplest model you can get away with. Unless you know what you are doing, most people are better of with a simpler model as it will be more robust to observability issues. If you dont know what those are, use a simpler model. Uncertainty propagation is very difficult to use for vision, and largely just modelling errors in vision as the error distributions, e.g. for anything observed or reconstructed from images are either subpixel accurate, or too non linear.
    - dima55 2y ago
      Avoiding rich models is a great thing to do if you don't model uncertainty: a beginner that didn't get enough useful calibration data will see poor uncertainties in the results. So I now use the splined models in pretty much all applications, and there are very few downsides. In my experience, every lens fits noticeably better with the richer model (the mrcal validation shows you this explicitly). I think you should look at the tour of mrcal; it's friendly.
  - amelius 2y ago
    A problem I always run into with OpenCV is that I need to preprocess the checkerboard images such that the lighting is just right. This is odd because that's the sort of thing I'd expect a computer vision library to excel at. Another problem I run into is that the transforms are ill-defined just outside of the screen. That means that if I want to draw e.g. a line in world coordinates onto the image from a camera, then I often get garbage if the line starts or ends outside the image (even if I divide the line into many segments).
    - midjji 2y ago
      The corner detection is bad. Not sure why. I also think its gotten worse since 3.5. I know the corner detection refinement is worse than the raw detections, so turn that shit off. Yeah thats a common problem if the calibration failed. It could also be that you are not cropping to what is in front of the camera, but if its really weird, its most likely the former. So the default, and probably most of the camera models in opencv requires a monotonic change lenses and bijective imaging. The former is common unless the lense has defects on the surface, and the latter is practically a physical constraint. The problem is that these constraints are difficult to add to the estimator, so they didnt. Meaning it will find a solution where they are not satisfied. If say the bijectiveness is not satisfied a bit outside the image, but stil valid accounting for infront and float accuracy, then that would absolutely account for the problem you describe. Is pretty obvious if you consider the function what the problem is, just hard to add the constraint in opencvs estimator. The solution is 1, verify the result is satisfied after estimation, 2 make sure you make the parameters are as observable as possible during calibration. This means spread out in the image, evenly distributed, and all the way to the edges. Also make verify it has not rotated one or more of the detected chessboards upside down, or 90 deg sideways. Finally, because it becomes harder and harder to avoid this problem with more parameters, always start with 1, then try 2, then the two variations of 3, and so on. More parameters always fit better, so use an appropriate test.
    - midjji 2y ago
      Also always do calibration in good light conditions.
    - backes 2y ago
why_only_15 2y ago
what is "mostly irrelevant in 2024" quoting from?
- usui 2y ago
  I was wondering this too, haha. Does someone believe this, and if so, who are you quoting?
- ryandamm 2y ago
  There is a series of comments further down in the discussion thread.
midjji 2y ago
Opencv is fine for basic models like the most popular one. This one is also sufficient in most cases, I have used it for subpixel 400m stereo with a 40cm base, visual odometry, high end 3d reconstruction etc, it also has fully functional variants for wide and fisheye. You just have to select the right model, calibrate without observability issues, verify the calibration, understand the camera, etc. The better tools help a little bit, but they do not eliminate the need to understand what you are doing in the slightest. The better tools are easier to use, but generally not more accurate. The spline field variants are more accurate if you know what you are doing and the right conditions apply, but much less robust if you dont. Opencv calibration is hard to get right if you dont know what you are doing, and at first glance it will look right despite being irrecoverably bad. I had to edit the docs and tutorials because the examples they had were of FAILED calibrations. Visualizing the resulting distortion field is critical to understand if the calibration succeeded, and that s what the better tools provide. That said, if you dont know you are looking at, they likely dont help, and the golden rule is if someone calibrated something and said it was easy, they didnt. If someone has a pdh in camera calibration said they calibratied them but that they havent used them for sfm but they plan to soon, odds are 20% its right.
- jvanderbot 2y ago
  Mrcal was written and verified for more difficult stereo tracking than that, approximately by a 10-15x factor (in, say, fisher information sense). It's a shame it doesn't have a better data set for verification but I expect there are reasons for that as well. Not to say that other tools are completely inappropriate for this domain, obviously, or denigrate anything or anyone.
  - midjji 2y ago
    Sounds like a paper, link?
  - midjji 2y ago
    For you to say that you would have needed the image res 2MPixel, but ignoring that, it should have been a paper I knew about if true. So I checked. mrcal is providing the same lens models as opencv, and it looks like its using opencv to do the estimate. For higher perf calibration checkout https://arxiv.org/abs/1912.02908 https://arxiv.org/abs/1912.02908
jvanderbot 2y ago
For those curious, keep in mind that the use case for open cv has been almost exclusively indoor, close range, or otherwise loose constraints. Graph models for localization famously make use of quite primitive features but tons of observations over time to produce excellent estimates and loop closures. There are applications that require pulling almost magical levels of signal from very small numbers of pixels, and for those extremely accurate results you really need to drill down on the camera model in particular. Even modelling the environment or atmosphere to get accurate results. For those types of applications opencv is absolutely not the right choice. It just so happens that dima has an accessible software kit that is more for the latter case. But all the better because it's perfectly fine to use this tool to calibrate any camera and then go use it with any other CV pipeline. His pain your gain.
- midjji 2y ago
  Nonsense. Opencv has been used widely for outdoor localisation too, in particular since it is much easier to do outdoor than indoor. Such algorithms always create a graph over the images, but if you mean the graphslam graph filter methods, those are substantially subpar compared to classic feature based methods as well as the more modern, dense and semidense methods.
  - jvanderbot 2y ago
    OK I was imprecise in my comments. let me try again. For localization work (indoor or outdoor), in which you mostly want to close loops and track 3D pose of features using measurements from, say 100s of m or less, (or perhaps scene recognition using effectively flat "very far" images), calibration using opencv is probably highly performant. It's standard, and I've certainly seen much success using it before feeding into regular gtsam etc. There are some unspoken assumptions about localization that don't translate to, say, tracking necessarily. Generally they are: Many observations, close-ish range (relative to stereo offset), mostly-correctly-pointed camera rigs (e.g., forward on a car), or perhaps assumptions about density of features, correlation across images, existence of dense prior maps, etc. I believe the use case for something like mrcal is to improve calibration for cameras and applications that don't fit this model well. In particular, you may need to track a target at extreme range, using a pixel or two in the corner of your image, with a particularly wide FOV. These specific use cases, mentioned in top level comment, do require additional care, especially in calibration. Thus mrcal. It just so happens that out of a domain where the very particular details of calibration matter, you may find a tool that helps with all calibration and I think that's the point in bringing up mrcal on a thread discussing calibration in general. I think that's as precise and non-controversial as I can be.

by dima55 2y ago

deleted 2y ago

[deleted]

seventytwo 2y ago

What makes this better than OpenCV?

dima55 2y ago
Uncertainty propagation. Richer models that fit better. Lots of feedback and metrics and visualization to evaluate the quality of the solve. Flexibility of the tool. Documentation.
- midjji 2y ago
  Richer models that fit better are often a trap, in particular for beginners. Use the simplest model you can get away with. Unless you know what you are doing, most people are better of with a simpler model as it will be more robust to observability issues. If you dont know what those are, use a simpler model. Uncertainty propagation is very difficult to use for vision, and largely just modelling errors in vision as the error distributions, e.g. for anything observed or reconstructed from images are either subpixel accurate, or too non linear.
  - dima55 2y ago
    Avoiding rich models is a great thing to do if you don't model uncertainty: a beginner that didn't get enough useful calibration data will see poor uncertainties in the results. So I now use the splined models in pretty much all applications, and there are very few downsides. In my experience, every lens fits noticeably better with the richer model (the mrcal validation shows you this explicitly). I think you should look at the tour of mrcal; it's friendly.
- amelius 2y ago
  A problem I always run into with OpenCV is that I need to preprocess the checkerboard images such that the lighting is just right. This is odd because that's the sort of thing I'd expect a computer vision library to excel at. Another problem I run into is that the transforms are ill-defined just outside of the screen. That means that if I want to draw e.g. a line in world coordinates onto the image from a camera, then I often get garbage if the line starts or ends outside the image (even if I divide the line into many segments).
  - midjji 2y ago
    The corner detection is bad. Not sure why. I also think its gotten worse since 3.5. I know the corner detection refinement is worse than the raw detections, so turn that shit off. Yeah thats a common problem if the calibration failed. It could also be that you are not cropping to what is in front of the camera, but if its really weird, its most likely the former. So the default, and probably most of the camera models in opencv requires a monotonic change lenses and bijective imaging. The former is common unless the lense has defects on the surface, and the latter is practically a physical constraint. The problem is that these constraints are difficult to add to the estimator, so they didnt. Meaning it will find a solution where they are not satisfied. If say the bijectiveness is not satisfied a bit outside the image, but stil valid accounting for infront and float accuracy, then that would absolutely account for the problem you describe. Is pretty obvious if you consider the function what the problem is, just hard to add the constraint in opencvs estimator. The solution is 1, verify the result is satisfied after estimation, 2 make sure you make the parameters are as observable as possible during calibration. This means spread out in the image, evenly distributed, and all the way to the edges. Also make verify it has not rotated one or more of the detected chessboards upside down, or 90 deg sideways. Finally, because it becomes harder and harder to avoid this problem with more parameters, always start with 1, then try 2, then the two variations of 3, and so on. More parameters always fit better, so use an appropriate test.
  - midjji 2y ago
    Also always do calibration in good light conditions.
  - backes 2y ago

why_only_15 2y ago

what is "mostly irrelevant in 2024" quoting from?

usui 2y ago
I was wondering this too, haha. Does someone believe this, and if so, who are you quoting?
ryandamm 2y ago
There is a series of comments further down in the discussion thread.

midjji 2y ago

Opencv is fine for basic models like the most popular one. This one is also sufficient in most cases, I have used it for subpixel 400m stereo with a 40cm base, visual odometry, high end 3d reconstruction etc, it also has fully functional variants for wide and fisheye. You just have to select the right model, calibrate without observability issues, verify the calibration, understand the camera, etc. The better tools help a little bit, but they do not eliminate the need to understand what you are doing in the slightest. The better tools are easier to use, but generally not more accurate. The spline field variants are more accurate if you know what you are doing and the right conditions apply, but much less robust if you dont. Opencv calibration is hard to get right if you dont know what you are doing, and at first glance it will look right despite being irrecoverably bad. I had to edit the docs and tutorials because the examples they had were of FAILED calibrations. Visualizing the resulting distortion field is critical to understand if the calibration succeeded, and that s what the better tools provide. That said, if you dont know you are looking at, they likely dont help, and the golden rule is if someone calibrated something and said it was easy, they didnt. If someone has a pdh in camera calibration said they calibratied them but that they havent used them for sfm but they plan to soon, odds are 20% its right.

jvanderbot 2y ago
Mrcal was written and verified for more difficult stereo tracking than that, approximately by a 10-15x factor (in, say, fisher information sense). It's a shame it doesn't have a better data set for verification but I expect there are reasons for that as well. Not to say that other tools are completely inappropriate for this domain, obviously, or denigrate anything or anyone.
- midjji 2y ago
  Sounds like a paper, link?
- midjji 2y ago
  For you to say that you would have needed the image res 2MPixel, but ignoring that, it should have been a paper I knew about if true. So I checked. mrcal is providing the same lens models as opencv, and it looks like its using opencv to do the estimate. For higher perf calibration checkout https://arxiv.org/abs/1912.02908 https://arxiv.org/abs/1912.02908

jvanderbot 2y ago

For those curious, keep in mind that the use case for open cv has been almost exclusively indoor, close range, or otherwise loose constraints. Graph models for localization famously make use of quite primitive features but tons of observations over time to produce excellent estimates and loop closures. There are applications that require pulling almost magical levels of signal from very small numbers of pixels, and for those extremely accurate results you really need to drill down on the camera model in particular. Even modelling the environment or atmosphere to get accurate results. For those types of applications opencv is absolutely not the right choice. It just so happens that dima has an accessible software kit that is more for the latter case. But all the better because it's perfectly fine to use this tool to calibrate any camera and then go use it with any other CV pipeline. His pain your gain.

midjji 2y ago
Nonsense. Opencv has been used widely for outdoor localisation too, in particular since it is much easier to do outdoor than indoor. Such algorithms always create a graph over the images, but if you mean the graphslam graph filter methods, those are substantially subpar compared to classic feature based methods as well as the more modern, dense and semidense methods.
- jvanderbot 2y ago
  OK I was imprecise in my comments. let me try again. For localization work (indoor or outdoor), in which you mostly want to close loops and track 3D pose of features using measurements from, say 100s of m or less, (or perhaps scene recognition using effectively flat "very far" images), calibration using opencv is probably highly performant. It's standard, and I've certainly seen much success using it before feeding into regular gtsam etc. There are some unspoken assumptions about localization that don't translate to, say, tracking necessarily. Generally they are: Many observations, close-ish range (relative to stereo offset), mostly-correctly-pointed camera rigs (e.g., forward on a car), or perhaps assumptions about density of features, correlation across images, existence of dense prior maps, etc. I believe the use case for something like mrcal is to improve calibration for cameras and applications that don't fit this model well. In particular, you may need to track a target at extreme range, using a pixel or two in the corner of your image, with a particularly wide FOV. These specific use cases, mentioned in top level comment, do require additional care, especially in calibration. Thus mrcal. It just so happens that out of a domain where the very particular details of calibration matter, you may find a tool that helps with all calibration and I think that's the point in bringing up mrcal on a thread discussing calibration in general. I think that's as precise and non-controversial as I can be.