3 ms·

I worked on this project. The images were indeed captured by a 3D microscope in about 9000 separate captures with non-uniform illumination, so this isn't so mu

I worked on this project. The images were indeed captured by a 3D microscope in about 9000 separate captures with non-uniform illumination, so this isn't so much a stitching artifact as it is an illumination artifact. The specular component of the varnish and the different illumination angles of the light sources make it almost impossible to capture a uniformly-colored field without the use of polarization filters, and the best stitching algorithms don't do well with different opinions from multiple images about the color of a pixel (they do fine with different opinions about brightness).

thomas43 6y ago
have you tried to extract the vignetting pattern form the captures and using it to normalize them? My first try would be to calculate the median grayscale image of all 9000 captures and then using this to normalize the intensity.
- erdmann 6y ago
  Yes, and the vignetting pattern isn't so much the problem. It's a color distortion problem (spatially varying chromaticity rather than brightness that is not just a function of the position within the field of view but also of the underlying material, unfortunately). So there isn't a nice way to correct each capture in a predictable way to ensure that overlapping pixels have the same colors consistently.
  - thomas43 6y ago
    i see. interesting problem
    - triplesec 6y ago
      Anyone have a nice primer to read to help nonspecialists understand what you are talking about in this thread here?
      - chaboud 6y ago
        They are talking about a bidirectional reflectance distribution function, a now common modeling technique for practical representation of complex surface properties when illuminated. That said, a full BRDF is not the only way to approach this problem, especially at reduced resolution where the artifacts are more apparent. If you're an ACM member, there is a wealth of information in SIGGRAPH publications. Having been in graphics since the 90s, I started with Foley and Van Dam (Computer Graphics: Principles and Practice) and Watt and Watt (Animation and Rendering Techniques) and then stayed on top of SIGRAPH (attending regularly, though not annually) since then. For a more whimsical survey from the pen of a straight up genius, Jim Blinn's books (e.g. Dirty Pixels) are fantastic reads.
  - snovv_crash 6y ago
    I've worked with similar problems for agricultural mapping from drone images. You would need to build a BRDF model for the different colour channels for various types of materials, then assign the material based on a combination of best representative models. Then you can re-render with uniform normal lighting.
jahlove 6y ago
Would it be hard to have uniform illumination?
- erdmann 6y ago
  Yes, because the surface isn't flat. At these magnifications, any given field of view may be tilted toward one of the light sources, changing the relative contribution of the specular reflection off of the varnish as well as the reflected color from the paint surface. The apparatus doesn't have the ability to tilt to maintain a constant angle to the surface; it can only pan in the x-y plane and do focus stacking in the z-direction. Additionally, the left and right light sources were hand-positioned and there is no way to calibrate their exact geometry and relative brightness and color.
  - jahlove 6y ago
    thanks for explaining!
nr2x 6y ago
Fantastic work, congratulations. Hope to see more of these.
ezconnect 6y ago
Was the 3D effect made from different focus points?
- tomerico 6y ago
  Very likely. The focus stacking software (such as Helicon) can output the 3d object in addition to the stacked image
- erdmann 6y ago
  The very limited depth of field of a microscope necessitates doing depth stacking for every field of view. This is done automatically by the apparatus. As a side effect, it gives a heightmap, but with the lens used for the whole-painting scan (what HIROX calls "35x", corresponding to ~5 µm sampling resolution), the elevations are not reliable, especially near the edge of the field. For selected areas, a higher magnification was used that gives much more reliable elevations. Sadly, only a small fraction of the painting was imaged using this higher resolution, so the 3D data is spotty.
chaboud 6y ago
For purposes of browsing, you can source your top-level "zoomed out" layer from a single photograph and blend at lower tiles in the quad tree or normalize the downsampled capture data against a reference. Nonetheless, thanks for doing this. It's an amazing piece of work.

thomas43 6y ago

have you tried to extract the vignetting pattern form the captures and using it to normalize them? My first try would be to calculate the median grayscale image of all 9000 captures and then using this to normalize the intensity.

erdmann 6y ago
Yes, and the vignetting pattern isn't so much the problem. It's a color distortion problem (spatially varying chromaticity rather than brightness that is not just a function of the position within the field of view but also of the underlying material, unfortunately). So there isn't a nice way to correct each capture in a predictable way to ensure that overlapping pixels have the same colors consistently.
- thomas43 6y ago
  i see. interesting problem
  - triplesec 6y ago
    Anyone have a nice primer to read to help nonspecialists understand what you are talking about in this thread here?
    - chaboud 6y ago
      They are talking about a bidirectional reflectance distribution function, a now common modeling technique for practical representation of complex surface properties when illuminated. That said, a full BRDF is not the only way to approach this problem, especially at reduced resolution where the artifacts are more apparent. If you're an ACM member, there is a wealth of information in SIGGRAPH publications. Having been in graphics since the 90s, I started with Foley and Van Dam (Computer Graphics: Principles and Practice) and Watt and Watt (Animation and Rendering Techniques) and then stayed on top of SIGRAPH (attending regularly, though not annually) since then. For a more whimsical survey from the pen of a straight up genius, Jim Blinn's books (e.g. Dirty Pixels) are fantastic reads.
- snovv_crash 6y ago
  I've worked with similar problems for agricultural mapping from drone images. You would need to build a BRDF model for the different colour channels for various types of materials, then assign the material based on a combination of best representative models. Then you can re-render with uniform normal lighting.

jahlove 6y ago

Would it be hard to have uniform illumination?

erdmann 6y ago
Yes, because the surface isn't flat. At these magnifications, any given field of view may be tilted toward one of the light sources, changing the relative contribution of the specular reflection off of the varnish as well as the reflected color from the paint surface. The apparatus doesn't have the ability to tilt to maintain a constant angle to the surface; it can only pan in the x-y plane and do focus stacking in the z-direction. Additionally, the left and right light sources were hand-positioned and there is no way to calibrate their exact geometry and relative brightness and color.
- jahlove 6y ago
  thanks for explaining!

nr2x 6y ago

Fantastic work, congratulations. Hope to see more of these.

ezconnect 6y ago

Was the 3D effect made from different focus points?

tomerico 6y ago
Very likely. The focus stacking software (such as Helicon) can output the 3d object in addition to the stacked image
erdmann 6y ago
The very limited depth of field of a microscope necessitates doing depth stacking for every field of view. This is done automatically by the apparatus. As a side effect, it gives a heightmap, but with the lens used for the whole-painting scan (what HIROX calls "35x", corresponding to ~5 µm sampling resolution), the elevations are not reliable, especially near the edge of the field. For selected areas, a higher magnification was used that gives much more reliable elevations. Sadly, only a small fraction of the painting was imaged using this higher resolution, so the 3D data is spotty.

chaboud 6y ago

For purposes of browsing, you can source your top-level "zoomed out" layer from a single photograph and blend at lower tiles in the quad tree or normalize the downsampled capture data against a reference. Nonetheless, thanks for doing this. It's an amazing piece of work.