5 ms·

> Each pixel must be labelled with metadata, such as illumination level, its 3D position, the scale, the sample type and how the sample was prepared. Each pixe

> Each pixel must be labelled with metadata, such as illumination level, its 3D position, the scale, the sample type and how the sample was prepared. Each pixel? Why? All of those except the 3D position apply to all the pixels in a given image, and the (2D) position of a pixel can be inferred from its location in the image. Wait - are there optical microscopes that can create 3D images? I know you can see a 3D image if you peer into a binocular microscope, but AFAIK cameras for those things are always 2D cameras.

ahns 3y ago
As far as I know, not technically (although I haven't kept up in the area), but you can definitely sweep through a volumetric sample; there are microscopes that can for example illuminate a thin z-plane of a transparent sample and collect the image or those that can reject out-of-focus (off-z) light for a particular z-plane, then move to another z-plane, etc. and then generate a volume on the software side.
- saltcured 3y ago
  Yes, I think you are talking about light-sheet microscopy. This tries to illuminate a thin layer and then images with a 2D sensor like in a normal digital camera. This is often a monochrome sensor, and extra metadata records what kind of optical filters were in place. A multi-channel image would then have separate planes captured with different optical band-pass filters. You need metadata to know how a set of planes relate to each other, whether shifting through space or changing filters or just measuring the same configuration again at different time points. There are also confocal "laser-scanning" microscopes which effectively illuminate a single point in space and image with what is effectively a single monochrome pixel sensor. In this case, the raw signal is pretty much a time series like you might imagine with digital audio. You need metadata to tell you how the optics were being shifted around, to interpret each sample as representing a point in space at a particular time and with what filters. Normally, the first round of interpretation is done during the initial file save, packing measurements into a file format that still expects planar, rectangular pixel arrays with a regular grid spacing for the neighboring pixels. You pretend the whole image plane was captured at a time point, but it was really measured sequentially somewhat like a cathode-ray tube displays an image one pixel at a time. A session might produce many such planes, and metadata is still needed to understand whether these planes represent the same plane observed over time and/or parallel planes spanning a volume and/or different channels of the same plane. On the other end of the spectrum, there are slide scanners which image a much larger stage area by shifting around a 2D camera and taking many overlapping images. So a 2048x2048 sensor might be used to produce a 100,000 x 60,000 pixel image plane. Each sub-image would record its position within the stage area and these do not necessarily line up at precise multiples of image pixels. There might also be gaps where the scanner first detects the overall shape of specimen(s) on the slide and then plans a set of images to cover the specimen while skipping over the background area. This sparsity saves time for imaging as well as storage and transfer times by skipping empty areas.
dexwiz 3y ago
There is a lot of specialized microscopes out there. Confocal is pretty widespread. https://en.wikipedia.org/wiki/Confocal_microscopy https://en.wikipedia.org/wiki/Confocal_microscopy
tomnicholas 3y ago
> Each pixel? Why? I use the Zarr format (for climate science data rather than microscope data), and I think this is just poor wording in the article. In the Zarr specification the metadata is stored separately from the actual chunks of compressed array data. So the metadata applies at the array level, not the pixel level. > Wait - are there optical microscopes that can create 3D images? I think so - they do it by scanning lots of images at different focal lengths to create a 3D section (I think?). There are whole projects just for visualizing the multi-TeraByte 3D image files produced - Napari is an open-source image viewer which opens OME-Zarr data.
- carreau 3y ago
  Even on classical 2D microscope the illumination can be non-uniform, and you might need to calibrate your image. Source: PhD in biolab with microscopes, and napari dev.
  - dekhn 3y ago
    OME-zarr doesn't really storage per-pixel illumination data. Instead, the illumination will typically be storaged as a per-2D-plane metadata. Flat fields, dark fields, and light fields can all be stored but would be their own arrays (structure of arrays rather than array of structures).
COGlory 3y ago
Probably not all these things need per-pixel metadata, but anisotropy exists and that means many of the variables you'd think are per-exposure are actually dependent on where in the exposure the individual pixel is. For instance, illumination level isn't uniform across a field of view for all cameras, and may need to be normalized.
- denton-scratch 3y ago
  > may need to be normalized. But that's post-processing; isn't the TFA's argument that it's hard for researchers to share raw data?
  - COGlory 3y ago
    Unless I'm mistaking the question, because the data need to be normalized later, you need per pixel intensity in the raw data.
dekhn 3y ago
They don't really "label every pixel" in the sense that I think about it. Instead, they have a collection of dense arrays representing the image data itself, then have metadata at the per-array, or overall level. A typical dataset I work with is multidimensional, it starts as: 1) 2D planes of multiple channel image intensities, typically 5K x 5K pixels, each covering just part of an overall field of view. These are like patches when you do panoramas- take 20 partly overlapping shots. Each plane contains multiple channels- that could be "red green and blue" or more complicated spectral distributions. 2) 3D information- the microscope takes photos at various depths, only the "in-focus" (within the volume of view) information. These can be stacked (like depth stacking) or turned into a 3D "volume". 3) Maybe the data was collected over multiple time points, so (1) and (2) repeat every hour. Other parameters- like temperature, etc, could also represent an entire dimension. 4) Every 2D plane has its own key-value metadata, such as "what color channels were used", "what objective was used" (magnification), and lots of other high-dimensional attributes (that's what they mean by "each pixel must be labelled with metadata"- the 3D position is the same for every pixel in a 2D plane. 5) Generally all of this is modelled as structures, arrays, and structures of arrays/arrays of structures. In the case of OME-zarr, it's modelled as an n-Dimensional array with dimensions expressed in a filesystem hierarchy (first dimension typicall the outermost directory, innermost dimension usually a flat file containing a block of scalar points using some compressed numpy storage. Then at each level of the directory you have additional .json files which contain attributes at that level of the data. Those partly overlapping 2D planes are often assembled into panoramas, which can be a lot more convenient to work with. There are various tools for working with this- I've used navigation map javascript, but napari is a desktop app wiht full suport for sectioned viewing of high-dimensional (7d) data. OME-zarr is nice because it sort of uses the same underlying tech that the machine learning folks use, and it's ostensibly "optimized for object storage", but I still have lots of complaints with the implementation details, but it's important for me not to distract the OME-zarr team from making the standard successful.

ahns 3y ago

As far as I know, not technically (although I haven't kept up in the area), but you can definitely sweep through a volumetric sample; there are microscopes that can for example illuminate a thin z-plane of a transparent sample and collect the image or those that can reject out-of-focus (off-z) light for a particular z-plane, then move to another z-plane, etc. and then generate a volume on the software side.

saltcured 3y ago
Yes, I think you are talking about light-sheet microscopy. This tries to illuminate a thin layer and then images with a 2D sensor like in a normal digital camera. This is often a monochrome sensor, and extra metadata records what kind of optical filters were in place. A multi-channel image would then have separate planes captured with different optical band-pass filters. You need metadata to know how a set of planes relate to each other, whether shifting through space or changing filters or just measuring the same configuration again at different time points. There are also confocal "laser-scanning" microscopes which effectively illuminate a single point in space and image with what is effectively a single monochrome pixel sensor. In this case, the raw signal is pretty much a time series like you might imagine with digital audio. You need metadata to tell you how the optics were being shifted around, to interpret each sample as representing a point in space at a particular time and with what filters. Normally, the first round of interpretation is done during the initial file save, packing measurements into a file format that still expects planar, rectangular pixel arrays with a regular grid spacing for the neighboring pixels. You pretend the whole image plane was captured at a time point, but it was really measured sequentially somewhat like a cathode-ray tube displays an image one pixel at a time. A session might produce many such planes, and metadata is still needed to understand whether these planes represent the same plane observed over time and/or parallel planes spanning a volume and/or different channels of the same plane. On the other end of the spectrum, there are slide scanners which image a much larger stage area by shifting around a 2D camera and taking many overlapping images. So a 2048x2048 sensor might be used to produce a 100,000 x 60,000 pixel image plane. Each sub-image would record its position within the stage area and these do not necessarily line up at precise multiples of image pixels. There might also be gaps where the scanner first detects the overall shape of specimen(s) on the slide and then plans a set of images to cover the specimen while skipping over the background area. This sparsity saves time for imaging as well as storage and transfer times by skipping empty areas.

dexwiz 3y ago

There is a lot of specialized microscopes out there. Confocal is pretty widespread. https://en.wikipedia.org/wiki/Confocal_microscopy https://en.wikipedia.org/wiki/Confocal_microscopy

tomnicholas 3y ago

> Each pixel? Why? I use the Zarr format (for climate science data rather than microscope data), and I think this is just poor wording in the article. In the Zarr specification the metadata is stored separately from the actual chunks of compressed array data. So the metadata applies at the array level, not the pixel level. > Wait - are there optical microscopes that can create 3D images? I think so - they do it by scanning lots of images at different focal lengths to create a 3D section (I think?). There are whole projects just for visualizing the multi-TeraByte 3D image files produced - Napari is an open-source image viewer which opens OME-Zarr data.

carreau 3y ago
Even on classical 2D microscope the illumination can be non-uniform, and you might need to calibrate your image. Source: PhD in biolab with microscopes, and napari dev.
- dekhn 3y ago
  OME-zarr doesn't really storage per-pixel illumination data. Instead, the illumination will typically be storaged as a per-2D-plane metadata. Flat fields, dark fields, and light fields can all be stored but would be their own arrays (structure of arrays rather than array of structures).

COGlory 3y ago

Probably not all these things need per-pixel metadata, but anisotropy exists and that means many of the variables you'd think are per-exposure are actually dependent on where in the exposure the individual pixel is. For instance, illumination level isn't uniform across a field of view for all cameras, and may need to be normalized.

denton-scratch 3y ago
> may need to be normalized. But that's post-processing; isn't the TFA's argument that it's hard for researchers to share raw data?
- COGlory 3y ago
  Unless I'm mistaking the question, because the data need to be normalized later, you need per pixel intensity in the raw data.

dekhn 3y ago

They don't really "label every pixel" in the sense that I think about it. Instead, they have a collection of dense arrays representing the image data itself, then have metadata at the per-array, or overall level. A typical dataset I work with is multidimensional, it starts as: 1) 2D planes of multiple channel image intensities, typically 5K x 5K pixels, each covering just part of an overall field of view. These are like patches when you do panoramas- take 20 partly overlapping shots. Each plane contains multiple channels- that could be "red green and blue" or more complicated spectral distributions. 2) 3D information- the microscope takes photos at various depths, only the "in-focus" (within the volume of view) information. These can be stacked (like depth stacking) or turned into a 3D "volume". 3) Maybe the data was collected over multiple time points, so (1) and (2) repeat every hour. Other parameters- like temperature, etc, could also represent an entire dimension. 4) Every 2D plane has its own key-value metadata, such as "what color channels were used", "what objective was used" (magnification), and lots of other high-dimensional attributes (that's what they mean by "each pixel must be labelled with metadata"- the 3D position is the same for every pixel in a 2D plane. 5) Generally all of this is modelled as structures, arrays, and structures of arrays/arrays of structures. In the case of OME-zarr, it's modelled as an n-Dimensional array with dimensions expressed in a filesystem hierarchy (first dimension typicall the outermost directory, innermost dimension usually a flat file containing a block of scalar points using some compressed numpy storage. Then at each level of the directory you have additional .json files which contain attributes at that level of the data. Those partly overlapping 2D planes are often assembled into panoramas, which can be a lot more convenient to work with. There are various tools for working with this- I've used navigation map javascript, but napari is a desktop app wiht full suport for sectioned viewing of high-dimensional (7d) data. OME-zarr is nice because it sort of uses the same underlying tech that the machine learning folks use, and it's ostensibly "optimized for object storage", but I still have lots of complaints with the implementation details, but it's important for me not to distract the OME-zarr team from making the standard successful.