3 ms·

Hey thanks, I tend to agree. Although if I read this right, OME-zarr punted on lossy compression (which is all that really matters) so IMHO it doesn't have much

Hey thanks, I tend to agree. Although if I read this right, OME-zarr punted on lossy compression (which is all that really matters) so IMHO it doesn't have much use in the real world yet even though people evidently put a lot of work into it: https://github.com/ome/ome-zarr-py https://github.com/ome/ome-zarr-py https://ngff.openmicroscopy.org/latest/ https://ngff.openmicroscopy.org/latest/ https://zarr.readthedocs.io/en/v2.1.0/api/codecs.html https://zarr.readthedocs.io/en/v2.1.0/api/codecs.html https://www.researchgate.net/publication/356622607_OME-NGFF_a_next-generation_file_format_for_expanding_bioimaging_data-access_strategies#read https://www.researchgate.net/publication/356622607_OME-NGFF_... https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1.full.pdf https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1.... i.munro (Ian Munro) August 15, 2023, 8:54am On the original point about size I suspect your answer is compression. I have found that a lot of image formats use lossy compression. I’m not sure about CZI specifically. AFAIK ome-zarr (sensibly IMO) uses only lossless compression. Best Ian and: sebi06 (Sebi06) August 15, 2023, 1:42pm Hi all, for CZI we offer currentyl two compression methods: JPG-XR ZSTD So we have lossless and lossy options depending on your needs. All of them are implemented in our C++, #NET or Python APIs and also supported from BioFormats. Looks like CZI is a competitor: https://www.zeiss.com/microscopy/en/products/software/zeiss-zen/czi-image-file-format.html https://www.zeiss.com/microscopy/en/products/software/zeiss-... https://github.com/cgohlke/czifile https://github.com/cgohlke/czifile http://www.physics.hmc.edu/~gerbode/wppriv/wp-content/uploads/2012/06/DS_ZISRAW-FileFormat_Rel_ZEN2011.pdf http://www.physics.hmc.edu/~gerbode/wppriv/wp-content/upload... CZI is basically JPEG compression and is probably missing the multidimensional compressor which is the secret sauce missing from the various formats. It also mentions licensing and legal so is maybe lost in the weeds and not open source. Without more info, it feels like this is still an open problem. My interest in this is for video games, since so many are multiple GB in size but are also missing this multidimensional compressor. If we had it, I'd guess a 10:1 to 100:1 reduction in size over what we have now, with no perceptual loss in quality.

rrock 3y ago
Lossy compression is a bad idea for scientific images. For instance, we often need to understand the statistics of photon detection events in background regions. That’s one of the first things to get tossed.
dekhn 3y ago
Please don't use lossy compression for scientific data. Disk space is cheap relative to people-time and data-collection-time. "Perceptual" is misleading- we're not just humans looking at jpegs. Many people are doing hard-code quantitative analysis for machine learning. better to downscale the images (2X, 4X, 8X or more) than to use lossy compression if you are truly low on disk budget.
- zackmorris 3y ago
  It sounds like there should be a standard for compression of scientific data, probably as a noise ceiling instead of a compression ratio, if the noise is normally distributed (a big if). But if I had to choose, I would side with you and vote to use lossless in order to avoid a whole class of errors that might come from lossy compression. Especially when we can get a 1 TB SSD for under $50. To play devil's advocate, I think there might be an opportunity for lossy compression of images below the Nyquist Rate of the microscope: https://svi.nl/Nyquist-Calculator https://svi.nl/Nyquist-Calculator I only bring this up because I worked with industrial cameras at a previous job and even the best lenses only got us to maybe 1/2 or 1/4 of the resolution that the sensors were capable of. Microscopes should probably include an effective resolution in each image file's metadata. If they don't, then we have no data on how noisy the image is, so this is all a moot point.
  - dekhn 3y ago
    I understand you're trying to help, but everything you are suggesting is basically not helpful. Microscopists are savvy enough to match sensors and lenses, it's typically called "diffraction limited" (https://en.wikipedia.org/wiki/Diffraction-limited_system https://en.wikipedia.org/wiki/Diffraction-limited_system) based on the Rayleigh criterion (https://en.wikipedia.org/wiki/Angular_resolution#The_Rayleigh_criterion https://en.wikipedia.org/wiki/Angular_resolution#The_Rayleig...). I rarely if ever hear microscopists refer to nyquist limits. They would not overimage something at a higher magnification, or more correctly, they would be aware it was happening and either downscale or not worry. Next, when you make a point like "we can get a 1TB SSD for under $50". It shows you're not understanding the domain microscopists work in. Their scopes typically cost $100K (with a camera for another $50K, and a stage that costs $10K). The cost of the storage system (or parking the data in S3) for 10 years is usually far less than the capital cost of the scope, or the operational expense of maintaining staff on site. I believe the effective resolution can already be computed from the captured metadata, which includes the objective's numerical aperture, as well as the frequency used for imaging (many microscopes today use lasers to image with a very specific frequency to cause fluorenscence, which itself has a fairly tight output spectrum).
thfuran 3y ago
>which is all that really matters Except for the domains where nobody uses it because it ruins the data.
- anthk 3y ago
  Imagine that in the Medicine World with DICOM images.
  - thfuran 3y ago
    DICOM technically supports lossy compression, but I think it's regarded as something of a historical mistake. It's pretty much never used and some viewers will plaster big warnings on lossily compressed series and others will outright refuse to load them.
_Microft 3y ago
> lossy compression (which is all that really matters) No, not in this case. This might be great for game files or whatever you are familiar with but lossy compression would basically mean to be mangling the data after going through all the effort of collecting it (I suspect you have only little experience with lab work?). When doing experiments, everything is documented in full detail. Settings on machines and equipment, preparation steps, version numbers of software used for processing, … You really don‘t want to lose information on your experiment.

rrock 3y ago

Lossy compression is a bad idea for scientific images. For instance, we often need to understand the statistics of photon detection events in background regions. That’s one of the first things to get tossed.

dekhn 3y ago

Please don't use lossy compression for scientific data. Disk space is cheap relative to people-time and data-collection-time. "Perceptual" is misleading- we're not just humans looking at jpegs. Many people are doing hard-code quantitative analysis for machine learning. better to downscale the images (2X, 4X, 8X or more) than to use lossy compression if you are truly low on disk budget.

zackmorris 3y ago
It sounds like there should be a standard for compression of scientific data, probably as a noise ceiling instead of a compression ratio, if the noise is normally distributed (a big if). But if I had to choose, I would side with you and vote to use lossless in order to avoid a whole class of errors that might come from lossy compression. Especially when we can get a 1 TB SSD for under $50. To play devil's advocate, I think there might be an opportunity for lossy compression of images below the Nyquist Rate of the microscope: https://svi.nl/Nyquist-Calculator https://svi.nl/Nyquist-Calculator I only bring this up because I worked with industrial cameras at a previous job and even the best lenses only got us to maybe 1/2 or 1/4 of the resolution that the sensors were capable of. Microscopes should probably include an effective resolution in each image file's metadata. If they don't, then we have no data on how noisy the image is, so this is all a moot point.
- dekhn 3y ago
  I understand you're trying to help, but everything you are suggesting is basically not helpful. Microscopists are savvy enough to match sensors and lenses, it's typically called "diffraction limited" (https://en.wikipedia.org/wiki/Diffraction-limited_system https://en.wikipedia.org/wiki/Diffraction-limited_system) based on the Rayleigh criterion (https://en.wikipedia.org/wiki/Angular_resolution#The_Rayleigh_criterion https://en.wikipedia.org/wiki/Angular_resolution#The_Rayleig...). I rarely if ever hear microscopists refer to nyquist limits. They would not overimage something at a higher magnification, or more correctly, they would be aware it was happening and either downscale or not worry. Next, when you make a point like "we can get a 1TB SSD for under $50". It shows you're not understanding the domain microscopists work in. Their scopes typically cost $100K (with a camera for another $50K, and a stage that costs $10K). The cost of the storage system (or parking the data in S3) for 10 years is usually far less than the capital cost of the scope, or the operational expense of maintaining staff on site. I believe the effective resolution can already be computed from the captured metadata, which includes the objective's numerical aperture, as well as the frequency used for imaging (many microscopes today use lasers to image with a very specific frequency to cause fluorenscence, which itself has a fairly tight output spectrum).

thfuran 3y ago

>which is all that really matters Except for the domains where nobody uses it because it ruins the data.

anthk 3y ago
Imagine that in the Medicine World with DICOM images.
- thfuran 3y ago
  DICOM technically supports lossy compression, but I think it's regarded as something of a historical mistake. It's pretty much never used and some viewers will plaster big warnings on lossily compressed series and others will outright refuse to load them.

_Microft 3y ago

> lossy compression (which is all that really matters) No, not in this case. This might be great for game files or whatever you are familiar with but lossy compression would basically mean to be mangling the data after going through all the effort of collecting it (I suspect you have only little experience with lab work?). When doing experiments, everything is documented in full detail. Settings on machines and equipment, preparation steps, version numbers of software used for processing, … You really don‘t want to lose information on your experiment.