9 ms·

Open-source could finally get the world’s microscopes speaking the same language

263 points by sohkamyung 3y ago

Havoc 3y ago
Academic gang seems quite good at converging on a solution in general though. Not always a very open solution but converge they do. See matlab etc.
Ericson2314 3y ago
Just like open access, I think at some point this sort of interoperability just has to be demanded by those actually funding the research. Two many narrow interests with no spare effort to coordinate otherwise.
- bafe 3y ago
  It's almost the same in the field of ELN/LIMs. Researchers are required to use one, but there's no wider strategy mandating the systems to be interoperable
- dguest 3y ago
  I'm in a field that would have to change significantly if funding agencies demanded open access. I would love it. Right not there's basically zero funding for people who work on making our data open. It's kind of a hobby project that you do if you have free time and no one has found a real job for you.
  - warkdarrior 3y ago
    I don't think anyone calls for funding just for making the data open. Funding should be for some scientific goal, with the added requirement that all data and code are made open source/open access once the funded project reaches a milestone (e.g., when publishing a paper).
    - michaelmior 3y ago
      I think part of the challenge here is that in some fields, proprietary tools and formats are so ingrained that requiring openness could be a massive burden to the point where it would be difficult to put together a reasonable budget that includes real open data access. I think it's important to acknowledge here that there are degrees of open access. Simply providing data files is relatively easy. Making reproducible workflows as a non-developer can be nearly impossible in some cases.
      - neltnerb 3y ago
        > Making reproducible workflows as a non-developer can be nearly impossible in some cases. Why is that the case? Size? I am very familiar with dealing with old proprietary things that have to be fought tooth and nail to give you a simple text format, but that doesn't seem to be nearly the scale of trouble as you describe.
        michaelmior 3y ago
        Old proprietary things can definitely be part of the problem. A lot of people will just cut and paste various scripts, download various packages, and make random edits until things work. I can't blame them. These things can be hard and for those not from a computer science background, it can be difficult to comprehend how things actually work. Just getting old software to compile can be a challenge in itself. You could say to just put everything in a Docker container. Then when you have it working, anyone can just use that container. That's not so easy for someone who doesn't even know what Docker is. I'm not saying that to suggest people are incompetent. As developers, we really should do better to make things easy.
      - dguest 3y ago
        Our software is open source already. What is ingrained is the idea among my collaborators that open data is kind of an "us vs them" problem: it's our data, and the public hasn't been demanding access, but if we gave it to them they would just scoop us and steal all the science, or they would do something stupid and force us to disprove it. It's all quite speculative but the limited data releases so far haven't caused the field to collapse as some doomsayers predicted. Meanwhile I'd just like more open data because it would make collaboration easier.
tecleandor 3y ago
Haven't seen the details, but I wonder what's the main difference between this OME-Zarr format and DICOM WSI, that's mostly aimed at the same type of images. Thinking out loud, this looks like the consolidation of several formats and projects. Bioformats, OME, Zarr... Back in the day I had the feeling that the OME/Bioformats people were more centered on research, and the DICOM crowd were mostly from the clinical side.
Joel_Mckay 3y ago
It should fall into an OpenCV module, and include an API standard for generic controller firmware. Research labs literally have a thousand other things to do... besides play wack-a-mole with vendor IT shenanigans. =)
- extragood 3y ago
  Absolutely. I spent an entire summer writing translation layers for a bunch of different microscopes so that they could interface with the detector software of the company I was interning for. It was tedious work!
ramraj07 3y ago
Spent a decade managing microscopy data and tbh I don’t know how important or useful this is. People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another. Not that I have ever felt the need to do such verifications. It’ll take weeks to months to do anything like that.
- COGlory 3y ago
  There's real overhead. For instance, one spec that comes to mind the vendor starts the 0,0 pixel in the lower right. The spec calls for 0,0 to be the lower left. That leads to an inverted handedness for all images. Not a problem in projection, but as soon as you start in 3D, it creates a real mess. Some software packages try to detect when the camera was this vendor, and correct for it (silently). Other software packages refuse to correct for it, because they don't have a reliable way to detect it. A pipeline might have 3-4 of these software packages involved, eventually you have no idea if you handedness is correct and no real way to tell without being at a high enough resolution that you can use biological landmarks of known handedness. Don't get me started on Euler angles. Caveat being I'm more used to electron microscopy, maybe these things aren't as important with light microscopy because the resolutions are lower?
- ahns 3y ago
  I work with microscopy data and we have to convert all the proprietary-but-still-readable-by-some-random-package image data generated by microscopy companies to ome-tiff/ome-zarr for it to be in a manageable format. I think it's great!
- LeifCarrotson 3y ago
  > People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another. It's one thing for a highly-skilled user with a decade of experience to be able to import it eventually, another for an unskilled user to just have off-the-shelf tooling do it the same way for everyone automatically. This is more about meta-studies and unlocking new use cases. It sucks that the state-of-the-art for sharing academic microscopy data right now is basically looking at raster images embedded in PDFs, or email the authors, and then write a custom Scikit-Image script. Imagine if you had to read a PDF catalog and then email someone to order something off Amazon, or if your favorite CRUD app consisted instead of having an expert read a PDF and email a screenshots to you. What if sending those very emails to different recipients required implementing each users custom IMAP-like mail client. That sounds absurd, but it's kind of the way academic data sharing works now, lots of people are re-inventing the wheel and creating custom file formats. Consider, for example, the work of Dr. Bik (example at [1]) who identifies cloned sections from microscopy data. Or what if, instead of each researcher having to generate their own images, or get lucky and remember a particular image there was a Getty Images/AP Newsroom platform where you could just filter for your particular subject and imaging parameters and share your data. A collection of proprietary RAW files with randomly-formatted Excel documents for metadata would allow individual researchers to get their work done, but would be pretty worthless in comparison. [1] https://scienceintegritydigest.com/2023/06/27/concerns-about-12-md-anderson-cancer-centers-pathology-papers/#more-3243 https://scienceintegritydigest.com/2023/06/27/concerns-about...
charcircuit 3y ago
>The cloud, however, treats data as a single unstructured entity that is either downloaded in its entirety or not S3 supports the range header. Google Cloud, Azure, and I'm sure other object stores support it too.
- saltcured 3y ago
  Yes, it's really frustrating that Zarr is promoting these exploded file/object trees and not providing a viable standard serialization for an entire image. It's ironic that they mention FAIR but don't see how much better it would be to have a serialized byte-stream you can cite and share with checksums etc.
denton-scratch 3y ago
> Each pixel must be labelled with metadata, such as illumination level, its 3D position, the scale, the sample type and how the sample was prepared. Each pixel? Why? All of those except the 3D position apply to all the pixels in a given image, and the (2D) position of a pixel can be inferred from its location in the image. Wait - are there optical microscopes that can create 3D images? I know you can see a 3D image if you peer into a binocular microscope, but AFAIK cameras for those things are always 2D cameras.
- ahns 3y ago
  As far as I know, not technically (although I haven't kept up in the area), but you can definitely sweep through a volumetric sample; there are microscopes that can for example illuminate a thin z-plane of a transparent sample and collect the image or those that can reject out-of-focus (off-z) light for a particular z-plane, then move to another z-plane, etc. and then generate a volume on the software side.
  - saltcured 3y ago
    Yes, I think you are talking about light-sheet microscopy. This tries to illuminate a thin layer and then images with a 2D sensor like in a normal digital camera. This is often a monochrome sensor, and extra metadata records what kind of optical filters were in place. A multi-channel image would then have separate planes captured with different optical band-pass filters. You need metadata to know how a set of planes relate to each other, whether shifting through space or changing filters or just measuring the same configuration again at different time points. There are also confocal "laser-scanning" microscopes which effectively illuminate a single point in space and image with what is effectively a single monochrome pixel sensor. In this case, the raw signal is pretty much a time series like you might imagine with digital audio. You need metadata to tell you how the optics were being shifted around, to interpret each sample as representing a point in space at a particular time and with what filters. Normally, the first round of interpretation is done during the initial file save, packing measurements into a file format that still expects planar, rectangular pixel arrays with a regular grid spacing for the neighboring pixels. You pretend the whole image plane was captured at a time point, but it was really measured sequentially somewhat like a cathode-ray tube displays an image one pixel at a time. A session might produce many such planes, and metadata is still needed to understand whether these planes represent the same plane observed over time and/or parallel planes spanning a volume and/or different channels of the same plane. On the other end of the spectrum, there are slide scanners which image a much larger stage area by shifting around a 2D camera and taking many overlapping images. So a 2048x2048 sensor might be used to produce a 100,000 x 60,000 pixel image plane. Each sub-image would record its position within the stage area and these do not necessarily line up at precise multiples of image pixels. There might also be gaps where the scanner first detects the overall shape of specimen(s) on the slide and then plans a set of images to cover the specimen while skipping over the background area. This sparsity saves time for imaging as well as storage and transfer times by skipping empty areas.
klysm 3y ago
Microscope vendors have gone to incredible efforts to obfuscate their file formats. Some inventing custom compression codecs. I worked for a company that spent enormous amounts of time reverse engineering these protocols.
- Ericson2314 3y ago
  Labs should not be allowed to spend grant money on this bullshit.
- ahns 3y ago
  :/ [tell me about it...](https://paulbourke.net/dataformats/svs/ https://paulbourke.net/dataformats/svs/)
  - yread 3y ago
    Hah SVS is child's play compared to MRXS: https://lists.andrew.cmu.edu/pipermail/openslide-users/2012-July/000373.html https://lists.andrew.cmu.edu/pipermail/openslide-users/2012-...
    - extraduder_ire 3y ago
      I'd hope imagemagick or some other program can at least get the data out into another format. I did not know about openslide before reading that mail, but I'll try to keep it in mind. Here's hoping this eventually gets fixed in a similar sort of way the .mkv format "fixed" video.
    - klysm 3y ago
      I wasn’t going to namedrop in my original comment but oh my the horrors this format brings back.
- dekhn 3y ago
  bioformats2raw https://github.com/glencoesoftware/bioformats2raw https://github.com/glencoesoftware/bioformats2raw handles a wide range of obfuscated formats.
zackmorris 3y ago
Without knowing anything about this, I wonder if standards are trying to work at the wrong level of abstraction. Like worrying about whether to use GIF or PNG to compress map images, when really it would be better to standardize the underlying data like OpenStreetMap. A better goal might be to decide on a single container format for discrete cosine transform (DCT) data in compressed images like how JPEG works. We probably need a multidimensional lossy compression scheme for 2D images, 3D volumes/video, and 4+D metadata involving multiple spectrums, exposures, etc. I'd probably start with a multidimensional DCT with a way to specify the coefficients/weights of each layer and metadata (optionally losslessly): https://en.wikipedia.org/wiki/Discrete_cosine_transform#Multidimensional_DCTs https://en.wikipedia.org/wiki/Discrete_cosine_transform#Mult... Looks like that's built into MATLAB with dct(x,n,dim): https://www.mathworks.com/help/signal/ref/dct.html https://www.mathworks.com/help/signal/ref/dct.html https://www.mathworks.com/help/images/ref/blockproc.html https://www.mathworks.com/help/images/ref/blockproc.html (coefficients) https://www.mathworks.com/help/images/discrete-cosine-transform.html https://www.mathworks.com/help/images/discrete-cosine-transf... (2D example) Then store the data in zipped JSON or BSON, or maybe TAR then ZIP. Then I'd write an importer and exporter from/to all of the other formats, with a single 0-1 quality argument or maybe an array of qualities for each layer, with a way to specify coefficients or load them from files. Once a known-good implementation worked, I'd add additional compression schemes that can compress across dimensions, so that a level of zoom only has to save a diff from a previous layer at a slightly different scale or focus (like a 3+D MPEG format, this is probably the standard that we're currently missing, does anyone know of one?). This all seems fairly straightforward though, so I wonder if proprietary software patents are the real blocker..
- dekhn 3y ago
  The OME-zarr creators are pretty experienced in this field and have already done the work you're describing. Let's not create another standard after OME-zarr.
  - zackmorris 3y ago
    Hey thanks, I tend to agree. Although if I read this right, OME-zarr punted on lossy compression (which is all that really matters) so IMHO it doesn't have much use in the real world yet even though people evidently put a lot of work into it: https://github.com/ome/ome-zarr-py https://github.com/ome/ome-zarr-py https://ngff.openmicroscopy.org/latest/ https://ngff.openmicroscopy.org/latest/ https://zarr.readthedocs.io/en/v2.1.0/api/codecs.html https://zarr.readthedocs.io/en/v2.1.0/api/codecs.html https://www.researchgate.net/publication/356622607_OME-NGFF_a_next-generation_file_format_for_expanding_bioimaging_data-access_strategies#read https://www.researchgate.net/publication/356622607_OME-NGFF_... https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1.full.pdf https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1.... i.munro (Ian Munro) August 15, 2023, 8:54am On the original point about size I suspect your answer is compression. I have found that a lot of image formats use lossy compression. I’m not sure about CZI specifically. AFAIK ome-zarr (sensibly IMO) uses only lossless compression. Best Ian and: sebi06 (Sebi06) August 15, 2023, 1:42pm Hi all, for CZI we offer currentyl two compression methods: JPG-XR ZSTD So we have lossless and lossy options depending on your needs. All of them are implemented in our C++, #NET or Python APIs and also supported from BioFormats. Looks like CZI is a competitor: https://www.zeiss.com/microscopy/en/products/software/zeiss-zen/czi-image-file-format.html https://www.zeiss.com/microscopy/en/products/software/zeiss-... https://github.com/cgohlke/czifile https://github.com/cgohlke/czifile http://www.physics.hmc.edu/~gerbode/wppriv/wp-content/uploads/2012/06/DS_ZISRAW-FileFormat_Rel_ZEN2011.pdf http://www.physics.hmc.edu/~gerbode/wppriv/wp-content/upload... CZI is basically JPEG compression and is probably missing the multidimensional compressor which is the secret sauce missing from the various formats. It also mentions licensing and legal so is maybe lost in the weeds and not open source. Without more info, it feels like this is still an open problem. My interest in this is for video games, since so many are multiple GB in size but are also missing this multidimensional compressor. If we had it, I'd guess a 10:1 to 100:1 reduction in size over what we have now, with no perceptual loss in quality.
amelius 3y ago
It's too bad that it's based on XML and that this is actually the best we have.
- dekhn 3y ago
  ome-zarr puts metadata in json not XML. There was an older standard that put the metadata in XML. I've already written converters (converted the XML schema to JSON schema), it wasn't really an issue. The raw data is in blocked compressed numpy ararys in a directory structure on disk.
tomnicholas 3y ago
This article misses one of the coolest things about the Zarr format - that it's flexible enough that it's also becoming widely used in climate science. In particular the Pangeo project (https://pangeo.io/architecture.html https://pangeo.io/architecture.html) uses large Zarr stores as a performant format in the cloud which we can analyse in parallel at scale using distributed computing frameworks like dask. More and more climate science data is being made publicly available as Zarr in the cloud, often through open data partnerships with cloud providers (e.g. on AWS (https://aws.amazon.com/blogs/publicsector/decrease-geospatial-query-latency-minutes-seconds-using-zarr-amazon-s3/ https://aws.amazon.com/blogs/publicsector/decrease-geospatia...) ERA-5 on GCP(https://cloud.google.com/storage/docs/public-datasets/era5 https://cloud.google.com/storage/docs/public-datasets/era5)). I personally think that the more that common tooling can be shared between scientific disciplines the better.
- fuzzfactor 3y ago
  >flexible enough that it's also becoming widely used in climate science. For scientific instruments just the opposite took place in the 1990's with chromatography. They adapted a climate file type for laboratory instrument use. It was quite a saga but in a nutshell it was the manufacturers who well understood the benefit of a common file type once computers had taken enough of a hold. Most lab analyses didn't need a computer except for the most data-intensive, which could never wait and were being handled by mainframes or minis before the PC appeared. Only by those that could afford such costly investments like oil companies and drug companies. Which are some of the most desirable customers for instrumentation. At the time HP's lab instrument division (spun off decades ago as part of Agilent) had the most agressive sales approach but if a potential customer was already entrenched with one of the few other vendors there was no efficient way to carry their past proprietary data forward into a new system. So the vendors got together, formed the Analytical Instrument Association (AIA) and began work in earnest toward a common file type they could all agree on for data interchange, without having to abandon each of the file types they had developed earlier which remained in everyday use within their proprietary software. Once this had been accomplished it would be more of a battle of the salespeople with one less (big) technical obstacle to overcome. They ended up choosing a well-established extensible file type, netCDF, which was in wide use at NOAA without encumbrance. Tailored it to the task and after a number of years the new standard was almost complete. Major vendors were already supporting it. Well, by that time HP had already become dominant so they stealthily lost interest for a while before pulling out of AIA completely. Fortunately with no further progress in sight the whole thing was dropped in the lap of ASTM, where it has remained unchanged for over two decades. That is even more fortunate because hindsight has shown the difference between an ideal standard and a nonideal standard is that an ideal standard never changes, and this is one of the best examples. These are often referred to as CDF files even though that does not stand for Chromatography Data File, they were just named netCDF by the weather service during the mainframe days. So for more than 20 years each PC chromatography data system has had the capability of handling AIA files for either input or output. The underlying proprietary files which each software package utilizes under the hood for real-time data acquisition may have some unique features, but the common netCDF interchange covers all the essential bases needed for post-processing. Anything less is sub-standard. Only in the last few years has a major software package appeared which for the first time does not fully support CDF's and that is the latest offering from Agilent of all places. It can only create optional CDF files from the data acquired by their own instruments, so far there is no allowance for inputting a standard CDF file for processing by their software any more.
dhfbshfbu4u3 3y ago
Why bother? Someone will come up with a model that will just suck it all up and convert it automatically... and by someone I mean some sufficiently motivated LLM.
- deleted 3y ago
  [deleted]
- wiml 3y ago
  It'll convert the image to something that has the same overall statistics as the real data but may not bear any resemblance to reality in the details
andbberger 3y ago
i hope a new format, be it N5 or OME-zarr, achieves critical mass. but i'm not holding my breath. NIfTI never really caught on, despite having a very nice spec. unless scope manufacturers unite behind a format, biologists are going to keep writing bespoke analysis pipelines for the formats their images come in. it's going to take some top down regulation from the NSF/NIH requiring that all scopes support the open format of choice by a certain date to be eligible for purchase with grant money.
deleted 3y ago
[deleted]
userbinator 3y ago
I feel like this is just going to lead to another design-by-committee overly complex standard that very few can actually implement, and the desire to reinvent instead of reuse existing standards with some additions is responsible for much of that. A lot of the features they mentioned like scalable compression / "region of interest" were also present in JPEG2000. IMHO "reduce, reuse, recycle" works well as a software design principle too.

263 points by sohkamyung 3y ago

Academic gang seems quite good at converging on a solution in general though. Not always a very open solution but converge they do. See matlab etc.

Ericson2314 3y ago

Just like open access, I think at some point this sort of interoperability just has to be demanded by those actually funding the research. Two many narrow interests with no spare effort to coordinate otherwise.

bafe 3y ago
It's almost the same in the field of ELN/LIMs. Researchers are required to use one, but there's no wider strategy mandating the systems to be interoperable
dguest 3y ago
I'm in a field that would have to change significantly if funding agencies demanded open access. I would love it. Right not there's basically zero funding for people who work on making our data open. It's kind of a hobby project that you do if you have free time and no one has found a real job for you.
- warkdarrior 3y ago
  I don't think anyone calls for funding just for making the data open. Funding should be for some scientific goal, with the added requirement that all data and code are made open source/open access once the funded project reaches a milestone (e.g., when publishing a paper).
  - michaelmior 3y ago
    I think part of the challenge here is that in some fields, proprietary tools and formats are so ingrained that requiring openness could be a massive burden to the point where it would be difficult to put together a reasonable budget that includes real open data access. I think it's important to acknowledge here that there are degrees of open access. Simply providing data files is relatively easy. Making reproducible workflows as a non-developer can be nearly impossible in some cases.
    - neltnerb 3y ago
      > Making reproducible workflows as a non-developer can be nearly impossible in some cases. Why is that the case? Size? I am very familiar with dealing with old proprietary things that have to be fought tooth and nail to give you a simple text format, but that doesn't seem to be nearly the scale of trouble as you describe.
      - michaelmior 3y ago
        Old proprietary things can definitely be part of the problem. A lot of people will just cut and paste various scripts, download various packages, and make random edits until things work. I can't blame them. These things can be hard and for those not from a computer science background, it can be difficult to comprehend how things actually work. Just getting old software to compile can be a challenge in itself. You could say to just put everything in a Docker container. Then when you have it working, anyone can just use that container. That's not so easy for someone who doesn't even know what Docker is. I'm not saying that to suggest people are incompetent. As developers, we really should do better to make things easy.
    - dguest 3y ago
      Our software is open source already. What is ingrained is the idea among my collaborators that open data is kind of an "us vs them" problem: it's our data, and the public hasn't been demanding access, but if we gave it to them they would just scoop us and steal all the science, or they would do something stupid and force us to disprove it. It's all quite speculative but the limited data releases so far haven't caused the field to collapse as some doomsayers predicted. Meanwhile I'd just like more open data because it would make collaboration easier.

tecleandor 3y ago

Haven't seen the details, but I wonder what's the main difference between this OME-Zarr format and DICOM WSI, that's mostly aimed at the same type of images. Thinking out loud, this looks like the consolidation of several formats and projects. Bioformats, OME, Zarr... Back in the day I had the feeling that the OME/Bioformats people were more centered on research, and the DICOM crowd were mostly from the clinical side.

Joel_Mckay 3y ago

It should fall into an OpenCV module, and include an API standard for generic controller firmware. Research labs literally have a thousand other things to do... besides play wack-a-mole with vendor IT shenanigans. =)

extragood 3y ago
Absolutely. I spent an entire summer writing translation layers for a bunch of different microscopes so that they could interface with the detector software of the company I was interning for. It was tedious work!

ramraj07 3y ago

Spent a decade managing microscopy data and tbh I don’t know how important or useful this is. People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another. Not that I have ever felt the need to do such verifications. It’ll take weeks to months to do anything like that.

COGlory 3y ago
There's real overhead. For instance, one spec that comes to mind the vendor starts the 0,0 pixel in the lower right. The spec calls for 0,0 to be the lower left. That leads to an inverted handedness for all images. Not a problem in projection, but as soon as you start in 3D, it creates a real mess. Some software packages try to detect when the camera was this vendor, and correct for it (silently). Other software packages refuse to correct for it, because they don't have a reliable way to detect it. A pipeline might have 3-4 of these software packages involved, eventually you have no idea if you handedness is correct and no real way to tell without being at a high enough resolution that you can use biological landmarks of known handedness. Don't get me started on Euler angles. Caveat being I'm more used to electron microscopy, maybe these things aren't as important with light microscopy because the resolutions are lower?
ahns 3y ago
I work with microscopy data and we have to convert all the proprietary-but-still-readable-by-some-random-package image data generated by microscopy companies to ome-tiff/ome-zarr for it to be in a manageable format. I think it's great!
LeifCarrotson 3y ago
> People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another. It's one thing for a highly-skilled user with a decade of experience to be able to import it eventually, another for an unskilled user to just have off-the-shelf tooling do it the same way for everyone automatically. This is more about meta-studies and unlocking new use cases. It sucks that the state-of-the-art for sharing academic microscopy data right now is basically looking at raster images embedded in PDFs, or email the authors, and then write a custom Scikit-Image script. Imagine if you had to read a PDF catalog and then email someone to order something off Amazon, or if your favorite CRUD app consisted instead of having an expert read a PDF and email a screenshots to you. What if sending those very emails to different recipients required implementing each users custom IMAP-like mail client. That sounds absurd, but it's kind of the way academic data sharing works now, lots of people are re-inventing the wheel and creating custom file formats. Consider, for example, the work of Dr. Bik (example at [1]) who identifies cloned sections from microscopy data. Or what if, instead of each researcher having to generate their own images, or get lucky and remember a particular image there was a Getty Images/AP Newsroom platform where you could just filter for your particular subject and imaging parameters and share your data. A collection of proprietary RAW files with randomly-formatted Excel documents for metadata would allow individual researchers to get their work done, but would be pretty worthless in comparison. [1] https://scienceintegritydigest.com/2023/06/27/concerns-about-12-md-anderson-cancer-centers-pathology-papers/#more-3243 https://scienceintegritydigest.com/2023/06/27/concerns-about...

charcircuit 3y ago

>The cloud, however, treats data as a single unstructured entity that is either downloaded in its entirety or not S3 supports the range header. Google Cloud, Azure, and I'm sure other object stores support it too.

saltcured 3y ago
Yes, it's really frustrating that Zarr is promoting these exploded file/object trees and not providing a viable standard serialization for an entire image. It's ironic that they mention FAIR but don't see how much better it would be to have a serialized byte-stream you can cite and share with checksums etc.

denton-scratch 3y ago

> Each pixel must be labelled with metadata, such as illumination level, its 3D position, the scale, the sample type and how the sample was prepared. Each pixel? Why? All of those except the 3D position apply to all the pixels in a given image, and the (2D) position of a pixel can be inferred from its location in the image. Wait - are there optical microscopes that can create 3D images? I know you can see a 3D image if you peer into a binocular microscope, but AFAIK cameras for those things are always 2D cameras.

ahns 3y ago
As far as I know, not technically (although I haven't kept up in the area), but you can definitely sweep through a volumetric sample; there are microscopes that can for example illuminate a thin z-plane of a transparent sample and collect the image or those that can reject out-of-focus (off-z) light for a particular z-plane, then move to another z-plane, etc. and then generate a volume on the software side.
- saltcured 3y ago
  Yes, I think you are talking about light-sheet microscopy. This tries to illuminate a thin layer and then images with a 2D sensor like in a normal digital camera. This is often a monochrome sensor, and extra metadata records what kind of optical filters were in place. A multi-channel image would then have separate planes captured with different optical band-pass filters. You need metadata to know how a set of planes relate to each other, whether shifting through space or changing filters or just measuring the same configuration again at different time points. There are also confocal "laser-scanning" microscopes which effectively illuminate a single point in space and image with what is effectively a single monochrome pixel sensor. In this case, the raw signal is pretty much a time series like you might imagine with digital audio. You need metadata to tell you how the optics were being shifted around, to interpret each sample as representing a point in space at a particular time and with what filters. Normally, the first round of interpretation is done during the initial file save, packing measurements into a file format that still expects planar, rectangular pixel arrays with a regular grid spacing for the neighboring pixels. You pretend the whole image plane was captured at a time point, but it was really measured sequentially somewhat like a cathode-ray tube displays an image one pixel at a time. A session might produce many such planes, and metadata is still needed to understand whether these planes represent the same plane observed over time and/or parallel planes spanning a volume and/or different channels of the same plane. On the other end of the spectrum, there are slide scanners which image a much larger stage area by shifting around a 2D camera and taking many overlapping images. So a 2048x2048 sensor might be used to produce a 100,000 x 60,000 pixel image plane. Each sub-image would record its position within the stage area and these do not necessarily line up at precise multiples of image pixels. There might also be gaps where the scanner first detects the overall shape of specimen(s) on the slide and then plans a set of images to cover the specimen while skipping over the background area. This sparsity saves time for imaging as well as storage and transfer times by skipping empty areas.

klysm 3y ago

Microscope vendors have gone to incredible efforts to obfuscate their file formats. Some inventing custom compression codecs. I worked for a company that spent enormous amounts of time reverse engineering these protocols.

Ericson2314 3y ago
Labs should not be allowed to spend grant money on this bullshit.
ahns 3y ago
:/ [tell me about it...](https://paulbourke.net/dataformats/svs/ https://paulbourke.net/dataformats/svs/)
- yread 3y ago
  Hah SVS is child's play compared to MRXS: https://lists.andrew.cmu.edu/pipermail/openslide-users/2012-July/000373.html https://lists.andrew.cmu.edu/pipermail/openslide-users/2012-...
  - extraduder_ire 3y ago
    I'd hope imagemagick or some other program can at least get the data out into another format. I did not know about openslide before reading that mail, but I'll try to keep it in mind. Here's hoping this eventually gets fixed in a similar sort of way the .mkv format "fixed" video.
  - klysm 3y ago
    I wasn’t going to namedrop in my original comment but oh my the horrors this format brings back.
dekhn 3y ago
bioformats2raw https://github.com/glencoesoftware/bioformats2raw https://github.com/glencoesoftware/bioformats2raw handles a wide range of obfuscated formats.

zackmorris 3y ago

Without knowing anything about this, I wonder if standards are trying to work at the wrong level of abstraction. Like worrying about whether to use GIF or PNG to compress map images, when really it would be better to standardize the underlying data like OpenStreetMap. A better goal might be to decide on a single container format for discrete cosine transform (DCT) data in compressed images like how JPEG works. We probably need a multidimensional lossy compression scheme for 2D images, 3D volumes/video, and 4+D metadata involving multiple spectrums, exposures, etc. I'd probably start with a multidimensional DCT with a way to specify the coefficients/weights of each layer and metadata (optionally losslessly): https://en.wikipedia.org/wiki/Discrete_cosine_transform#Multidimensional_DCTs https://en.wikipedia.org/wiki/Discrete_cosine_transform#Mult... Looks like that's built into MATLAB with dct(x,n,dim): https://www.mathworks.com/help/signal/ref/dct.html https://www.mathworks.com/help/signal/ref/dct.html https://www.mathworks.com/help/images/ref/blockproc.html https://www.mathworks.com/help/images/ref/blockproc.html (coefficients) https://www.mathworks.com/help/images/discrete-cosine-transform.html https://www.mathworks.com/help/images/discrete-cosine-transf... (2D example) Then store the data in zipped JSON or BSON, or maybe TAR then ZIP. Then I'd write an importer and exporter from/to all of the other formats, with a single 0-1 quality argument or maybe an array of qualities for each layer, with a way to specify coefficients or load them from files. Once a known-good implementation worked, I'd add additional compression schemes that can compress across dimensions, so that a level of zoom only has to save a diff from a previous layer at a slightly different scale or focus (like a 3+D MPEG format, this is probably the standard that we're currently missing, does anyone know of one?). This all seems fairly straightforward though, so I wonder if proprietary software patents are the real blocker..

dekhn 3y ago
The OME-zarr creators are pretty experienced in this field and have already done the work you're describing. Let's not create another standard after OME-zarr.
- zackmorris 3y ago
  Hey thanks, I tend to agree. Although if I read this right, OME-zarr punted on lossy compression (which is all that really matters) so IMHO it doesn't have much use in the real world yet even though people evidently put a lot of work into it: https://github.com/ome/ome-zarr-py https://github.com/ome/ome-zarr-py https://ngff.openmicroscopy.org/latest/ https://ngff.openmicroscopy.org/latest/ https://zarr.readthedocs.io/en/v2.1.0/api/codecs.html https://zarr.readthedocs.io/en/v2.1.0/api/codecs.html https://www.researchgate.net/publication/356622607_OME-NGFF_a_next-generation_file_format_for_expanding_bioimaging_data-access_strategies#read https://www.researchgate.net/publication/356622607_OME-NGFF_... https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1.full.pdf https://www.biorxiv.org/content/10.1101/2023.02.17.528834v1.... i.munro (Ian Munro) August 15, 2023, 8:54am On the original point about size I suspect your answer is compression. I have found that a lot of image formats use lossy compression. I’m not sure about CZI specifically. AFAIK ome-zarr (sensibly IMO) uses only lossless compression. Best Ian and: sebi06 (Sebi06) August 15, 2023, 1:42pm Hi all, for CZI we offer currentyl two compression methods: JPG-XR ZSTD So we have lossless and lossy options depending on your needs. All of them are implemented in our C++, #NET or Python APIs and also supported from BioFormats. Looks like CZI is a competitor: https://www.zeiss.com/microscopy/en/products/software/zeiss-zen/czi-image-file-format.html https://www.zeiss.com/microscopy/en/products/software/zeiss-... https://github.com/cgohlke/czifile https://github.com/cgohlke/czifile http://www.physics.hmc.edu/~gerbode/wppriv/wp-content/uploads/2012/06/DS_ZISRAW-FileFormat_Rel_ZEN2011.pdf http://www.physics.hmc.edu/~gerbode/wppriv/wp-content/upload... CZI is basically JPEG compression and is probably missing the multidimensional compressor which is the secret sauce missing from the various formats. It also mentions licensing and legal so is maybe lost in the weeds and not open source. Without more info, it feels like this is still an open problem. My interest in this is for video games, since so many are multiple GB in size but are also missing this multidimensional compressor. If we had it, I'd guess a 10:1 to 100:1 reduction in size over what we have now, with no perceptual loss in quality.

amelius 3y ago

It's too bad that it's based on XML and that this is actually the best we have.

dekhn 3y ago
ome-zarr puts metadata in json not XML. There was an older standard that put the metadata in XML. I've already written converters (converted the XML schema to JSON schema), it wasn't really an issue. The raw data is in blocked compressed numpy ararys in a directory structure on disk.

tomnicholas 3y ago

This article misses one of the coolest things about the Zarr format - that it's flexible enough that it's also becoming widely used in climate science. In particular the Pangeo project (https://pangeo.io/architecture.html https://pangeo.io/architecture.html) uses large Zarr stores as a performant format in the cloud which we can analyse in parallel at scale using distributed computing frameworks like dask. More and more climate science data is being made publicly available as Zarr in the cloud, often through open data partnerships with cloud providers (e.g. on AWS (https://aws.amazon.com/blogs/publicsector/decrease-geospatial-query-latency-minutes-seconds-using-zarr-amazon-s3/ https://aws.amazon.com/blogs/publicsector/decrease-geospatia...) ERA-5 on GCP(https://cloud.google.com/storage/docs/public-datasets/era5 https://cloud.google.com/storage/docs/public-datasets/era5)). I personally think that the more that common tooling can be shared between scientific disciplines the better.

fuzzfactor 3y ago
>flexible enough that it's also becoming widely used in climate science. For scientific instruments just the opposite took place in the 1990's with chromatography. They adapted a climate file type for laboratory instrument use. It was quite a saga but in a nutshell it was the manufacturers who well understood the benefit of a common file type once computers had taken enough of a hold. Most lab analyses didn't need a computer except for the most data-intensive, which could never wait and were being handled by mainframes or minis before the PC appeared. Only by those that could afford such costly investments like oil companies and drug companies. Which are some of the most desirable customers for instrumentation. At the time HP's lab instrument division (spun off decades ago as part of Agilent) had the most agressive sales approach but if a potential customer was already entrenched with one of the few other vendors there was no efficient way to carry their past proprietary data forward into a new system. So the vendors got together, formed the Analytical Instrument Association (AIA) and began work in earnest toward a common file type they could all agree on for data interchange, without having to abandon each of the file types they had developed earlier which remained in everyday use within their proprietary software. Once this had been accomplished it would be more of a battle of the salespeople with one less (big) technical obstacle to overcome. They ended up choosing a well-established extensible file type, netCDF, which was in wide use at NOAA without encumbrance. Tailored it to the task and after a number of years the new standard was almost complete. Major vendors were already supporting it. Well, by that time HP had already become dominant so they stealthily lost interest for a while before pulling out of AIA completely. Fortunately with no further progress in sight the whole thing was dropped in the lap of ASTM, where it has remained unchanged for over two decades. That is even more fortunate because hindsight has shown the difference between an ideal standard and a nonideal standard is that an ideal standard never changes, and this is one of the best examples. These are often referred to as CDF files even though that does not stand for Chromatography Data File, they were just named netCDF by the weather service during the mainframe days. So for more than 20 years each PC chromatography data system has had the capability of handling AIA files for either input or output. The underlying proprietary files which each software package utilizes under the hood for real-time data acquisition may have some unique features, but the common netCDF interchange covers all the essential bases needed for post-processing. Anything less is sub-standard. Only in the last few years has a major software package appeared which for the first time does not fully support CDF's and that is the latest offering from Agilent of all places. It can only create optional CDF files from the data acquired by their own instruments, so far there is no allowance for inputting a standard CDF file for processing by their software any more.

dhfbshfbu4u3 3y ago

Why bother? Someone will come up with a model that will just suck it all up and convert it automatically... and by someone I mean some sufficiently motivated LLM.

deleted 3y ago
[deleted]
wiml 3y ago
It'll convert the image to something that has the same overall statistics as the real data but may not bear any resemblance to reality in the details

andbberger 3y ago

i hope a new format, be it N5 or OME-zarr, achieves critical mass. but i'm not holding my breath. NIfTI never really caught on, despite having a very nice spec. unless scope manufacturers unite behind a format, biologists are going to keep writing bespoke analysis pipelines for the formats their images come in. it's going to take some top down regulation from the NSF/NIH requiring that all scopes support the open format of choice by a certain date to be eligible for purchase with grant money.

deleted 3y ago

[deleted]

userbinator 3y ago

I feel like this is just going to lead to another design-by-committee overly complex standard that very few can actually implement, and the desire to reinvent instead of reuse existing standards with some additions is responsible for much of that. A lot of the features they mentioned like scalable compression / "region of interest" were also present in JPEG2000. IMHO "reduce, reuse, recycle" works well as a software design principle too.