3 ms·

Spent a decade managing microscopy data and tbh I don’t know how important or useful this is. People can share data in whatever format they have and it wouldn’t

Spent a decade managing microscopy data and tbh I don’t know how important or useful this is. People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another. Not that I have ever felt the need to do such verifications. It’ll take weeks to months to do anything like that.

COGlory 3y ago
There's real overhead. For instance, one spec that comes to mind the vendor starts the 0,0 pixel in the lower right. The spec calls for 0,0 to be the lower left. That leads to an inverted handedness for all images. Not a problem in projection, but as soon as you start in 3D, it creates a real mess. Some software packages try to detect when the camera was this vendor, and correct for it (silently). Other software packages refuse to correct for it, because they don't have a reliable way to detect it. A pipeline might have 3-4 of these software packages involved, eventually you have no idea if you handedness is correct and no real way to tell without being at a high enough resolution that you can use biological landmarks of known handedness. Don't get me started on Euler angles. Caveat being I'm more used to electron microscopy, maybe these things aren't as important with light microscopy because the resolutions are lower?
ahns 3y ago
I work with microscopy data and we have to convert all the proprietary-but-still-readable-by-some-random-package image data generated by microscopy companies to ome-tiff/ome-zarr for it to be in a manageable format. I think it's great!
LeifCarrotson 3y ago
> People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another. It's one thing for a highly-skilled user with a decade of experience to be able to import it eventually, another for an unskilled user to just have off-the-shelf tooling do it the same way for everyone automatically. This is more about meta-studies and unlocking new use cases. It sucks that the state-of-the-art for sharing academic microscopy data right now is basically looking at raster images embedded in PDFs, or email the authors, and then write a custom Scikit-Image script. Imagine if you had to read a PDF catalog and then email someone to order something off Amazon, or if your favorite CRUD app consisted instead of having an expert read a PDF and email a screenshots to you. What if sending those very emails to different recipients required implementing each users custom IMAP-like mail client. That sounds absurd, but it's kind of the way academic data sharing works now, lots of people are re-inventing the wheel and creating custom file formats. Consider, for example, the work of Dr. Bik (example at [1]) who identifies cloned sections from microscopy data. Or what if, instead of each researcher having to generate their own images, or get lucky and remember a particular image there was a Getty Images/AP Newsroom platform where you could just filter for your particular subject and imaging parameters and share your data. A collection of proprietary RAW files with randomly-formatted Excel documents for metadata would allow individual researchers to get their work done, but would be pretty worthless in comparison. [1] https://scienceintegritydigest.com/2023/06/27/concerns-about-12-md-anderson-cancer-centers-pathology-papers/#more-3243 https://scienceintegritydigest.com/2023/06/27/concerns-about...
yread 3y ago
I write a slide management platform that doesn't convert images to a single format and it mostly works. From time to time you do get issues though (cause you see millions of slides). Sometimes vendor's own libraries fail to open the slides (cough Phillips cough). Or there is a new version of firmware that sets some flag that wasn't used before. We support DICOM supp 145 too, but it's no panacea. There are still vendor specific quirks. The "surface" is larger (cause you expect all the metadata to be there in the standard format) so you still sometimes see differences.

COGlory 3y ago

There's real overhead. For instance, one spec that comes to mind the vendor starts the 0,0 pixel in the lower right. The spec calls for 0,0 to be the lower left. That leads to an inverted handedness for all images. Not a problem in projection, but as soon as you start in 3D, it creates a real mess. Some software packages try to detect when the camera was this vendor, and correct for it (silently). Other software packages refuse to correct for it, because they don't have a reliable way to detect it. A pipeline might have 3-4 of these software packages involved, eventually you have no idea if you handedness is correct and no real way to tell without being at a high enough resolution that you can use biological landmarks of known handedness. Don't get me started on Euler angles. Caveat being I'm more used to electron microscopy, maybe these things aren't as important with light microscopy because the resolutions are lower?

ahns 3y ago

I work with microscopy data and we have to convert all the proprietary-but-still-readable-by-some-random-package image data generated by microscopy companies to ome-tiff/ome-zarr for it to be in a manageable format. I think it's great!

LeifCarrotson 3y ago

> People can share data in whatever format they have and it wouldn’t be hard for me or others to import it one way or another. It's one thing for a highly-skilled user with a decade of experience to be able to import it eventually, another for an unskilled user to just have off-the-shelf tooling do it the same way for everyone automatically. This is more about meta-studies and unlocking new use cases. It sucks that the state-of-the-art for sharing academic microscopy data right now is basically looking at raster images embedded in PDFs, or email the authors, and then write a custom Scikit-Image script. Imagine if you had to read a PDF catalog and then email someone to order something off Amazon, or if your favorite CRUD app consisted instead of having an expert read a PDF and email a screenshots to you. What if sending those very emails to different recipients required implementing each users custom IMAP-like mail client. That sounds absurd, but it's kind of the way academic data sharing works now, lots of people are re-inventing the wheel and creating custom file formats. Consider, for example, the work of Dr. Bik (example at [1]) who identifies cloned sections from microscopy data. Or what if, instead of each researcher having to generate their own images, or get lucky and remember a particular image there was a Getty Images/AP Newsroom platform where you could just filter for your particular subject and imaging parameters and share your data. A collection of proprietary RAW files with randomly-formatted Excel documents for metadata would allow individual researchers to get their work done, but would be pretty worthless in comparison. [1] https://scienceintegritydigest.com/2023/06/27/concerns-about-12-md-anderson-cancer-centers-pathology-papers/#more-3243 https://scienceintegritydigest.com/2023/06/27/concerns-about...

yread 3y ago

I write a slide management platform that doesn't convert images to a single format and it mostly works. From time to time you do get issues though (cause you see millions of slides). Sometimes vendor's own libraries fail to open the slides (cough Phillips cough). Or there is a new version of firmware that sets some flag that wasn't used before. We support DICOM supp 145 too, but it's no panacea. There are still vendor specific quirks. The "surface" is larger (cause you expect all the metadata to be there in the standard format) so you still sometimes see differences.