3 ms·

Hi, once upon a time, I worked with medical image data. I literally had a DBA suggest storing the volumetric data in a database. I worked to diligently to exp

Hi, once upon a time, I worked with medical image data. I literally had a DBA suggest storing the volumetric data in a database. I worked to diligently to explain to him why he was incorrect. When he resisted my explanations and tried to push forward with his plan, I worked diligently to get him fired. I was successful.

Twisol 7y ago
For those of us who don’t work with medical image data, can you share why he was incorrect? I realize that relational databases are not the right box to fit certain kinds of data into, but you have to put your data somewhere that allows it to be efficiently manipulated. What is that if not a “data base”?
- pbourke 7y ago
  Stuff like this, back in the day: a RAIDed filesystem. Nowadays: a blob object store (like S3 or Azure Blob Store) or distributed FS such as HDFS
- aspaceman 7y ago
  Medical imaging data is typically dense 3D grids of density samples. A relational database could hold a pointer to the file, but not the 5GB-500GB dataset files. One dataset is typically a folder of various files, metadata, and other information representing an entire filesystem. Back in the day, the datasets would be RAID across several machines. Now they're virtual filesystems. The database could manage this info, but not hold the actual dataset itself. I imagine this is what the commentter means. The DBA probably said "it all needs to be in the database", and that just doesn't make sense.
  - xapata 7y ago
    Holds true for lots of data varieties; the database maintains links to compressed archives.
- user5994461 7y ago
  Short version is that a medical image, like a typical MRI, is a 3D picture at least 1024 x 1024 x 1024 pixels, meaning each image is multiple gigabytes. SQL databases don't do well at storing huge binary blobs like that.
  - Twisol 7y ago
    Right, but SQL databases aren't the only kind of database. I gather that the guy in the original anecdote was referring to a relational database specifically, but not all databases are relational. The filesystem is a perfectly good hierarchical document-based database, and it's worth thinking about it that way in both theory and in practice.
- VikingCoder 7y ago
  Let me try to make a metaphor for you. Imagine that you create HTML content, and you're considering sharing it with the world. You propose to create a "web server". The DBA next to you says you should just use MySQL. Stunned, you try to explain that your proposed "web server" will communicate using this new "HyperText Transfer Protocol." The DBA counters that SQL is fine. You try to explain about dynamic content and PHP, etc. The DBA counters that SQL stored procedures can do all of that. Next thing you know, grandma is typing: SELECT * FROM GOOGLE.COM WHERE TITLE LIKE "%CAT PICTURES%"; That's just an analogy, of course. But perhaps you can imagine your own list of reasons why MySQL didn't replace Apache. The answer to your question is that we could not possibly use any off the shelf software to provide the interfaces we needed (we were writing our own). By the time we had bytes that we needed to store somewhere, our data was about a GB of complicated structure that we accessed as memory-mapped files. (Look at https://capnproto.org/ https://capnproto.org/ for a rough analogy to the kind of access we needed.) And what I didn't say was that the DBA was literally recommending MySQL BLOBs, storing a bit more than 0.5 MB (512 * 512 * 2 bytes of image data) in each row, having a thousand rows or more per CT scan. The performance of that would have been absolute crap. It made literally zero sense.
zbobet2012 7y ago
I'm sure you are aware, but some databases have specific extensions and technology for this. See https://en.wikipedia.org/wiki/Spatial_database https://en.wikipedia.org/wiki/Spatial_database (though that tends to focus on GIS stuff). However, if you are simply "pulling blobs up" of compressed (or uncompressed) volumetric data... well yeah. Don't put that in a sql database.

Twisol 7y ago

For those of us who don’t work with medical image data, can you share why he was incorrect? I realize that relational databases are not the right box to fit certain kinds of data into, but you have to put your data somewhere that allows it to be efficiently manipulated. What is that if not a “data base”?

pbourke 7y ago
Stuff like this, back in the day: a RAIDed filesystem. Nowadays: a blob object store (like S3 or Azure Blob Store) or distributed FS such as HDFS
aspaceman 7y ago
Medical imaging data is typically dense 3D grids of density samples. A relational database could hold a pointer to the file, but not the 5GB-500GB dataset files. One dataset is typically a folder of various files, metadata, and other information representing an entire filesystem. Back in the day, the datasets would be RAID across several machines. Now they're virtual filesystems. The database could manage this info, but not hold the actual dataset itself. I imagine this is what the commentter means. The DBA probably said "it all needs to be in the database", and that just doesn't make sense.
- xapata 7y ago
  Holds true for lots of data varieties; the database maintains links to compressed archives.
user5994461 7y ago
Short version is that a medical image, like a typical MRI, is a 3D picture at least 1024 x 1024 x 1024 pixels, meaning each image is multiple gigabytes. SQL databases don't do well at storing huge binary blobs like that.
- Twisol 7y ago
  Right, but SQL databases aren't the only kind of database. I gather that the guy in the original anecdote was referring to a relational database specifically, but not all databases are relational. The filesystem is a perfectly good hierarchical document-based database, and it's worth thinking about it that way in both theory and in practice.
VikingCoder 7y ago
Let me try to make a metaphor for you. Imagine that you create HTML content, and you're considering sharing it with the world. You propose to create a "web server". The DBA next to you says you should just use MySQL. Stunned, you try to explain that your proposed "web server" will communicate using this new "HyperText Transfer Protocol." The DBA counters that SQL is fine. You try to explain about dynamic content and PHP, etc. The DBA counters that SQL stored procedures can do all of that. Next thing you know, grandma is typing: SELECT * FROM GOOGLE.COM WHERE TITLE LIKE "%CAT PICTURES%"; That's just an analogy, of course. But perhaps you can imagine your own list of reasons why MySQL didn't replace Apache. The answer to your question is that we could not possibly use any off the shelf software to provide the interfaces we needed (we were writing our own). By the time we had bytes that we needed to store somewhere, our data was about a GB of complicated structure that we accessed as memory-mapped files. (Look at https://capnproto.org/ https://capnproto.org/ for a rough analogy to the kind of access we needed.) And what I didn't say was that the DBA was literally recommending MySQL BLOBs, storing a bit more than 0.5 MB (512 * 512 * 2 bytes of image data) in each row, having a thousand rows or more per CT scan. The performance of that would have been absolute crap. It made literally zero sense.

zbobet2012 7y ago

I'm sure you are aware, but some databases have specific extensions and technology for this. See https://en.wikipedia.org/wiki/Spatial_database https://en.wikipedia.org/wiki/Spatial_database (though that tends to focus on GIS stuff). However, if you are simply "pulling blobs up" of compressed (or uncompressed) volumetric data... well yeah. Don't put that in a sql database.