3 ms·
What databases seem to be REALLY bad at though is storing large blocks of CDATA/"unsorted data" with meta data attached. A small business has grown up using no
by UnoriginalGuy 13y ago
What databases seem to be REALLY bad at though is storing large blocks of CDATA/"unsorted data" with meta data attached.
A small business has grown up using nothing but files. These files store large blocks of text which cannot be altered, just meta data pulled out (they're flat files, between 10 KB and 5 MB each, which is basically like dumping a large char[] arrays into files, and unfortunately they need to be kept in this format because the formats vary/incompatibilities/flexibility).
Anyway... Basically they have 30+ GB of these flat files. Literally millions of them. All in *.dat files. It works quite well like that. But the downsides as discussed in the link crop up and cause problems.
So we tried to put them into a few SQL databases as LONGBLOB, and after the first couple of gig the thing just became unmanageably slow, badly formed queries would cause it to lock up, and the thing took up way more space on the file-system than files did (and was WAY harder to manage with the performance issues discussed).
So I guess my point is, SQL databases are great for sorted data. They're freaking nightmares for CDATA/BLOBs. Just a complete waste of space. Almost no database is designed to handle that amount of data and continue to work.
- deleted 13y ago[deleted]
- icebraining 13y agoNot that the filesystem isn't a valid option, but there are more databases than SQL databases. That cases seems great for a key-value store (which is essentially what a filesystem is).
- UnoriginalGuy 13y agoTrue there are, and I'd be lying if I said I had tried everything. But in general even key-value stores are designed to store sorted/structured data. Not really massive BLOBs with a little meta data sprinkled in to explain it. You just run into serious performance issues when you try and store/access >2 GB of "junk" data via some clever database management engine, which is really more accustomed to storing tiny little columns of explain-able data you can write clever queries to. We've wound up in a situation now where we just use databases to store the meta-data and have left the actual data in files, which from a programming point of view makes things harder (since you're hooking into two methods of data access as opposed to one).
- icebraining 13y agoWell, PostgreSQL supports the Large Objects, which are designed to store big amounts of data (outside the normal tables) and reference them using special IDs. It may work for your use case.
- reefab 13y agoI've used CouchDB's attachment feature for that purpose: http://onabai.wordpress.com/2012/05/08/working-with-attachments/ http://onabai.wordpress.com/2012/05/08/working-with-attachme...