3 ms·
While generally good advice, sometimes it does make sense to put binary images in the database: At Loopt, long before maps were included on mobile platforms, w
by kogir 15y ago
While generally good advice, sometimes it does make sense to put binary images in the database:
At Loopt, long before maps were included on mobile platforms, we served our own map tiles. These 64x64 pixel PNGs were often less than 100 bytes. Even though we only covered the US at the time, there were upwards of 20 million of these tiny files.
They caused file system destruction. We tried ext3, reiserfs, ufs, ntfs - they all choked. In the best of cases each file took as little as 512 bytes on disk, 5x larger than the data within. Performance was awful, no matter how we bucketed the files in directories or tuned the file systems. Reiser 4 trunk was looking promising, but then Hans was arrested[1].
The solution was to store the images in a database. They fit in-row without needing blob storage, and the DB indexes were much faster than any of the file systems we tried. Backups took less space and less time.
[1] He was eventually convicted of killing his wife: http://en.wikipedia.org/wiki/Hans_Reiser http://en.wikipedia.org/wiki/Hans_Reiser
- zokier 15y agoFor one application I was designing, I planned to use Berkeley DB (key-value database) as a content-addressable data-store, ie. hash the binary blobs and use the hash as key. Then the hash would be stored at the application (SQL) database. For very small files (such as your <100 byte PNGs) it probably wouldn't be worth it, but for larger blobs it be used to avoid the 'millions of files' problem. I don't know if it would have worked, but I though it was a cute idea.
- getsat 15y agoXFS rules at handling lots of little files. It also has a ridiculously high "files per directory" limit. Something like Redis might've been sweet, too, but would require you to hold all that data in ram.
- kogir 15y agoThis was in 2005. Should have tried XFS if we didn't. I think we might have.