4 ms·

I swear to god, I don't understand why people don't think about this. total sadness about having to deal with a 150GB SQL database, of which 148 GBs are blob s

I swear to god, I don't understand why people don't think about this. total sadness about having to deal with a 150GB SQL database, of which 148 GBs are blob storage for PDFs "We have big data!! We need enterprise scale!" Nope. You have big stupid.

coldtea 9y ago
I don't see the sadness -- or the stupid. There are several very valid business cases to store files as blobs in a DB. What's the problem is the DB is 150GB? It's not like the working set (which for the files will just be the metadata) will be that big for storing file blobs. 10GB Database + 140GB of pdfs on the filesystem are not any different to a 150GB DB with everything in. And you have other issues (consistency, transactional issues, backups, etc).
- sheeshkebab 9y ago
  How about a 100TB database with 95TB being pdf's? It seems there are better ways of managing lots of immutable small files (including for backups/replication) than shoving them all into a database.
  - coldtea 9y ago
    >How about a 100TB database with 95TB being pdf's? How about it? >It seems there are better ways of managing lots of immutable small files (including for backups/replication) than shoving them all into a database. Depends on the business case. A database gives certain guarantees you'll have to replicate (usually badly) in any other way. Conceptually, it's absolutely cleaner. As for from a scientific or engineering standpoint, there are again no laws that dictate saying whether this is bad or good. Even the performance characteristics depend on the implementation of the particular DB storage engine. They might be totally on par with storing in the filesystem (or close enough not to matter). Not to mention that some filesystems might even have worse overhead depending on the type, size, etc. of files. In fact this very FA speaks of "SQLite small blob storage" being "35% Faster Than the Filesystem". Plus filesystem storage and DBs are not that different in most cases -- they share algorithms for storage and indexing (logstorage, btrees, etc), and the main difference is the access layer. The FreeBSD filesystem, for one, was more like a DB storage layer than a 70s style Unix filesystem.

coldtea 9y ago

I don't see the sadness -- or the stupid. There are several very valid business cases to store files as blobs in a DB. What's the problem is the DB is 150GB? It's not like the working set (which for the files will just be the metadata) will be that big for storing file blobs. 10GB Database + 140GB of pdfs on the filesystem are not any different to a 150GB DB with everything in. And you have other issues (consistency, transactional issues, backups, etc).

sheeshkebab 9y ago
How about a 100TB database with 95TB being pdf's? It seems there are better ways of managing lots of immutable small files (including for backups/replication) than shoving them all into a database.
- coldtea 9y ago
  >How about a 100TB database with 95TB being pdf's? How about it? >It seems there are better ways of managing lots of immutable small files (including for backups/replication) than shoving them all into a database. Depends on the business case. A database gives certain guarantees you'll have to replicate (usually badly) in any other way. Conceptually, it's absolutely cleaner. As for from a scientific or engineering standpoint, there are again no laws that dictate saying whether this is bad or good. Even the performance characteristics depend on the implementation of the particular DB storage engine. They might be totally on par with storing in the filesystem (or close enough not to matter). Not to mention that some filesystems might even have worse overhead depending on the type, size, etc. of files. In fact this very FA speaks of "SQLite small blob storage" being "35% Faster Than the Filesystem". Plus filesystem storage and DBs are not that different in most cases -- they share algorithms for storage and indexing (logstorage, btrees, etc), and the main difference is the access layer. The FreeBSD filesystem, for one, was more like a DB storage layer than a 70s style Unix filesystem.