4 ms·
Wouldn’t it be better to keep such files inside a database?
by andix 5y ago
Wouldn’t it be better to keep such files inside a database?
- marginalia_nu 5y agoA filesystem is a database.
- amelius 5y agoSo what people think of as a database is a database inside a database?
- samus 5y agoIndeed, some database servers (Oracle IIRC) don't use a filesystem, but write to a filesystem partition directly.
- marginalia_nu 5y agoYeah. It's like that old Xzibit meme. File systems aren't relational databases, but more like NoSQL half a lifetime before it was cool. They're typically backed by the same data structures and solve the same problems.
- andix 5y agoBut a special kind of database, that is usually bad with a lot of small entries, especially if they don't have a hierarchy (many files in one folder). Most SQL or NoSQL databases are much better at organizing small "files".
- marginalia_nu 5y agoThe difficult use case isn't many files in a folder, it's just many files period.
- lillecarl 5y agohttps://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html They're selling it quite well here, though integration into other products might be the hard thing.
- hughrr 5y agoAdd https://github.com/guardianproject/libsqlfs https://github.com/guardianproject/libsqlfs
- pmarreck 5y agoI literally discovered this just the other day, the whole concept is extremely intriguing but it hasn't been updated in some time (I'm guessing some forks have been though). Relatedly, it would be pretty cool to be able to build your own filesystem features as layers on each other (such as block-level compression or encryption, block-level forward-error correction, cache promote, striping or other redundancy etc.). A modular FS with pluggable (but also upgradable) components and some sort of structured feature-flag schema metadata, and a way to migrate between variations (such as switching to a different compression or encryption algorithm).
- greggyb 5y agoSome of what you're describing is in ZFS. E.g., it has multiple compression algorithms; you can create many datasets -- each of these is either a ZFS filesystem or a volume which can have another FS deployed to it, but with compression and encryption handled by the "host" ZFS. You may also be interested in FreeBSD's GEOM subsystem for storage.
- tych0 5y agoIn my experience, integration with products is not really that hard. I've used sqlite3 bindings in half a dozen languages over the years, and they are generally fairly robust.