4 ms·
To add to that list: when you do a .tar.gz, the compression algorithm works across file boundaries (very useful when you have a lot of small files), but neither
by OskarS 5y ago
To add to that list: when you do a .tar.gz, the compression algorithm works across file boundaries (very useful when you have a lot of small files), but neither sqlar nor .zip does this.
I think the idea of using SQLite as an application file format is excellent, and it's great that it can serve as a general store of files. As a general purpose compressed archive format, however, I agree it leaves a lot to be desired.
- lifthrasiir 5y agoYeah, I even think a full file system based on SQLite format is possible; pretty much all ingredients are there, just waiting to be cooked. But it should have been cooked in the first place.
- brandmeyer 5y agoI don't think so. An important missing element in SQLite's format is larger block support. Modern filesystems do this via extents. SQLite has a strictly fixed block size and no support for forming larger blocks from contiguous runs of basic blocks. Another missing element is that SQLite's support for large blobs is via linked lists of pages. That's fine for sequential access, but unacceptable for random access. For a database to be usable as a filesystem backend, it must at least be composable with itself without sacrificing performance. Lets say you built a kernel module or fuse module using sqlite as the backend. How well would sqlite itself perform on a file in this filesystem?
- lifthrasiir 5y ago> Another missing element is that SQLite's support for large blobs is via linked lists of pages. This surprised me and you were correct, there is no guarantee that related payload overflow pages are consecutively ordered. I thought extents can be implemented with those pages but it would be pretty bad for random access as is. In fact one of the reasons I thought SQLite can function as a file system was a built-in support of random-accessible BLOBs, and some more research revealed that the support was based on the fact that SQLite loads the entire BLOB into memory. Oops.
- tobias3 5y agoAlready there. E.g. https://github.com/guardianproject/libsqlfs https://github.com/guardianproject/libsqlfs but I'd also count something like s3ql where the data is stored as s3 objects.
- conradev 5y agoThat is a trade off in that you can no longer randomly access a single file without decompressing the entire stream. That is often desired for file archives. For use cases like genomics, though, reads from random offsets are common, hence the bgzf format used by BAM files which is a sequence of gzip chunks and an index. Lower compression ratio, but faster seeking.
- masklinn 5y ago> To add to that list: when you do a .tar.gz, the compression algorithm works across file boundaries (very useful when you have a lot of small files), but neither sqlar nor .zip does this. That's mentioned both in the article (3.2) and GP ("solid compression" is the term of art for a single compression stream for the entire archive).