3 ms·
Have you compared LZ4 to other compression algorithms, zstd for example? ( https://github.com/phiresky/sqlite-zstd https://github.com/phiresky/sqlite-zstd ) Gi
by infogulch 4y ago
Have you compared LZ4 to other compression algorithms, zstd for example? ( https://github.com/phiresky/sqlite-zstd https://github.com/phiresky/sqlite-zstd )
Given that LiteFS operates at the filesystem layer via FUSE, have you considered it against designs that use built-in features native to some filesystems? For example, I've considered a similar system design based on a single ZFS primary node that streams ZFS snapshots to reader nodes. With some coordination service (e.g. consul) it could still allow for the whole node promotion process.
- benbjohnson 4y ago> Have you compared LZ4 to other compression algorithms, zstd for example? I looked at zstd as well. IIRC, LZ4 compresses faster and zstd has a better compression ratio. A lot of the files being compressed are short-lived so it's typically better to compress faster with LiteFS. Maybe we'll support more compression formats in the future if it's useful. > have you considered it against designs that use built-in features native to some filesystems The ZFS idea is interesting. I targeted FUSE because it seemed like a lower barrier to entry and easier for users to get set up.