3 ms·
Yes, Litestream uses LZ4 compression. I originally used gzip but the compression speed was pretty slow. B-tree pages tend compress well because they tend to be
by benbjohnson 6y ago
Yes, Litestream uses LZ4 compression. I originally used gzip but the compression speed was pretty slow. B-tree pages tend compress well because they tend to be 50-75% full because they need space to insert new records and because pages split when they get full.
I'm seeing files shrink down to 14% of their size (1.7MB WAL compressed to 264KB). However, your exact compression will vary depending on your data.
- hinkley 6y agoAh, that makes sense. Most inserts don't split pages, so are around n worst case pages, but once in a while you get 2n updates where most of them are half full, and so compress better. So how does that compare to logical replication? (Also I imagine packet size plays a role, since you have to flush the stream quite frequently, right? 1000 bytes isn't much more expensive than 431)
- benbjohnson 6y agoLitestream defaults to flushing out to S3 every 10 seconds but that's mainly because of PUT costs. Each request costs $0.00005 so it costs about $1.30 per month. If you flushed every second then it'd cost you $13/month. Logical replication would have significantly smaller sizes although the size cost isn't a huge deal on S3. Data transfer in to S3 is free and so are DELETE requests. The data only stays on S3 for as long as your Litestream retention specifies. So if you're retaining for a day then you're just keeping one day's worth of WAL changes on the S3 at any given time.