3 ms·
We have a delta-encoded binary format for trade logs that we use and zstd still does a pretty good job on them for the last bit. It's just a pity that storage
by renewiltord 4y ago
We have a delta-encoded binary format for trade logs that we use and zstd still does a pretty good job on them for the last bit.
It's just a pity that storage providers won't implement transparent compression. We store our logs on S3 and on-prem and it's just annoying that we have to store them compressed on S3. Would be cool to just store without the compression stage and have S3 do that bit.
I haven't tried btrfs with the transparent zstd compression, but perhaps worth a shot.
- deleted 4y ago[deleted]
- ericpauley 4y agoOne upshot of compressing outside s3 is that you save on data transfer. In theory I suppose the compressed data could be sent over to the client and decompressed but then it’s basically the same thing.
- skrtskrt 4y agoit’s also a conflation of two things that are each much more sustainable if kept independent. You don’t want a particular compression pattern becoming part of the S3 standard IMO
- fnordpiglet 4y agoYeah I’ve suggested to the s3 leadery folks many times to provide some sort of managed compression and other sorts of codec interfaces. They could do other things like convert between structured encodings as well. Unfortunately it’s never gone anywhere.
- coredog64 4y agoIsn’t that what S3 object lambda access points are for? https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-create.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-c...
- fnordpiglet 4y agoIt’s too expensive. The thing is s3 has a ton of compute sitting around. The disks have a full system behind them on the storage nodes. A big question always was how can we do more with the edge storage? S3 select was one effort (although I don’t know if ever got pushed to the edge, it originally ran in the front end servers). But I always figured compression was a real obvious s3 edge storage operation, along with as I said format conversions.
- abotsis 4y agoI still wish s3 object select supported zstd on json.
- acdha 4y agoCould you use a Lambda to compress with zstd on write and S3 Object Lambda to decompress the stream seamlessly on GET (and update the content length on HEAD)?
- dantillberg 4y agoOne reason this could be messy to add is that the S3 API includes support for reading arbitrary byte ranges within a file. There's an inherent conflict between that API and support for transparent compression. Compression performs best when compressing the entire file from beginning to end; when decompressing you'd similarly decompress from the beginning. To support byte range reads, the data could be compressed in short blocks, but those blocks will not compress as well.
- 323 4y agozstd is quite slow at higher compression levels, if S3 would compress who pays the compute? Maybe they could offer it as an extra, and spawn EC2 instances as needed and bill you.