3 ms·
One upshot of compressing outside s3 is that you save on data transfer. In theory I suppose the compressed data could be sent over to the client and decompresse
by ericpauley 4y ago
One upshot of compressing outside s3 is that you save on data transfer. In theory I suppose the compressed data could be sent over to the client and decompressed but then it’s basically the same thing.
- skrtskrt 4y agoit’s also a conflation of two things that are each much more sustainable if kept independent. You don’t want a particular compression pattern becoming part of the S3 standard IMO
- fnordpiglet 4y agoYeah I’ve suggested to the s3 leadery folks many times to provide some sort of managed compression and other sorts of codec interfaces. They could do other things like convert between structured encodings as well. Unfortunately it’s never gone anywhere.
- coredog64 4y agoIsn’t that what S3 object lambda access points are for? https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-create.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-c...
- fnordpiglet 4y agoIt’s too expensive. The thing is s3 has a ton of compute sitting around. The disks have a full system behind them on the storage nodes. A big question always was how can we do more with the edge storage? S3 select was one effort (although I don’t know if ever got pushed to the edge, it originally ran in the front end servers). But I always figured compression was a real obvious s3 edge storage operation, along with as I said format conversions.