3 ms·
One reason this could be messy to add is that the S3 API includes support for reading arbitrary byte ranges within a file. There's an inherent conflict between
by dantillberg 4y ago
One reason this could be messy to add is that the S3 API includes support for reading arbitrary byte ranges within a file. There's an inherent conflict between that API and support for transparent compression. Compression performs best when compressing the entire file from beginning to end; when decompressing you'd similarly decompress from the beginning. To support byte range reads, the data could be compressed in short blocks, but those blocks will not compress as well.