3 ms·
Standard Disclaimer: I work at AWS. Not only are all opinion my own. I’m so far away removed from the service teams, that I might as well be a just like any oth
by scarface74 4y ago
Standard Disclaimer: I work at AWS. Not only are all opinion my own. I’m so far away removed from the service teams, that I might as well be a just like any other customer - I work in the consulting department.
That being said, looking at things from the outside, I tend to think that objects are not stored in a compressed form. It would make many operations much slower.
1. There are APIs to fetch byte ranges from objects. Would that work without decompressing the entire object?
https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing...
2. You can query over multiple objects using SQL either via Athena or S3 Select.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-reference.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-gla...
Edit:
Someone has already explained how byte range requests would work.
- bhedgeoser 4y ago> Someone has already explained how byte range requests would work. Where?
- xtreak29 4y agoProbably a reference to https://news.ycombinator.com/item?id=32531141 https://news.ycombinator.com/item?id=32531141
- latchkey 4y ago> There are APIs to fetch byte ranges from objects. Would that work without decompressing the entire object? I wonder if you could store the objects in chunks and then individually compressed. Create an index of the byte ranges for those chunked objects so they can be looked up easily. For example: range 0-4000 -> objectA, objectB, objectC Then just return those 3 objects, decompressed.