18 ms·
AWS switch from gzip to zstd – about 30% reduction in compressed S3 storage
- charcircuit 4y agoI think he means a 30% reduction in used storage and not that 30% of objects were no longer being stored. It would be a reduction in data, but not compressed data.
- jl6 4y agoIs there some nuance in the phrase “compressed storage” here? Like, is that a subset of overall storage, with some objects not being compressed? I imagine that a great proportion of S3 objects are not amenable to zstd compression, either because they are natively in a compressed format (e.g. JPEG) or because the customer has already compressed them to realise the saving themselves.
- badpun 4y agoCertainly, if used as a data store for a data-lake-like solution, it's much more convenient to store data in uncompressed form, knowing that the underlying storage solution will take care of that for you. This way, you can just dump Json files in without it feeling like a collosal waste (or having to do extra work of converting them to a compressed format, such as Avro).
- paranoidrobot 4y agoStorage level compression usually benefits whomever owns the storage. I doubt Amazon intends to bill on compressed size.
- infogulch 4y agoI believe zstd can detect incompressible data quickly and falls back to a low-overhead passthrough mode in that case. So you can just run everything through zstd and it handles those cases pretty optimally already.
- le-mark 4y agoHmm this made me chuckle, is that not equivalent to a test for randomness, which is very very hard indeed?
- Cyberdog 4y agoI think a useful test for this case could be pretty simple. Compress maybe 10% of the file (up to some reasonable maximum, like maybe a few megabytes) and check the compression ratio. If it's not something better than maybe 1.1x, give up, throw out the compressed data, and just pass through the file. Otherwise keep going.
- hexxagone 4y agoIt is not. Just look at the frequencies in the data (histogram) and compute the order 0 entropy. Then you know if your data is compressible.
- hyuijk 4y agoIf the customers stored .gz/.zip/... files you could transparently transcompress them to .zstd and back (with an added test of course)
- sgtnoodle 4y agoIt seems like that would cause problems. Right off the bat, any integrity hashes like md5 or sha256 for the original compressed files would likely be corrupted. Also, the compressed archive could have been structurally baked in a specific way that's meaningful to the customer. Zip archives in particular can have arbitrary data pretended to them. I suppose you could speculatively decompress and then re-compress and see if you get the original compressed file back, and maybe most people happen to use the same compression implementations with default settings.
- mike_hock 4y agoThat wastes a lot of CPU compared to just running it through zstd.
- spockz 4y agoYou would only apply the compression on the internally stored file and then decompress when retrieving it for the customer again. That way all the hashes and original structure of the user are retained.
- klauspost 4y agoYou would need to be able to reconstruct the input file bit-by-bit. S3 Clients expect to get back what they sent, exactly. This puts a serious limitation on your compression. You would only be able to re-do the entropy coding part of DEFLATE, which is actually pretty good. You would still need to store the original Huffman tables for each block, so you can reconstruct the entropy coding exactly. I doubt this would even gain you a single percentage.
- mkup 4y agoBesides ZIP metadata, there may be flushes in specific points in the deflate stream (which applies to .gz files as well). These flushes reset the compression dictionary and make further compressed data independent from previous data (at the expense of losing some compression efficiency). So: AWS S3 customer may have injected these flushes to their .gz files (for whatever reason, e.g. steganography), and after gzip-to-zstd-to-gzip transcompression this steganographic data will be be lost (and of course sha256 and other similar hashes will be different, as you already said).
- scrame 4y agoIt's written through a virtual file system which stores in blocks and not files, the implementation of that could happily run zstd on the way to the multiple physical disks. This isn't about them shrinking your individual files, its compression on their internal transport.
- ksec 4y agoAt the scale of AWS S3, exabyte scale, I assume that could be close to a billion dollar in cost savings?
- jankeymeulen 4y agoAmazon will pay perhaps $50 for a 10TB disk, 10x it to cover for redundancy and the servers and datacenters to put the disks in, and you're looking at "only" 16 mil savings for an exabyte.
- social_quotient 4y agoAnyone by chance know how many exabytes s3 is? Did some searches but came up with some guesses like https://maisonbisson.com/post/how-big-is-s3/ https://maisonbisson.com/post/how-big-is-s3/ but they are dated and still just guesses.
- ksec 4y agoLet's say a 100 exabyte, which is only 100 times larger than Backblaze. And Amazon pay $100 per 10TB, it is unrealistically low, but just to round down the number nicely . ( It is not $50 for 10TB, part of the reason why HN dont understand Supply Chains and BOM cost. ) On pure storage that is 100,000,000TB. That is 1 billion dollar already excluding any redundancy required by S3.
- IMSAI8080 4y agoThere's a balance between compression ratio and CPU utilisation. Zstd is several times faster at compressing than gzip to produce a file of similar size (it's good, give it a try if you haven't already). I guess he means by using zstd they were able to crank up the compression ratio and maintain the same CPU usage maybe? I don't think AWS routinely compress customer data that I've noticed. I guess he must mean for their internal products that use S3 perhaps?
- notimetorelax 4y agoI doubt there’s a single byte stored to disk that is not compressed and encrypted at AWS. It’s transparent to the customer.
- deleted 4y ago[deleted]
- smueller1234 4y agoIt's not quite that simple. If you have customer data that's already encrypted, then compression won't do much because it looks random. But of course by the time you get to your infrastructure layers, that'll be the case (or you really messed up your security story!). Which means you'd have to compress right at the edge. They might be doing that (which would basically mean it's the customer compressing it before they encrypt it with their keys because AWS has no business seeing the clear text), but then you get to compress each item separately, which might not be very effective for small values. tl;dr:There's a real efficiency/security/insider risk trade-off here. Edit: I should disclose that I work for a competitor. Don't intend any astroturfing.
- notimetorelax 4y agoI agree with you, there could be scenarios where customers supply their own keys and compress the data on their own. My original statement is still true though, the data at rest ends up being compressed and encrypted. That said, of course, customers can upload encrypted blobs of uncompressed data. But I’d call it an exception that proves the rule. Here service simplicity should win and those blobs may end up recompressed.
- seydor 4y agoWhich of course will not be passed to the consumer
- deleted 4y ago[deleted]
- booi 4y agothey should at least etch our names into the yacht that the savings will buy
- cloudymeatballs 4y agoAs a cloud employee I am willing to make the public commitment that should I ever be in the position of buying a yacht, I will etch "booi" into the hull for you.
- NavinF 4y agoWell yeah. S3 is ~$23/TB/month vs $10/TB HDD NRC + $0.20/TB/month electricity for a data hoarder NAS. I can’t imagine S3 users are very price sensitive. Most of them store <1PB. Of course there are cheaper cloud services like B2 at $5/TB/mo, but even that can’t compete on price with physical infra when you have more than a handful of drives.
- Dylan16807 4y agoNot that it changes the math much, but: $15/TB is hard enough for a home buyer to reach for hard drives, excluding server cost. Where did you get $10?
- cmeacham98 4y agoNot them, but searching for "8TB HDD" on Amazon (huh, ironic) shows several options for $90, which is almost $10/TB. And this was 60 seconds of searching, no real effort put in.
- NavinF 4y ago
- tester756 4y agoIt uses some black magic wizardy >FSE, short for Finite State Entropy, is an entropy codec based on ANS. FSE encoding/decoding involves a state that is carried over between symbols, so decoding must be done in the opposite direction as encoding. Therefore, all FSE bitstreams are read from end to beginning. Note that the order of the bits in the stream is not reversed, we just read the elements in the reverse order they are written. https://en.wikipedia.org/wiki/Asymmetric_numeral_systems https://en.wikipedia.org/wiki/Asymmetric_numeral_systems
- Dylan16807 4y agoIt's not that crazy if you separate it into two parts. A) Use arithmetic coding to encode symbols as fractions instead of using an integer number of bits for each symbol. https://en.wikipedia.org/wiki/Arithmetic_coding https://en.wikipedia.org/wiki/Arithmetic_coding http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE523_files/Ch_04%20Arithmetic%20Coding%20%28PPT%29.pdf http://www.ws.binghamton.edu/fowler/fowler%20personal%20page... B) Shuffle the bits around so that you can use faster math and lookup tables instead of directly operating on fractions.
- shargs 4y agoFor what it's worth, shameless plug, I've written a whole series on how FSE works: https://shargs.github.io/data-compression/2022/07/28/entropy9.html https://shargs.github.io/data-compression/2022/07/28/entropy...
- renewiltord 4y agoThe only way to get the savings yourself is to zstd yourself in the first place, which is what we do.
- fourfourthree 4y agoI'd be really curious to hear the security discussion around S3 compression, or if they added any additional mitigations. Compressing data and then encrypting it will be vulnerable to a padding oracle attack[1]. [1] https://en.m.wikipedia.org/wiki/CRIME https://en.m.wikipedia.org/wiki/CRIME
- londons_explore 4y agoI want to know why a company as big as Amazon was still using gzip in 2022.... Sure - it's a handy tool. But at scale surely someone had enough time to research better options? And at Amazon scale, it probably even pays to hire a team to write a custom compression algorithm perfectly tuned to your compute and storage?
- prima-facie 4y agoA parallel implementation like pigz is still pretty good and widely backwards compatible.
- bick_nyers 4y agoI'm surprised they aren't using an FPGA solution
- hbbio 4y agoKnowing such implementation exists, is open source, and since they don't state what they're using... you may just be right https://pure.tudelft.nl/ws/portalfiles/portal/94907930/Jianyu_zstd_compression.pdf https://pure.tudelft.nl/ws/portalfiles/portal/94907930/Jiany...
- pclmulqdq 4y agoFPGAs are really expensive in volume compared to other options, so it's not surprising to me.
- londons_explore 4y agoBut on an FPGA you can probably implement super fast and effective compression. If it is far enough above the curve of speed Vs ratio, then it will pay for itself in saved storage. Remember there is no need for said FPGA to be in every machine - in a data center with ten's of gigabits of bandwidth to every node, you can send data to another machine for compression and receive back the compressed data to store.
- swaranga 4y ago
- tamsaraas 4y agozstd not good for everything. Yes, it's fast in terms of compression. Easy & fast you can make compressed archive. But if you need to use the zstd on daily basis - i have bad news. ZSTD not suitable for general purpose archiver. Depends on the data - you can have totally different output compressed archive if compare to winrar for example. Example: I doing backups time to time, and archive important files into compressed archive containers (rar / 7zstd). I've noticed, that my dev folder with tons of repositories, images, and different work related fines - vary damn too much. /dev/ size = ~11GB rar output (normal compression) = ~2.1 GB zstd archive output (normal compression) = ~4.7GB Why? linked files, same files not treated as a 1 file + links to these files. Instead these files compressed each 1 by 1 instead of copy 1 identical, and compress the file. And many things like that. Suggestion for 7z-zstd -> add ability to save links to files, not treat them as separate files, and adding an option like in winrar to search for identical files first and re-link all of them and remove duplicates, instead of compression each.
- CodesInChaos 4y agoYour criticisms seem to be about 7z, not zstd. Ztd doesn't have an archive concept, it's just compresses a sequence of bytes.
- tamsaraas 4y agoi'm totally correct in what am i writting. zstd not wide range suitable compression. And all current implementation that used non for 1 file compression - awful.
- lifthrasiir 4y agoZstandard doesn't accept multiple files at all, so it's an archiving format's job to convert files into byte sequences and compress them accordingly. It looks like that 7z wasn't able to deduplicate entirely or partially identical files, in the other words zstd could never know that it is compressing almost identical files over and over.
- e63f67dd-065b 4y ago
- kukx 4y agoDoes it mean that gzip is a legacy compression than should be replaced with zstd where possible?
- wolf550e 4y agoYes, if you control both sides, always use zstd instead of deflate/gzip/zip.
- wongarsu 4y agoYes. Gzip used to be a decent middle ground when you needed neither super fast compression/decompression nor super high compression ratio, but zstd beats it by a good margin over the whole range from gzip -1 to gzip -9, while offering you a wider range of compression ratios to choose from. And at settings that provide compression ratios comparable to gzip it's blazingly fast. gzip will stay with us because of ubiquity, just as we mostly use image formats from the dot-com era. But if you control both sides, zstd is a big upgrade.
- Beltalowda 4y ago> the whole range from gzip -1 to gzip -9 gzip compression levels are almost useless and typically result in very little actual compression ratio differences, but typically with massive CPU usage differences. zstd doesn't do everything better than gzip though; I believe gzip still offers somewhat faster compression for equal compression ratios last time I looked (but decompression is much faster). I mostly replaced gzip with zstd myself, but there are still scenarios where gzip might be preferable.
- linuxhansl 4y agoIn all the testing I have been doing across a variety of use cases, from BigData, to databases, to games, I have not come across a scenario where Zstd is doing worse than Zstd. Compression time CPU is lower for equivalent ratios. Maximum compression is better. Decompression CPU is much lower (i.e. faster), and that is independent of the level used at compression time.
- markdog12 4y agoGood article on zstd: https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/ https://gregoryszorc.com/blog/2017/03/07/better-compression-...
- scarface74 4y agoStandard Disclaimer: I work at AWS. Not only are all opinion my own. I’m so far away removed from the service teams, that I might as well be a just like any other customer - I work in the consulting department. That being said, looking at things from the outside, I tend to think that objects are not stored in a compressed form. It would make many operations much slower. 1. There are APIs to fetch byte ranges from objects. Would that work without decompressing the entire object? https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing... 2. You can query over multiple objects using SQL either via Athena or S3 Select. https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-reference.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-gla... Edit: Someone has already explained how byte range requests would work.
- bhedgeoser 4y ago> Someone has already explained how byte range requests would work. Where?
- xtreak29 4y agoProbably a reference to https://news.ycombinator.com/item?id=32531141 https://news.ycombinator.com/item?id=32531141
- latchkey 4y ago> There are APIs to fetch byte ranges from objects. Would that work without decompressing the entire object? I wonder if you could store the objects in chunks and then individually compressed. Create an index of the byte ranges for those chunked objects so they can be looked up easily. For example: range 0-4000 -> objectA, objectB, objectC Then just return those 3 objects, decompressed.
- havblue 4y agoQuick question, for personal file storage, such as family photos, would zstd be preferable now to, say, tar.gz?
- kevincox 4y agoPhotos generally won't get much compression anyways since the image data is already compressed. However in general yes, zstd is almost always preferable to gzip these days.
- antisthenes 4y agoFor archiving RAW files, yes. For RAW --> lossy format, storing them in JPEG-XL would be preferable, and I doubt most compression algos would do better than JPEG-XL.
- 404mm 4y agoRandom shallow rant, I really wish they came up with a different abbreviation than zstd. Not only it sounds strange but it’s also too long for abbreviation, imo. It could have been “zs” or “zd”.
- hdante 4y agohttps://github.com/facebook/zstd/issues/1333 https://github.com/facebook/zstd/issues/1333
- wging 4y agoAs a semi-educated guess, I wonder if he might just be referring to the compression of log files via standard Amazon-internal mechanisms, which use S3 as the backing storage, for many services.
- funstuff007 4y agoNot be that guy, but we never send an object to S3 without compressing it on our end first.
- flatiron 4y agoWe never send anything unencrypted to S3. I have no clue why someone would not want to encrypt their data before putting it in the cloud…
- cmeacham98 4y agoFor people also using other AWS services (in particular compute ones like EC2) this doesn't do anything meaningful.
- flatiron 4y agoWe use other aws services. We decrypt the files on those services.
- thecleaner 4y agoBecause there is already server side encryption. Which I guess means data on disk is always encrypted not sure about in-memory.
- alrlroipsp 4y agoBut what about the % increase in power consumption?
- akmittal 4y agoZstd is very efficient in decompression https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/ https://engineering.fb.com/2016/08/31/core-data/smaller-and-...
- andrewmcwatters 4y agoNot too long ago I started compressing statistics data I was collecting from GitHub[1] with ZStandard after noticing that GitHub Actions CI caching used it. It’s really remarkable for my use case. I compress gigabyte-sized SQLite databases under the GitHub 100 MB file limit since Git LFS is a non-starter. My tables are collections of follower and stargazer counts to measure user and repository popularity and understand where one might be on a distribution compared to a subset of other GitHub data points. [1]: https://github.com/andrewmcwattersandco/github-statistics https://github.com/andrewmcwattersandco/github-statistics
- _0w8t 4y agogzip has rsync-friendly mode allowing for much smaller deltas between files during rsync transfer or backups that use rsync-compatible algorithms. No other compressor implements that as far as I know.
- dudeinjapan 4y agoEverything counts in large amounts.
- togs 4y agoZstd, it’s okay. 600 .rvz files = 450 Gb -> 163 GB. Wanted it more like 10 GB.
- abotsis 4y agoBut they still only support gzip or bzip2 for S3 Select’ing json files :(
- klauspost 4y agoIf relevant for you, we have an unadvertised feature in MinIO, that adds Zstandard, LZ4, Snappy support: https://github.com/minio/minio/blob/master/internal/s3select/select.go#L67 https://github.com/minio/minio/blob/master/internal/s3select...
- obikeyz 4y ago
- vbezhenar 4y agoNo support in CompressionStream, that's too bad.
- deleted 4y ago[deleted]
- amzn-throw 4y agoEveryone's interpreting this as AWS is zipping every customer's S3 data and improved storage of their data with ztsd. That's now what Adrian said, and this is not an announcement - this was a "by-the-way" from a retired Amazonian. AWS switched our own service's log storage (in S3) from gzip to ztsd a bunch of years ago -and reduced storage costs by 30%. This is AWS's service data (which is still exabytes) and AWS realized the efficiency. So all the comments of "where is my 30% discount" are offbase. Source: I've worked on AWS for a bunch of years.
- banana-19 4y agoI'm guessing he's NOT talking about S3's user data. He's talking about runtime logs of the S3 service, or possibly logs from Amazon's services that are written to S3. Amazon has ~100K services running on millions of EC2 instances. (Generalizing) each service writes logs to the local machine, and then later compresses and transmits them to a set of systems that are backed by S3 for monitoring / further bulk processing etc. Some back of the envelope math on this: For 1 exabyte to be the number, that equates to < 1TB average logs kept per host. Depending on what the service is doing log retention is 3mo / 1y / 10y. Using those numbers we're talking per service average 12-460 MB/hour.
- DedlySnek 4y agoYep, someone from aws clarified this on r/programming: https://old.reddit.com/r/programming/comments/wtd61q/aws_switch_from_gzip_to_zstd_about_30_reduction/il4uu67/ https://old.reddit.com/r/programming/comments/wtd61q/aws_swi...