6 ms·
We could save petabytes of cache storage with Zstandard and Pingora
- deleted 2mo ago[deleted]
- MayeulC 2mo ago> We initially considered limiting transcoding to popular content Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
- genxy 2mo agoI would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients. Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor. One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.
- mort96 2mo agoIt has mostly become my go-to as well. Just wish it wasn't a Facebook product.
- articulatepang 2mo agoI agree! I came to the comment section to say exactly this. In any cache hierarchy you want to put colder content in cheaper but slower storage. Here, compression is the cheaper but slower form of storage.
- nijave 2mo agoThat part is a little bit confusing. I think they probably don't care about storage on the devices that do the compressing and are optimizing for quickly pushing hot content to edge locations. So the compression at the source saves bandwidth during the pushing to edge phase and allows the edges to hold more (reducing churn, further saving bandwidth back to the source). Put a different way, they're trying to make cache evictions cheaper (less bandwidth to refill) and less likely (bigger cache on same disk size) "Ignore cold assets" makes more sense with that framing Although if that's the case, the CPU statement still is a bit confusing.
- donavanm 2mo agoWhen I worked on a large CDN the content popularity distribution was heavily skewed. Think 20-40% of throughput from top 1% if content, and 80-90% from the top 10%. Anything outside of that had a very low probability if ever being read again in the effective cache lifetime. Then the effective cost of scaling CPU > RAM > storage > network due to power & space limitations. Spending extra processing time on 50% of your content would be wasted effort as its never read again. And yes, increasing effective storage might increase cache width/lifetime, but its not by that much to dramatically inprove access rates. Especially when most content by unique bytes is compressed audio/video in the first place.
- im3w1l 2mo agoYou are making an argument against caching it at all which is clearly not what they want. So the comparison must be uncompressed storage vs compressed storage. The compression cost is always the same. The storage cost depends on how long you keep something in cache. The decode cost is proportional to number of total hits. So compression makes the most sense for something you want to keep a long time that will be accessed very rarely. And the least sense for something you will drop very soon and will have many people requesting it.
- donavanm 2mo agoMaybe cloudflares workload is substantially different, but i think youre missing just how long that tail was. A “lot”, maybe half, of unique objects werent requested a second time in any meaningful period time. Like days. And edge nodes have nowhere near the iops or cycles to spend doing _any_ extra work. So yes it is a waste of resources to cache or process in any way.
- CodesInChaos 2mo agoI'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
- pkulak 2mo agoI assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.
- CodesInChaos 2mo agoWhich would have terrible performance for range requests starting late in a large file. For files that are frequently accessed that way, this could be prohibitive. You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking. Or they have an upper size limit for the file size they compress, since large files are rarely compressible text. In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.
- mgerdts 2mo agoFor the cost of a small amount of metadata the offsets of every MiB or so could be stored. I did something like this with pigz as I was implementing multithreaded compressed and encrypted kernel zone suspend and resume for Solaris.
- genxy 2mo agoNot with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.
- thinkindie 2mo agoWhy not serving files compressed if the client supports it even though the origin served an uncompressed file?
- theandrewbailey 2mo agoI was thinking this. Zstd is widely supported in browsers, over 80% right now and will increase over time: https://caniuse.com/?search=zstd https://caniuse.com/?search=zstd
- llama052 2mo agoThey already do this, at least for paid accounts. You can even decide what compression model you want them to serve on your behalf.
- deleted 2mo ago[deleted]
- nijave 2mo agoCurious how this application scheme compares to filesystem + transport compression. You'd end up potentially compressing and decompressing more often but the higher software doesn't need to know what's happening and the compression happens in kernel space. ie btrfs You could also layer on out of band dedupe and probably push out cache updates with btrfs snapshots although maybe that ends too convoluted
- r3trohack3r 2mo agoTangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files. For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks. There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!
- a_t48 2mo agoI've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.
- Twirrim 2mo agoI'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.
- hinkley 2mo agoSome compression libraries have an 'rsync compatibility mode', that plays some games with the block sizes to make it easier to rsync to not have to completely re-transmit a large compressed file because not the whole file changes every time. I've never been entirely sure how it works, whether it only does particular things when clobbering an existing file or does some other heuristic to make it more likely that changing one function in the middle of the archive requires only a small part of it to need to be transferred instead of every byte from that point onward.
- colechristensen 2mo agoYou may or may not be familiar with the content defined chunking family of algorithms like FastCDC. https://joshleeb.com/posts/chunking.html https://joshleeb.com/posts/chunking.html https://www.usenix.org/conference/atc16/technical-sessions/presentation/xia https://www.usenix.org/conference/atc16/technical-sessions/p... On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.
- zahlman 2mo agoIt could save PyPI petabytes per month of bandwidth, too. (But it seems like this is also caused by broken CI systems failing to cache things locally.)
- repsilat 2mo agoYeah. It's worth looking at your own code and infrastructure as well. In my previous job there were dozens of opportunities to realise big savings by swapping out gzip for other compression schemes -- usually zstd, occasionally lz4 or bz2 or brotli. Huge assets took less hard drive space, took less time to download, and took less time to decompress. The differences were not small, and resulted in appreciable improvements both in infra cost and dev productivity.
- nithril 2mo agoOne thing that bothered me enough to comment on: "transcoding" doesn't seem like the right term in this context. "Encode/decode" is technically correct, but "compress/decompress" would have made the intent much clearer.
- walrus01 2mo agoTranscoding is generally used within the context of audio or video codecs, to convert a file from one format to another. And usually from one lossy format to another (eg: not raw uncompressed YUV420, YUV422P video or whatever that is stored in a lossless compression format). It's not clear to me why they're using it in this web page.
- TacticalCoder 2mo agoYeah, the following sentence in TFA is weird: > Transcoding does not mean compressing everything. Images, video, and fonts are usually compressed already. It's because they're already lossily compressed that, precisely, they're the typical targets for transcoding. EDIT: oops, wrote lossless instead of lossy, sorry about that
- walrus01 2mo agoAdditionally modern fonts are normally vector point data, no? So there's plenty of opportunity for shrinking file size by lossless compession. But I don't think that's what they're talking about here?
- londons_explore 2mo agoZstd allows a 'dictionary'. With a dictionary, small objects - even just a couple of bytes - compress well. Compress the headers too, and suddenly it's worth doing
- chickenbig 2mo agoZstandard is an awesome piece of technology. Even the very low compression levels -1, 1 or 2 (IIRC the default of 0 is actually a much higher number) can be very effective, especially for more situations where CPU is a bottleneck. It is particularly effective when you can 'rotate' the data to enhance compressibility; https://github.com/google/riegeli https://github.com/google/riegeli does this automatically for wire format protobufs by splitting data by protobuf field (well, recursively). It doesn't even have to know the message schema. Shame the project is not more widely known.
- NostraDavid 2mo ago> compression levels -1, 1 or 2 Are the available compression levels not 1-22? Though that might depend on the specific library used - the official lib at least uses 1-22: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md#operation-modifiers https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
- andersmurphy 2mo agoThere's negative levels for faster decoding and compression. level -1 uses half the CPU of level 1 and with a custom window size can be really good for streaming compression. I think the negative levels go down to -7 if I remember correctly?
- squirrellous 2mo agoRiegeli is great! Packed full of features. It could be the standard storage format for serialized pb binary if there were more official support.
- quarrykit 2mo ago[dead]
- andersmurphy 2mo agoI'm surprised the faster decompression speed of zstd wasn't mentioned. Feels like a big win for cache (assuming decompress on read). A shame browsers don't support concatenating zstd frames (which zstd does support). Then you wouldn't need to decompress, just serve zstd straight to the browsers after concatenating the frames you care about.
- wood_spirit 2mo agoWait, what? Don’t browsers support concatenated zstd streams? My reading - and I care because I’ve been using concatenation tricks in parquet - is that this is required by the standard. In the distant past I made a jinja2-like template system for server side rendering that precompressed the static bits and injected the short dynamic bits into the output stream as literals with some basic lz match rewriting in the following chunks etc. This kind of trick ought be better with zstd frames.
- andersmurphy 2mo agoSadly they do not. They stop after the first frame (at least last time I tested it). They also don't always support window sizes above 8MB which means levels above 19 can fail to decompress (if they use the more than 8MB of window).
- nchmy 2mo agoIt's extremely surprising me to that something like this - be it with gzip, lz4, or whatever - wasn't implemented at Cloudflare ages ago... And that the task to investigate fell to an intern. These are non-negligible numbers...
- michelb 2mo agoI suspect it just hasn’t been a priority problem in the past. This could very well have been a premature optimization when compared to getting services up and running reliably.