9 ms·
They could be using hardware compression which can be orders of magnitude faster than doing it on the CPU. Hardware compression is sadly not widely available,
by erk__ 4y ago
They could be using hardware compression which can be orders of magnitude faster than doing it on the CPU.
Hardware compression is sadly not widely available, I think the only consumer product I know with it is the PlayStation 5.
The mainframes from IBM have had hardware zlib since Z14 iirc and in my small tests it is very fast compared to the CPU implementation
- sigmoid10 4y agoI think a lot if datacenter SSDs already come with in-drive hardware compression these days, since it not only increases speed but also longevity. So it would actually save money anyways.
- snoopy_telex 4y agoThey do not. It would be difficult to plan correctly if your free disk space is… variable. Example: You have an existing 40 gigabyte file It happened to compress well You delete it and your free disk space goes up by 4 gigabytes. You then write a new 40 gigabyte file that doesn’t compress well Replacing an existing file of the same size just ate an extra 36 gigabytes. How would you plan around that? SSDs should store the bytes given and don’t play fancy games.
- anamexis 4y agoYes they do. https://www.intel.com/content/www/us/en/support/articles/000006354/memory-and-storage.html https://www.intel.com/content/www/us/en/support/articles/000...
- wmf 4y agoNote that these are pre-2017 consumer SSDs. I think SSD compression fell out of favor due to the rise of FDE.
- ChrisLomont 4y agoFDE can be done on the SSD after compression, and often is. Adding life to SSDs is a terribly useful feature
- ChrisLomont 4y agoSSDs could still do it for speed and to write less pages, making the drive last longer, and simply report the uncompressed space as used. They already do all sorts of tricks on pages such as moving them logically, having more internally than they report to use as pages wear out, and so on. Given that, it's good sense to compress if at all possible simply to make the drive live longer. And guess what - I just googled, tons of hits, and this has been done for a long time :) So it makes sense, is done, and is important for modern SSD behavior.
- tgsovlerkhgsel 4y agoNone of this works if the data is encrypted. I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough to make such optimizations worth it. Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal.
- ChrisLomont 4y agoPlenty of SSDs implement strong encryption, and there's an entire standard API for it. Why waste cpu on tasks your hardware already does? Bitlocker supports on drive hardware encryption, and I'd be surprised if other major file systems didn't. If I recall, it's a FIPS requirement for data at rest now.
- tgsovlerkhgsel 4y ago> Why waste cpu on tasks your hardware already does? Because the hardware cannot be trusted to do it correctly. IIRC Bitlocker stopped relying on it for this reason. https://www.howtogeek.com/fyi/you-cant-trust-bitlocker-to-encrypt-your-ssd-on-windows-10/ https://www.howtogeek.com/fyi/you-cant-trust-bitlocker-to-en... (see the updates)
- gspr 4y ago> Why waste cpu on tasks your hardware already does? Because the storage hardware might be loyal to someone who is not the owner of the data.
- sp332 4y agoIt's not great, but it's also not unheard of. Tape capacities are often quoted at double the actual storage space, with fine print that says "assuming 50% compression". Also, if compression makes IOPs faster or reduces wear on SSDs, people might not complain so much.
- natmaka 4y agoIsn't it solved by adding an abstraction layer between the storage and the filesystem, the way "vdo" ( https://github.com/dm-vdo/vdo https://github.com/dm-vdo/vdo ) does it?
- collegeburner 4y agohow is this any different than e.g. fs level compression with zstd on btrfs? seems enough people find that useful.
- yunohn 4y agoWhat do you mean? Even without compression, you have half the problem. Obviously, at scale, it’s all just statistics and planning on trends instead of individual files.
- sigmoid10 4y agoYes they do. Intel writes this for example (not all drives use it, but some definitely do): >Data compression via encoding algorithms enables a solid state drive (SSD) to write less data, which in turn yields higher write bandwidth. With a significant amount of data being compressible, performance benefits can be substantial. https://www.intel.com/content/www/us/en/support/articles/000006354/memory-and-storage.html https://www.intel.com/content/www/us/en/support/articles/000...
- sexy_panda 4y agoI wonder if this could be done on a FPGA..
- Frost1x 4y agoWith enough gates you can do anything on an FPGA
- vasco 4y agoFPGA for development, AWS can afford to order custom ICs.
- toomuchtodo 4y agoCould surplus crypto miner GPUs be repurposed for this?
- vasco 4y agoNo
- pclmulqdq 4y agoHow much would someone pay for this? I have a half-written zstd core, but I doubt the market for $100-150 FPGA-based compression accelerators is all that large.
- tgsovlerkhgsel 4y agoProbably not worth it as a FPGA solution or even in general as an add-on card (the overhead of dealing with such extra hardware means that the threshold for "worth it" is very high). I would expect this to become part of newer generations of CPUs once it becomes popular.
- wmf 4y agoIntel sells a gzip PCIe card called QAT. Not many seem to be sold.
- estebarb 4y agoHardware compression IS available in Graviton 2: "1Tbit/s of compression accelerators • 2xlarge and larger instances will have a compression device • DPDK and Linux kernel drivers will be available ahead of GA • Data compression at up to 15GB/s and decompression at up to 11GB/"