4 ms·

Compression has a CPU time cost, though. You spend less on storage but use your CPUs more. Is the extra load from compression cause your cluster to autoscale? i

Compression has a CPU time cost, though. You spend less on storage but use your CPUs more. Is the extra load from compression cause your cluster to autoscale? if so, you may not be saving money.

lawrjone 6y ago
Author here! A few others have pointed this out, but to restate: in our situation, compression was costing us almost nothing. 60TB of logs was 60 days worth of retention, so 1TB a day. That means we process about 11MB/s on average, peaking at 100MB/s. A single CPU core can manage 100MB/s of compression, so if you assume we compress and decompress in multiple places, let's say we're paying about 4 CPU cores constantly for this. That's a pretty worse case scenario, and it would cost us $37.50/month on GCP for those cores, in order to save about 100x that amount. The takeaway (for me at least) is that compressing an in-flight transfer is almost always worthwhile if you're working in the cloud and the transfer will eventually be stored somewhere. The economics of CPU vs the total amount of data storage cost is a no brainer. Hope that makes sense!
- 29athrowaway 6y ago
  Thanks for the clarification.
zbjornson 6y ago
FTA: "This post aimed to cover a scenario where the cost of compression, in both compute resource and time-to-build, was significantly outweighed by the savings it would make."
dilatedmind 6y ago
my intuition is you can save cpu time when compressing before sending over the network (and certainly wall time). a quick test copying a 24M file (with similiar compression ratios) to s3 showed a 6% decrease in cpu time when piping through gzip.
- EricBurnett 6y ago
  Depends on selected compressor, but yes, you can. I've definitely observed zstd-1 to be a net savings, where compression/decompression costs were offset by pushing fewer bytes through the RPC and network layers - and this was only from observing the endpoints, not even counting CPU on intermediate proxies/firewalls/etc. I wouldn't normally expect gzip to be a net savings (it's comparatively more expensive), but depending on compression ratio achieved and what layers you're passing the bytes through, I'd definitely believe it can be in some contexts.
- saurik 6y ago
  Data sent to S3 is usually hashed (depending on authentication type) in addition to being transport encrypted; I imagine the majority of this cost here is the encryption of a larger payload (which many would consider indispensable, but I point this out because I do not generally assume this when I merely consider "over the network").
  - user5994461 6y ago
    You assume incorrectly. SSL encryption is in the order of 1 GB/s on a recent CPU with AES instructions (anything from this decade). Gzip is in the order of 10 MB/s with default settings, down to 1 MB/s with the strongest compression setting. It's really really slow.
    - Kranar 6y ago
      GNU gzip the application is slow on the order of 10 MB/s because of how it does file IO, but the DEFLATE algorithm that gzip is based off of is much faster than 10 MB/s at the default "level 6". For example the slz implementation of DEFLATE compresses text at 1 GB/s [1]. Even the fairly common zlib implementation can compress text at close to 300 MB/s. http://www.libslz.org/ http://www.libslz.org/
      - user5994461 6y ago
        DEFLATE at level 6 is really doing 10 MB/s, doesn't matter if you're using gzip or zlib or another library. slz is closer to level 20 (if there was a level 20). It's fast but the compression ratio is meh. You're better of using lz4 or zstd.

Compression has a CPU time cost, though. You spend less on storage but use your CPUs more. Is the extra load from compression cause your cluster to autoscale? if so, you may not be saving money.

lawrjone 6y ago

Author here! A few others have pointed this out, but to restate: in our situation, compression was costing us almost nothing. 60TB of logs was 60 days worth of retention, so 1TB a day. That means we process about 11MB/s on average, peaking at 100MB/s. A single CPU core can manage 100MB/s of compression, so if you assume we compress and decompress in multiple places, let's say we're paying about 4 CPU cores constantly for this. That's a pretty worse case scenario, and it would cost us $37.50/month on GCP for those cores, in order to save about 100x that amount. The takeaway (for me at least) is that compressing an in-flight transfer is almost always worthwhile if you're working in the cloud and the transfer will eventually be stored somewhere. The economics of CPU vs the total amount of data storage cost is a no brainer. Hope that makes sense!

29athrowaway 6y ago
Thanks for the clarification.

zbjornson 6y ago

FTA: "This post aimed to cover a scenario where the cost of compression, in both compute resource and time-to-build, was significantly outweighed by the savings it would make."

dilatedmind 6y ago

my intuition is you can save cpu time when compressing before sending over the network (and certainly wall time). a quick test copying a 24M file (with similiar compression ratios) to s3 showed a 6% decrease in cpu time when piping through gzip.

EricBurnett 6y ago
Depends on selected compressor, but yes, you can. I've definitely observed zstd-1 to be a net savings, where compression/decompression costs were offset by pushing fewer bytes through the RPC and network layers - and this was only from observing the endpoints, not even counting CPU on intermediate proxies/firewalls/etc. I wouldn't normally expect gzip to be a net savings (it's comparatively more expensive), but depending on compression ratio achieved and what layers you're passing the bytes through, I'd definitely believe it can be in some contexts.
saurik 6y ago
Data sent to S3 is usually hashed (depending on authentication type) in addition to being transport encrypted; I imagine the majority of this cost here is the encryption of a larger payload (which many would consider indispensable, but I point this out because I do not generally assume this when I merely consider "over the network").
- user5994461 6y ago
  You assume incorrectly. SSL encryption is in the order of 1 GB/s on a recent CPU with AES instructions (anything from this decade). Gzip is in the order of 10 MB/s with default settings, down to 1 MB/s with the strongest compression setting. It's really really slow.
  - Kranar 6y ago
    GNU gzip the application is slow on the order of 10 MB/s because of how it does file IO, but the DEFLATE algorithm that gzip is based off of is much faster than 10 MB/s at the default "level 6". For example the slz implementation of DEFLATE compresses text at 1 GB/s [1]. Even the fairly common zlib implementation can compress text at close to 300 MB/s. http://www.libslz.org/ http://www.libslz.org/
    - user5994461 6y ago
      DEFLATE at level 6 is really doing 10 MB/s, doesn't matter if you're using gzip or zlib or another library. slz is closer to level 20 (if there was a level 20). It's fast but the compression ratio is meh. You're better of using lz4 or zstd.