5 ms·
I was surprised that he didn't try to use on the flight compression, provided by rsync: -z, --compress compress file data during the transfer
by mromanuk 1y ago
I was surprised that he didn't try to use on the flight compression, provided by rsync:
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
Probably it's faster to compress to gzip and later transfer. But it's nice to have the possibility to improve the transfer with a a flag.
- jddj 1y agoOr better yet, since they cite corruption issues, sqlite3_rsync (https://sqlite.org/rsync.html https://sqlite.org/rsync.html) with -z sqlite transaction- and WAL-aware rsync with inflight compression.
- worldsavior 1y agoI believe compression is only good on slow speed networks.
- berbec 1y agoValve tends to take a different view...
- stackskipton 1y agoValve has different needs then most. Their files are rarely change so they only need to do expensive compression once and they save a ton in bandwidth/storage along with fact that their users are more tolerant of download responsiveness.
- rollcat 1y agoDepends. Run a benchmark on your own hardware/network. ZFS uses in-flight compression because CPUs are generally faster than disks. That may or may not be the case for your setup.
- PhilipRoman 1y agoIt would have to be one really fast network... zstd compresses and decompresses at 5+ GB (bytes, not bits) per second.
- worldsavior 1y agoWhere are you getting this performance? On the average computer this is by far not the speed.
- o11c 1y agoI just tested on a ramdisk: tool cspeed size dspeed zstd 361 MB/s 16% 1321 MB/s lzop 512 MB/s 29% 539 MB/s lz4 555 MB/s 29% 1190 MB/s If working from files on disk that happen not to be cached, the speed differences are likely to disappear, even on many NVMe disks. (It just so happens that the concatenation of all text-looking .tar files I happen to have on this machine is roughly a gigabyte (though I did the math for the actual size)).
- PhilipRoman 1y agoLooks like it depends heavily on choice of file, but I see good performance on both compressible and uncompressible files. Small files tend to perform (relatively) bad though. Here is a sample of 3 large files with different compression ratios: zstd -b1 --fast -i10 some-rpi-os-image-idk.img -1#-os-image-idk.img :2097152000 -> 226798302 (x9.247), 6765.0 MB/s, 5897.3 MB/s zstd -b1 --fast -i10 jdk-11.0.8+10.tar -1#jdk-11.0.8+10.tar : 688844800 -> 142114709 (x4.847), 2660.7 MB/s, 2506.8 MB/s zstd -b1 --fast -i10 archlinux-2025.04.01-x86_64.iso -1#.04.01-x86_64.iso :1236303872 ->1221271223 (x1.012), 3643.5 MB/s, 7836.6 MB/s
- masklinn 1y agoAin't no way zstd compresses at 5+, even at -1. That's the sort of throughputs you see on lz4 running on a bunch of core (either half a dozen very fast, or 12~16 merely fast).
- cogman10 1y agoIs the network only doing an rsync? Then you are probably right. For every other network, you should compress as you are likely dealing with multiple tenants that would all like a piece of your 40Gbps bandwidth.
- worldsavior 1y agoIn your logic, you should not compress as multiple tenants would all like a piece of your CPU.
- cogman10 1y agoThis will always be something you have to determine for your own situation. At least at my work, CPU cores are plentiful, IO isn't. We rarely have apps that need more than a fraction of the CPU cores (barring garbage collection). Yet we are often serving fairly large chunks of data from those same apps.
- creatonez 1y agoWhat? Compression is absolutely essential throughout computing as a whole, especially as CPUs have gotten faster. If you have compressible data sent over the network (or even on disk / in RAM) there's a good chance you should be compressing it. Faster links have not undercut this reality in any significant way.
- bityard 1y agoWhether or not to compress data before transfer is VERY situationally dependent. I have seen it go both ways and the real-world results do not not always match intuition. At the end of the day, if you care about performance, you still have to do proper testing. (This is the same spiel I give whenever someone says swap on Linux is or is not always beneficial.)
- berbec 1y agoor used --remove-source-files so they didn't have to ssh back to rm
- crazygringo 1y agoThe main point is to skip the indices, which you have to do pre-compression. When I do stuff like this, I stream the dump straight into gzip. (You can usually figure out a way to stream directly to the destination without an intermediate file at all.) Plus this way it stays stored compressed at its destination. If your purpose is backup rather than a poor man's replication.
- schnable 1y agoThe main point was decreasing the transfer time - if rsync -z makes it short enough, it doesn't matter if the indices are there or not, and you also skip the step of re-creating the DB from the text file.
- crazygringo 1y agoThe point of the article is that it does matter if the indices are there. And indices generally don't compress very well anyways. What compresses well are usually things like human-readable text fields or booleans/enums.
- Jyaif 1y agoHe absolutely should be doing this, because by using rsync on a compressed file he's passing by the whole point of using rsync, which is the rolling-checksum based algorithm that allows to transfer diffs.