4 ms·

Switch your blind faith to Go (s5cmd) and get ~1200 MB/s upload speed! zfs send tank/pgdata@snapshot | pbzip2 > mybackup.zfs.bz2 s5cmd cp mybackup.zfs.bz2 s3:

• attentive 2y ago
  If s3 is the goal, there is gof3r which can be piped into i.e. skip storing mybackup.zfs.bz2 locally. Overall, I didn't see if they've identified the bottleneck. My guess pbzip2 is the slowest, ssh second. For compression bw I'd check zstd. For ssh there are various cipher/compression options. Or perhaps skip altogether and use wireguard.
  • tarasglek 2y ago
    gof3r looks amazing, thank you!