3 ms·

It would be interesting to see tarsnap added to the evaluation.

cperciva 6y ago
Not entirely sure that I'm reproducing the benchmark correctly, but running Amazon Linux 2 I have: # mount -t tmpfs -o size=32G tempfs tmp # gdir=~/linux/.git # mkdir tmp/linux # for commit in $(git --git-dir="$gdir" rev-list v5.9 | head -n 20 | tac); do \ mkdir ~/tmp/linux/$commit; \ git "--git-dir=$gdir" archive "$commit" | tar -C ~/tmp/linux/$commit -xf -; \ done # ~ec2-user/tarsnap-autoconf-1.0.39/tarsnap -c --dry-run --print-stats tmp tarsnap: Performing dry-run archival without keys (sizes may be slightly inaccurate) Total size Compressed size All archives 20278980504 4406609078 (unique data) 1767400183 258587867 This archive 20278980504 4406609078 New data 1767400183 258587867 So (assuming I reproduced the benchmark correctly!) that's 258 MB, or about 1/3 less space than bupstash. I also had "while true; do ps aux | grep tarsnap | grep -v grep; sleep 5; done" running in a different window to monitor the RSS (I don't know what the right flags are to time(1) to get that data on Linux) and it maxed out at 11472 kB. I suspect that Tarsnap is slower than the other options, though. That's something which I'm not able to compare easily. If the author is interested in adding Tarsnap to his comparison, I'd love to help (and can provide a free account for benchmarking).
- andrewchambers 6y ago
  Really interesting to see, thanks for doing that, I feel like many fast and good rolling hash functions are yet to be invented.
- array 6y ago
  I’ve had no experience with the other tools mentioned in this post so I cant provide benchmark feedback, but just wanted to say that I’ve been a happy tarsnap customer for a couple of years now. It’s well documented and does exactly what it’s supposed to. Well done.
- majewsky 6y ago
  Since you have the well-known "ps aux | grep $THING | grep -v grep" pattern in there, I'll plug one of my favorite shell functions: psgrep () { ps aux | sed -n '1p;/\<sed\>/d;/'"$1"'/p' } This does basically the same, but includes the header line from `ps aux` in the output. For instance: $ psgrep alacritty USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND majewsky 3500 0.4 0.5 1427280 93644 ? Sl 12:55 0:01 alacritty
  - cpach 6y ago
    Interesting! Might also be useful for CSV mogrification then.

Not entirely sure that I'm reproducing the benchmark correctly, but running Amazon Linux 2 I have: # mount -t tmpfs -o size=32G tempfs tmp # gdir=~/linux/.git # mkdir tmp/linux # for commit in $(git --git-dir="$gdir" rev-list v5.9 | head -n 20 | tac); do \ mkdir ~/tmp/linux/$commit; \ git "--git-dir=$gdir" archive "$commit" | tar -C ~/tmp/linux/$commit -xf -; \ done # ~ec2-user/tarsnap-autoconf-1.0.39/tarsnap -c --dry-run --print-stats tmp tarsnap: Performing dry-run archival without keys (sizes may be slightly inaccurate) Total size Compressed size All archives 20278980504 4406609078 (unique data) 1767400183 258587867 This archive 20278980504 4406609078 New data 1767400183 258587867 So (assuming I reproduced the benchmark correctly!) that's 258 MB, or about 1/3 less space than bupstash. I also had "while true; do ps aux | grep tarsnap | grep -v grep; sleep 5; done" running in a different window to monitor the RSS (I don't know what the right flags are to time(1) to get that data on Linux) and it maxed out at 11472 kB. I suspect that Tarsnap is slower than the other options, though. That's something which I'm not able to compare easily. If the author is interested in adding Tarsnap to his comparison, I'd love to help (and can provide a free account for benchmarking).

andrewchambers 6y ago
Really interesting to see, thanks for doing that, I feel like many fast and good rolling hash functions are yet to be invented.
array 6y ago
I’ve had no experience with the other tools mentioned in this post so I cant provide benchmark feedback, but just wanted to say that I’ve been a happy tarsnap customer for a couple of years now. It’s well documented and does exactly what it’s supposed to. Well done.
majewsky 6y ago
Since you have the well-known "ps aux | grep $THING | grep -v grep" pattern in there, I'll plug one of my favorite shell functions: psgrep () { ps aux | sed -n '1p;/\<sed\>/d;/'"$1"'/p' } This does basically the same, but includes the header line from `ps aux` in the output. For instance: $ psgrep alacritty USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND majewsky 3500 0.4 0.5 1427280 93644 ? Sl 12:55 0:01 alacritty
- cpach 6y ago
  Interesting! Might also be useful for CSV mogrification then.