4 ms·

Does rsync auto-resume after failed connections?

CarolineW 10y ago
I've written a script that every minute tests to see if the appropriate rsync command is running. If not, it simply runs it again. In that way it effectively restarts and gracefully resumes. This Google search: https://www.google.co.uk/search?q=rsync+auto-restart+failed+connection https://www.google.co.uk/search?q=rsync+auto-restart+failed+... returns this link: http://superuser.com/questions/302842/resume-rsync-over-ssh-after-broken-connection http://superuser.com/questions/302842/resume-rsync-over-ssh-... which contains this script: #!/bin/bash while [ 1 ] do rsync -avz --partial source dest if [ "$?" = "0" ] ; then echo "rsync completed normally" exit else echo "Rsync failure. Backing off and retrying..." sleep 180 fi done The comment says: When the connection dies, rsync will quit with a non-zero exit code. This script simply keeps re-running rsync, letting it continue until the synchronisation completes normally. That's pretty much what I've done.
- kraemate 10y ago
  Wow, thanks for the script. Surprising in its simplicity. I would have thought this use-case was popular enough to warrant specialized tools etc. Especially in the scientific community where they transfer large files.
  - CarolineW 10y ago
    It's simple enough that it's the sort of thing I type out in 30 seconds and there it is. No need for specialist tools - finding the tool, remembering how to use it, working out the right parameters ... Easier, faster, and more flexible just to write the script. It's what I do.
Piskvorrr 10y ago
You mean, if the connection fails before completing? I don't think so (I believe that's a feature). For automated transfers ("retry until done"), I use the lsyncd wrapper (which also watches the source files, so you don't need to poll for changes: it wakes up by inotify).
atsaloli 10y ago
Sure, just use --partial switch: --partial keep partially transferred files
- Piskvorrr 10y ago
  That is incorrect. This will keep partial transfers in destination (and when invoked again, will continue tranferring from that point onwards), but will not restart an aborted transfer (e.g. for a broken connection). In other words, this feature is a prerequisite for auto-resume, not auto-resume itself (which can be scripted in ~10 lines, as shown in a sibling thread).
  - atsaloli 10y ago
    Thanks! But no need to script, you can just use --append append data onto shorter files to resume interrupted transfers.
atsaloli 10y ago
Yes, it it does, if you use: --partial keep partially transferred files --append append data onto shorter files

I've written a script that every minute tests to see if the appropriate rsync command is running. If not, it simply runs it again. In that way it effectively restarts and gracefully resumes. This Google search: https://www.google.co.uk/search?q=rsync+auto-restart+failed+connection https://www.google.co.uk/search?q=rsync+auto-restart+failed+... returns this link: http://superuser.com/questions/302842/resume-rsync-over-ssh-after-broken-connection http://superuser.com/questions/302842/resume-rsync-over-ssh-... which contains this script: #!/bin/bash while [ 1 ] do rsync -avz --partial source dest if [ "$?" = "0" ] ; then echo "rsync completed normally" exit else echo "Rsync failure. Backing off and retrying..." sleep 180 fi done The comment says: When the connection dies, rsync will quit with a non-zero exit code. This script simply keeps re-running rsync, letting it continue until the synchronisation completes normally. That's pretty much what I've done.

kraemate 10y ago
Wow, thanks for the script. Surprising in its simplicity. I would have thought this use-case was popular enough to warrant specialized tools etc. Especially in the scientific community where they transfer large files.
- CarolineW 10y ago
  It's simple enough that it's the sort of thing I type out in 30 seconds and there it is. No need for specialist tools - finding the tool, remembering how to use it, working out the right parameters ... Easier, faster, and more flexible just to write the script. It's what I do.

Piskvorrr 10y ago

You mean, if the connection fails before completing? I don't think so (I believe that's a feature). For automated transfers ("retry until done"), I use the lsyncd wrapper (which also watches the source files, so you don't need to poll for changes: it wakes up by inotify).

atsaloli 10y ago

Sure, just use --partial switch: --partial keep partially transferred files

Piskvorrr 10y ago
That is incorrect. This will keep partial transfers in destination (and when invoked again, will continue tranferring from that point onwards), but will not restart an aborted transfer (e.g. for a broken connection). In other words, this feature is a prerequisite for auto-resume, not auto-resume itself (which can be scripted in ~10 lines, as shown in a sibling thread).
- atsaloli 10y ago
  Thanks! But no need to script, you can just use --append append data onto shorter files to resume interrupted transfers.

atsaloli 10y ago

Yes, it it does, if you use: --partial keep partially transferred files --append append data onto shorter files