5 ms·

So I was wondering, how do you do two-way sync with inheritly one-way tool like rsync? And turns out you cannot do it, not reliably. The way script works, it r

by theamk 8y ago

So I was wondering, how do you do two-way sync with inheritly one-way tool like rsync? And turns out you cannot do it, not reliably. The way script works, it runs change monitor on both sides; if there is a change on local side, it will do local->remote sync; if there is a change on remote side, it will do remote->local sync. This can go wrong in many, many ways. Here is the first example that came to my mind: you started a process on remote machine which creates lots of small files -- maybe extracting an archive, or generating images. So the syncer keeps syncing those files in remote->local direction. Meanwhile, you got bored watching the script and decided to edit some code. POOF! Any edits you make are continuously reverted. Oh, and there is no error checking anywhere. Did your network had a hickup? Tough, we will march on anyway. Let's it was not in lines 101 or 104 -- if these commands have transient failures, then your newly made changes would just get reverted. If you care about your data, please do not use this. Use anything else -- syncthing, osync, unison were named in this thread, they are all good.

api 8y ago
Are there any good open source live sync tools? Even one way sync with rsync is slow and two way is as you say a kludge. Seems like there is not much open source stuff out there for this and certainly nothing that can sync like these cloud drive apps.
- theamk 8y ago
  Well, I like "unison" -- it is a well designed, two way sync app. I personally run it in "-auto" mode -- every time I run the program, it shows me all actions it wants to do, any conflicts it detected, and asks me if I want to proceed. If you want to run live mode, you can just run: unison -ui text -repeat 5 . ssh://remote-host/dir it will check for changes every 5 seconds, sync over any changes, and skip all the conflicts. You'll have to re-run it without -repeat option to resolve the conflicts.
  - api 8y ago
    I tried it but it never seems to work all that well for me. It also has terrible UX with confusing options.
    - theamk 8y ago
      Unison's UX has options? I did not know. I recommend treating unison like you do rsync -- read the manpage, configure it with config files and shell wrappers. If you run it with "-ui text -auto", then it will print the list of changes, and ask: "do you want to proceed (y/n)", which is not that confusing. Treat it as mostly text-based syncer with UI as an extra bonus, and it will be much less confusing.
- m-p-3 8y ago
  Syncthing? It added Variable Size Blocks to speed up transfer of large files last year. Works will for one-way or multi-ways syncing https://docs.syncthing.net/users/foldertypes.html https://docs.syncthing.net/users/foldertypes.html
headgasket 8y ago
(author here) Hey thanks for the scrutiny! I'm using the -u option in rsync, preventing overwriting newer changes. So in your described case, it will signal a change remote (a lot of changes, but collapsed every 3 seconds) sync remote-local first then local-remote, both with the u option, so not updating if there's a newer version at the other end. No POOF, unless I'm mistaken. Cheers! edit for clarity The trick is every sync step is with -u (not replacing updated files) and done again in the other direction before restarting the watch.
- theamk 8y ago
  But you still have --delete there, and -u won't affect it. So if you create a local file, it will be deleted (it might stay if by chance it was created during second rsync, but this is not guaranteed). And the same deletion applies to network errors -- if ssh fails, then newly created files would get deleted. That said, I agree that -u options does makes it less likely that files get overwritten. This option approach a some caveats, however -- archive extraction will mess the modification time, symlinks and directories are not handled properly. Still, regular IDE editing will work. Still not sure how is it better than unison though :)
  - headgasket 8y ago
    Good thinking. You are right that there's a short window between the 2 rsyncs that allows for a newly-created file being created at say the remote end just after (or maybe during? --would need to test) the remote-local rsync that would get deleted by the subsequent local-remote rsync call. Since there's no lock or anything this vanishing new file just after creation is a possibility. I just wrote this yesterday, I'll be using it extensively, I'll see how annoying this is, and if there's something to do about it. As a side note, ssh failure has not been a problem (yet), since the script does the same strategy when starting up. In fact I kill and restart this script a lot. I havent played with archive extraction, this is mainly for source code editing. edit It would seem that a small modification to the --delete behaviour of rsync to only delete files at the other end that are older than say 30 seconds would handle this edge case. I'll see how annoying it is and if it warrants the time to investigate this.
vegardx 8y ago
You basically can't get around this without distributed locking.
- theamk 8y ago
  Locking is not needed as long as you have conflict detection. It might be a bit annoying, depending on UI, but at least it won't result in data loss. (Note that local locking can be helpful to prevent simulatenous modifications by editor and sync tool; but unfortunately Linux ecosystem is not designed with such locking in mind)

by theamk 8y ago

api 8y ago

Are there any good open source live sync tools? Even one way sync with rsync is slow and two way is as you say a kludge. Seems like there is not much open source stuff out there for this and certainly nothing that can sync like these cloud drive apps.

theamk 8y ago
Well, I like "unison" -- it is a well designed, two way sync app. I personally run it in "-auto" mode -- every time I run the program, it shows me all actions it wants to do, any conflicts it detected, and asks me if I want to proceed. If you want to run live mode, you can just run: unison -ui text -repeat 5 . ssh://remote-host/dir it will check for changes every 5 seconds, sync over any changes, and skip all the conflicts. You'll have to re-run it without -repeat option to resolve the conflicts.
- api 8y ago
  I tried it but it never seems to work all that well for me. It also has terrible UX with confusing options.
  - theamk 8y ago
    Unison's UX has options? I did not know. I recommend treating unison like you do rsync -- read the manpage, configure it with config files and shell wrappers. If you run it with "-ui text -auto", then it will print the list of changes, and ask: "do you want to proceed (y/n)", which is not that confusing. Treat it as mostly text-based syncer with UI as an extra bonus, and it will be much less confusing.
m-p-3 8y ago
Syncthing? It added Variable Size Blocks to speed up transfer of large files last year. Works will for one-way or multi-ways syncing https://docs.syncthing.net/users/foldertypes.html https://docs.syncthing.net/users/foldertypes.html

headgasket 8y ago

(author here) Hey thanks for the scrutiny! I'm using the -u option in rsync, preventing overwriting newer changes. So in your described case, it will signal a change remote (a lot of changes, but collapsed every 3 seconds) sync remote-local first then local-remote, both with the u option, so not updating if there's a newer version at the other end. No POOF, unless I'm mistaken. Cheers! edit for clarity The trick is every sync step is with -u (not replacing updated files) and done again in the other direction before restarting the watch.

theamk 8y ago
But you still have --delete there, and -u won't affect it. So if you create a local file, it will be deleted (it might stay if by chance it was created during second rsync, but this is not guaranteed). And the same deletion applies to network errors -- if ssh fails, then newly created files would get deleted. That said, I agree that -u options does makes it less likely that files get overwritten. This option approach a some caveats, however -- archive extraction will mess the modification time, symlinks and directories are not handled properly. Still, regular IDE editing will work. Still not sure how is it better than unison though :)
- headgasket 8y ago
  Good thinking. You are right that there's a short window between the 2 rsyncs that allows for a newly-created file being created at say the remote end just after (or maybe during? --would need to test) the remote-local rsync that would get deleted by the subsequent local-remote rsync call. Since there's no lock or anything this vanishing new file just after creation is a possibility. I just wrote this yesterday, I'll be using it extensively, I'll see how annoying this is, and if there's something to do about it. As a side note, ssh failure has not been a problem (yet), since the script does the same strategy when starting up. In fact I kill and restart this script a lot. I havent played with archive extraction, this is mainly for source code editing. edit It would seem that a small modification to the --delete behaviour of rsync to only delete files at the other end that are older than say 30 seconds would handle this edge case. I'll see how annoying it is and if it warrants the time to investigate this.

vegardx 8y ago

You basically can't get around this without distributed locking.

theamk 8y ago
Locking is not needed as long as you have conflict detection. It might be a bit annoying, depending on UI, but at least it won't result in data loss. (Note that local locking can be helpful to prevent simulatenous modifications by editor and sync tool; but unfortunately Linux ecosystem is not designed with such locking in mind)