5 ms·

Show HN: Simple realtime two-way sync with SSH, fswatch and rsync

71 points by headgasket 8y ago

etaioinshrdlu 8y ago
I have a few qualms with this app: 1. For a Linux user, you can already build such a system yourself quite trivially by using Dropbox. 2. It doesn't actually replace Dropbox. 3. It does not seem very "viral" or income-generating. I know this is premature at this point, but without charging users for the service, is it reasonable to expect to make money off of this? (Silly satire.)
- the_pwner224 8y ago
  This can sync any folder and (presumably) can have multiple instances running; Dropbox is limited to a single instance per user syncing ~/Dropbox. This is P2P and works without an internet connection. It seems to be built for a different use case; Nextcloud (server/client) and Syncthing (P2P) are already excellent Dropbox alternatives.
  - durub 8y ago
    All valid points. However, etaioinshrdlu is making a reference to the time Dropbox was first posted to Hacker News: https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863
rasengan 8y ago
Cool project! Another alternative is to just use unison. Not only is it cross platform for windows, Mac and Linux, but on top of that it “just works”. Amazing stuff. Brew, Cygwin and your favorite package managers have it.
- beagle3 8y ago
  When I checked a few years ago, unison worked but was (a) relatively slow, and (b) very picky about which versions are at which end. almost any two rsyncs from the last 20 years speak to each other and do it rather quickly.
  - theamk 8y ago
    For picky versions, there is nice `addversionno = true` option, which will let you have multiple unison versions installed simultaneously. It will always pick the right one. I agree with slowness -- I feel the recent versions are pretty fast on Linux, but it can be faster. Also it's "scan, then transfer" approach means it can take lots of memory if there is a lot of files. Still, if you can your rsync in your application, use it. For automated backups, you cannot beat it. But unison has two unique advantages: - Proper two way sync -- doing it safely with rsync is almost impossible. - GUI/TUI which shows what changed and allows conflict resolution. - GUI which shows
    - beagle3 8y ago
      Indeed. But you still need multiple versions installed, which means you have to sidestep package managers (and generally common usage practices); which is really inconvenient if you have multiple systems with different os’s and distros.
- NelsonMinar 8y ago
  Seconding unison. It's kind of a forgotten tool but it's remarkably robust and good at what it does. (I have to think being written in OCaml has been part of why it doesn't have more of a community around it.)
- headgasket 8y ago
  I had never come across that one either, thanks for the link! It does look a little on the heavy side for my use case, but it might come in handy at some point! It looks like it has all the bells and whistles. Cheers.
- nine_k 8y ago
  Also, syncthing. It seems to be better maintained. It does not preserve special file attributes and symlinks, though.
systemspeed 8y ago
These sorts of solutions are always tantalizing, especially when developing on a Mac but deploying on Linux, but the killer feature for such an application would be IDE/ST3/Atom/VSC[/etc...] support. If this sync process could be orchestrated from such a development environment directly, it would avoid many design pitfalls, such as degenerating into rapid fs polling.
- headgasket 8y ago
  hey! thanks for your comment! I wrote this yesterday because I wanted the read speed of a local copy, hot-reload of the remote side app, and the ability to edit at both ends. I came across fswatch, I've been quite impressed; it does a pretty good job resource-wise. On linux it uses ionotify; I have not seen it degenerate into rapid fs polling (yet). Cheers!
  - systemspeed 8y ago
    It's not a critique of your design. You solve the problem as you see it pretty spot on. It's more so that I wish IDEs supported software like this. There's a plethora of such offerings such as CyberDuck, Expandrive, etc that would benefit from reduced read/seek activity if the IDE could just orchestrate when to emit changes to what it thinks is the "disk". As you noted on GitHub, such software gets really laggy when working in directories that aren't trivially small.
orliesaurus 8y ago
I use a few rsync scripts and build shortcuts, but gonna try yours honestly!
senorsmile 8y ago
I highly recommend osync(1). It has been around a while and has had quite a few users report bugs. The maintainer is active and very helpful. I've used it for a few years now with great success. 1: https://github.com/deajan/osync https://github.com/deajan/osync
- headgasket 8y ago
  (author here) Interesting, thanks for the link! I had never seen this one. It does look a bit more heavyweight; that might be a good thing for some use cases. I currently use this on many different containerized setups, so simpler and less setup/config is better for me at this point. Cheers!
techntoke 8y ago
Do you feel that it is production ready and that the sync won't become corrupt? Any plans for Android?
- headgasket 8y ago
  Hum a day old might be early to say, but the sync is handled by rsync and that has been around for ages, corruption due to rsync would be surprising. (although hash and adler32 checksums collisions are theoretically possible.. :-)
  - techntoke 8y ago
    It's not rsync I'm concerned about. It is fswatch and adding/deleting multiple files (possibly from multiple devices) and having them automatically synced.
    - headgasket 8y ago
      More than 2 end points would be a bridge too far for this simple script. With regards to fswatch, from what I gather, it blocks a iotcl call on all the files in your watched folder. This script fires rsync (always from the local end to prevent confusion) as soon as a change is detected on either end, starting with the end where the change has been detected. Pretty simple. I guess if you change files and delete files at the same time on both ends, some deleted files might get recreated. At this point this is the worse I can see, but I could be very wrong... :-)
mamcx 8y ago
Related, any option to sync files that work from iOS/Android (excluding dropbox, just to make things hard)?
- headgasket 8y ago
  well if you can get bash, ssh, fswatch, rsync and socat we could get something going... ;-) For iOS coda is pretty good. I've used Codeanywhere for android; 2-way sync would need to be integrated in the app itself on these platforms, I would think...
jasonhansel 8y ago
For one-way sync, I'd also suggest: https://axkibe.github.io/lsyncd/ https://axkibe.github.io/lsyncd/
m-p-3 8y ago
Nice system, but I think I'll stick to Syncthing for the time being.
theamk 8y ago
So I was wondering, how do you do two-way sync with inheritly one-way tool like rsync? And turns out you cannot do it, not reliably. The way script works, it runs change monitor on both sides; if there is a change on local side, it will do local->remote sync; if there is a change on remote side, it will do remote->local sync. This can go wrong in many, many ways. Here is the first example that came to my mind: you started a process on remote machine which creates lots of small files -- maybe extracting an archive, or generating images. So the syncer keeps syncing those files in remote->local direction. Meanwhile, you got bored watching the script and decided to edit some code. POOF! Any edits you make are continuously reverted. Oh, and there is no error checking anywhere. Did your network had a hickup? Tough, we will march on anyway. Let's it was not in lines 101 or 104 -- if these commands have transient failures, then your newly made changes would just get reverted. If you care about your data, please do not use this. Use anything else -- syncthing, osync, unison were named in this thread, they are all good.
- api 8y ago
  Are there any good open source live sync tools? Even one way sync with rsync is slow and two way is as you say a kludge. Seems like there is not much open source stuff out there for this and certainly nothing that can sync like these cloud drive apps.
  - theamk 8y ago
    Well, I like "unison" -- it is a well designed, two way sync app. I personally run it in "-auto" mode -- every time I run the program, it shows me all actions it wants to do, any conflicts it detected, and asks me if I want to proceed. If you want to run live mode, you can just run: unison -ui text -repeat 5 . ssh://remote-host/dir it will check for changes every 5 seconds, sync over any changes, and skip all the conflicts. You'll have to re-run it without -repeat option to resolve the conflicts.
    - api 8y ago
      I tried it but it never seems to work all that well for me. It also has terrible UX with confusing options.

71 points by headgasket 8y ago

I have a few qualms with this app: 1. For a Linux user, you can already build such a system yourself quite trivially by using Dropbox. 2. It doesn't actually replace Dropbox. 3. It does not seem very "viral" or income-generating. I know this is premature at this point, but without charging users for the service, is it reasonable to expect to make money off of this? (Silly satire.)

the_pwner224 8y ago
This can sync any folder and (presumably) can have multiple instances running; Dropbox is limited to a single instance per user syncing ~/Dropbox. This is P2P and works without an internet connection. It seems to be built for a different use case; Nextcloud (server/client) and Syncthing (P2P) are already excellent Dropbox alternatives.
- durub 8y ago
  All valid points. However, etaioinshrdlu is making a reference to the time Dropbox was first posted to Hacker News: https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863

rasengan 8y ago

Cool project! Another alternative is to just use unison. Not only is it cross platform for windows, Mac and Linux, but on top of that it “just works”. Amazing stuff. Brew, Cygwin and your favorite package managers have it.

beagle3 8y ago
When I checked a few years ago, unison worked but was (a) relatively slow, and (b) very picky about which versions are at which end. almost any two rsyncs from the last 20 years speak to each other and do it rather quickly.
- theamk 8y ago
  For picky versions, there is nice `addversionno = true` option, which will let you have multiple unison versions installed simultaneously. It will always pick the right one. I agree with slowness -- I feel the recent versions are pretty fast on Linux, but it can be faster. Also it's "scan, then transfer" approach means it can take lots of memory if there is a lot of files. Still, if you can your rsync in your application, use it. For automated backups, you cannot beat it. But unison has two unique advantages: - Proper two way sync -- doing it safely with rsync is almost impossible. - GUI/TUI which shows what changed and allows conflict resolution. - GUI which shows
  - beagle3 8y ago
    Indeed. But you still need multiple versions installed, which means you have to sidestep package managers (and generally common usage practices); which is really inconvenient if you have multiple systems with different os’s and distros.
NelsonMinar 8y ago
Seconding unison. It's kind of a forgotten tool but it's remarkably robust and good at what it does. (I have to think being written in OCaml has been part of why it doesn't have more of a community around it.)
headgasket 8y ago
I had never come across that one either, thanks for the link! It does look a little on the heavy side for my use case, but it might come in handy at some point! It looks like it has all the bells and whistles. Cheers.
nine_k 8y ago
Also, syncthing. It seems to be better maintained. It does not preserve special file attributes and symlinks, though.

systemspeed 8y ago

These sorts of solutions are always tantalizing, especially when developing on a Mac but deploying on Linux, but the killer feature for such an application would be IDE/ST3/Atom/VSC[/etc...] support. If this sync process could be orchestrated from such a development environment directly, it would avoid many design pitfalls, such as degenerating into rapid fs polling.

headgasket 8y ago
hey! thanks for your comment! I wrote this yesterday because I wanted the read speed of a local copy, hot-reload of the remote side app, and the ability to edit at both ends. I came across fswatch, I've been quite impressed; it does a pretty good job resource-wise. On linux it uses ionotify; I have not seen it degenerate into rapid fs polling (yet). Cheers!
- systemspeed 8y ago
  It's not a critique of your design. You solve the problem as you see it pretty spot on. It's more so that I wish IDEs supported software like this. There's a plethora of such offerings such as CyberDuck, Expandrive, etc that would benefit from reduced read/seek activity if the IDE could just orchestrate when to emit changes to what it thinks is the "disk". As you noted on GitHub, such software gets really laggy when working in directories that aren't trivially small.

orliesaurus 8y ago

I use a few rsync scripts and build shortcuts, but gonna try yours honestly!

senorsmile 8y ago

I highly recommend osync(1). It has been around a while and has had quite a few users report bugs. The maintainer is active and very helpful. I've used it for a few years now with great success. 1: https://github.com/deajan/osync https://github.com/deajan/osync

headgasket 8y ago
(author here) Interesting, thanks for the link! I had never seen this one. It does look a bit more heavyweight; that might be a good thing for some use cases. I currently use this on many different containerized setups, so simpler and less setup/config is better for me at this point. Cheers!

techntoke 8y ago

Do you feel that it is production ready and that the sync won't become corrupt? Any plans for Android?

headgasket 8y ago
Hum a day old might be early to say, but the sync is handled by rsync and that has been around for ages, corruption due to rsync would be surprising. (although hash and adler32 checksums collisions are theoretically possible.. :-)
- techntoke 8y ago
  It's not rsync I'm concerned about. It is fswatch and adding/deleting multiple files (possibly from multiple devices) and having them automatically synced.
  - headgasket 8y ago
    More than 2 end points would be a bridge too far for this simple script. With regards to fswatch, from what I gather, it blocks a iotcl call on all the files in your watched folder. This script fires rsync (always from the local end to prevent confusion) as soon as a change is detected on either end, starting with the end where the change has been detected. Pretty simple. I guess if you change files and delete files at the same time on both ends, some deleted files might get recreated. At this point this is the worse I can see, but I could be very wrong... :-)

mamcx 8y ago

Related, any option to sync files that work from iOS/Android (excluding dropbox, just to make things hard)?

headgasket 8y ago
well if you can get bash, ssh, fswatch, rsync and socat we could get something going... ;-) For iOS coda is pretty good. I've used Codeanywhere for android; 2-way sync would need to be integrated in the app itself on these platforms, I would think...

jasonhansel 8y ago

For one-way sync, I'd also suggest: https://axkibe.github.io/lsyncd/ https://axkibe.github.io/lsyncd/

m-p-3 8y ago

Nice system, but I think I'll stick to Syncthing for the time being.

theamk 8y ago

So I was wondering, how do you do two-way sync with inheritly one-way tool like rsync? And turns out you cannot do it, not reliably. The way script works, it runs change monitor on both sides; if there is a change on local side, it will do local->remote sync; if there is a change on remote side, it will do remote->local sync. This can go wrong in many, many ways. Here is the first example that came to my mind: you started a process on remote machine which creates lots of small files -- maybe extracting an archive, or generating images. So the syncer keeps syncing those files in remote->local direction. Meanwhile, you got bored watching the script and decided to edit some code. POOF! Any edits you make are continuously reverted. Oh, and there is no error checking anywhere. Did your network had a hickup? Tough, we will march on anyway. Let's it was not in lines 101 or 104 -- if these commands have transient failures, then your newly made changes would just get reverted. If you care about your data, please do not use this. Use anything else -- syncthing, osync, unison were named in this thread, they are all good.

api 8y ago
Are there any good open source live sync tools? Even one way sync with rsync is slow and two way is as you say a kludge. Seems like there is not much open source stuff out there for this and certainly nothing that can sync like these cloud drive apps.
- theamk 8y ago
  Well, I like "unison" -- it is a well designed, two way sync app. I personally run it in "-auto" mode -- every time I run the program, it shows me all actions it wants to do, any conflicts it detected, and asks me if I want to proceed. If you want to run live mode, you can just run: unison -ui text -repeat 5 . ssh://remote-host/dir it will check for changes every 5 seconds, sync over any changes, and skip all the conflicts. You'll have to re-run it without -repeat option to resolve the conflicts.
  - api 8y ago
    I tried it but it never seems to work all that well for me. It also has terrible UX with confusing options.