18 ms·

Apache Helix – Near-Realtime Rsync Replicated File System

296 points by GordonS 6y ago

bigbubba 6y ago
How is this using rsync? Does it invoke an rsync process, or implement the rsync wire protocol? One of my gripes with rsync is I've not found another way to integrate it into other systems. librsync doesn't implement the wire protocol and parsing the output of rsync itself is fraught with peril (for instance the formatting of --no-h has changed before.) Edit: it seems to invoke an a rsync process but doesn't parse stdout for progress. A bit disappointing but I suppose that's all which is needed for this application.
- pengaru 6y ago
  librsync has absolutely no connection to the rsync utility, it's a distinct implementation of the underlying differential algorithm. The rsync utility doesn't use it, and they don't even share a common data format at any level.
  - bigbubba 6y ago
    Yes I'm aware, but usually when I voice my gripes with integrating rsync somebody unfamiliar with the matter quickly googles 'rsync library', finds librsync, and assumes it must be the answer to my problem. Then I have to explain to them that it's not. I thought to head that off this time by mentioning upfront that librsync is not the answer.
    - pengaru 6y ago
      I see, yeah, librsync's misleading name has clearly caused a lot of wasted time.
    - techcode 6y ago
      Just to be sure I'm getting the question - you want to use rsync(1), and you're wondering if there's a good way to guard your code that's using/calling it from underlying changes in rsync options and output, how to read status/progress and such? Here's an example of library that wraps rsync https://metacpan.org/pod/File::Rsync https://metacpan.org/pod/File::Rsync - you can provide callback functions that will be called for each line of stdout/stderr. Granted it's in Perl - and chances are your code isn't written in Perl. So if you can't find something similar for your language of choice, and if you can't be bothered with checking out implementation in Perl and rewriting it... I would suggest searching & checking out "Inline::" namespace (e.g.: https://metacpan.org/pod/distribution/Inline-Java/lib/Inline/Java.pod https://metacpan.org/pod/distribution/Inline-Java/lib/Inline..., and there's also JS, C ...etc). And next to those - Perlito can "Compile Perl to Java/JavaScript/Python/Ruby/Go/etc" https://github.com/fglock/Perlito https://github.com/fglock/Perlito. I think we had/have some Perl code running in production Hadoop/JVM.
    - danmur 6y ago
      No way to win :)
- strbean 6y ago
  rclone might hold some promise as an alternative, if you don't mind using Go.
- rakoo 6y ago
  Indeed there is no widespread documentation on how to play with rsync itself. A long time ago I played with the idea of replicating rproxy (https://rproxy.samba.org/ https://rproxy.samba.org/), which was just a cool idea. The results are here (https://rproxy.samba.org/ https://rproxy.samba.org/) but are very very basic, it was just a toy. I did have to invent my own protocol, but I did it over HTTP.
varispeed 6y ago
I used to run distributed system based on a few bash scripts and rsync to work in real time. It has become more complex when each "cell" exceeded the node capacity. The file naming system was not flexible enough to split these cells into smaller nodes and I abandoned writing a router for Nginx to direct requests to right server based on the filenames. I just decided to throw a couple of big machines at the problem instead and that removed all complexity. It wouldn't be possible just a couple of years ago though. For a couple of years the rsync solution worked very well.
3np 6y ago
I'm using a distributed filesystem for persisted files (config, assets and resources, etc - anything that'd need to be in a docker volume and persisted between restarts) in my container workloads. This mostly works fine, except for sqlite applications that insist on using WAL - this completely breaks and results in corrupted database files. Maybe this could be an alternative for those!
- teraflop 6y ago
  "Near-realtime" isn't good enough for SQLite. It has a strong dependency on the consistency guarantees provided by a POSIX-like filesystem. The approach described in this page does make some attempt to provide consistency. The master generates a single stream of file updates, and all of the replicas consistently apply those changes in the same order. But rsync is not guaranteed to observe changes to files on the master in the same order that they were written. In particular, SQLite (by default) uses a rollback journal to recover from failures. While a transaction is in process and some parts of the database might have been partially updated, the journal stores the old contents of the modified pages. Since the old data is fsync'ed to the journal before the new data is written to the database file, the data on disk at any moment in time is recoverable. But there's no clean way for rsync to read an atomic snapshot of both the database file and the journal at the same instant in time. If it reads them at different times, a database change might be partially applied or partially rolled back, which has a high likelihood of corrupting the database.
- jlokier 6y ago
  This won't work for database files, and it won't work for any other application that requires a filesystem to behave consistently over a network either. The property you are looking for in a distributed filesystem is called strong consistency, or cache coherent (different name, same effect). If a distributed filesystem does genuinely provide that property, it should be ok for SQLite and other applications, because it means the filesystem behaves the same as if it were multiple processes on the same local system accessing it. Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. There is another property called durability, which you might also care about. That affects whether you get corruption when devices fail or are rebooted suddenly.
traceroute66 6y ago
Minor rant: The trouble with all these lovely Apache projects is their dependency on Java. If these things were written in C, Go or Rust then I'd leap at the opportunity to explore them and maybe use them in production. But Java dependency just brings so much baggage with it. Its all well and good if you've got a Java pro or two on your team. But if you don't, you usually end up spending hours and days troubleshooting obscure JVM problems or somesuch.
- ktpsns 6y ago
  I second this so much! I have the feeling that the ASF tries to be so enterprisy with Java. I have no idea. (Note that there are a lot of very high quality Apache java projects.)
  - pjmlp 6y ago
    One of their best features.
  - jaaron 6y ago
    You have to go back almost 20 years to understand the relationship between the ASF and the Java community. Java at the time was positioned to be a core web technology (originally as applets, then as as backend tech via J2EE) and Apache was interested in ensuring the web ecosystem as a whole wasn't locked up in proprietary tech. The ASF was rather influential in working with Sun at the time to foster was eventually became a vibrant open source community. I know the ASF isn't as trendy these days, but that's never really bothered those of us who volunteer. The focus has always been about a stable, open source ecosystem, primarily, but not exclusively for the internet and its underlying infrastructure. The goal is measured in decades. It wasn't about being "enterprisey" or anything like that. The ASF as a foundation doesn't really care about the language or tech stack. Being a more mature non-profit, many corporations have found it easier to interface with the ASF than many other open source organizations out there, which does lend itself to seeing a lot of enterprisey donations.
- buryat 6y ago
  java is very easy to troubleshoot compared to c, go, and rust
redm 6y ago
I wonder how near-realtime this is. It seems that will dictate the types of applications this will be applicable for. I currently use Gluster which has its downsides, but is what I consider near-realtime, but with full filesystem features in a nice fuse wrapper. I'd love it if someone would contrast the performance and more of the features.
- late2part 6y ago
  In the mid-2000s (the "oughts?") I went to my doctor at PAMF with a sore throat and he said he would do a strep test. I asked him if we would get "real time results?" He stopped, and looked at me puzzled with a tilted head... "What other kind of time is there?" Your question is highly reasonable, but I heard a colleague today say that the issue could happen only in a very, very short time period of, say, 2-3 minutes. We routinely work with corner cases of 1-2 ms; so the context is relative :-)
  - jjtheblunt 6y ago
    "context is relative" made me smile, thinking "PAMF" is Palo Alto Medical Foundation, but why would anyone in general have the context to understand that
- StavrosK 6y ago
  What are the downsides of Gluster?
  - rodgerd 6y ago
    Gluster's performance with metadata heavy workloads is very poor. If you're running files with a high ratio of data:metadata, it works well; if you're running something like a git repo on Gluster it is not a great experience. It's also thrashing a bit development wise; there's a lot of features being pruned, and I'm a little concerned what its future looks like as Red Hat push harder down the Ceph route, since they're the main source of code contributions to Gluster.
par 6y ago
Looks like a great way to keep my photos, videos and documents backed up :)
- coffeeri 6y ago
  I wouldn't consider this as a backup solution, more as one for high-availability of data.
- mceachen 6y ago
  Please don't use this as your only backup. Consider it an experimental cross-machine RAID. You'll adopt the oft-repeated "RAID isn't a backup" mantra as soon as any of these issues hit you (like several have hit me): https://photostructure.com/faq/raid-is-not-a-backup/#why-isnt-raid-a-backup https://photostructure.com/faq/raid-is-not-a-backup/#why-isn...
gfodor 6y ago
This might be an appropriate thread: does anyone have a OSS alternative to EFS they have run in production? (A NFS-mountable, elastic storage system.)
- cyberpunk 6y ago
  Leofs
- JoshTriplett 6y ago
  Does it specifically need to be NFS, or just mountable?
  - gfodor 6y ago
    Just mountable
- gluteousminimus 6y ago
  In the HPC space, Lustre.
pengaru 6y ago
A major problem with rsync is it relies on filenames matching at the source and destination for an accelerated delta transfer to occur. If a file gets renamed or copied, modified or not, rsync will transfer the whole file. They added --fuzzy to try improve on this situation, but it generally only helps with unmodified files copied/renamed within the same directory. You can build robust solutions around rsync that work efficiently most of the time, but pathological cases such as users having a huge file regularly regenerated using something like a $(date --iso-8601=seconds) filename (think mysqldump backups) are surprisingly common once you have enough users. Even if you can convince them to uncompress such files for delta transfer sake, something as common as a versioned filename prevents rsync from automatically finding the relationship for use in the delta transfer.
- catmanjan 6y ago
  How else could they do it? All systems I've seen that have relied on files on a file system use the file name as the "primary key" I believe you can use journalling to detect file name change events but I don't think it scales well
  - pengaru 6y ago
    Who is "they"? Are you asking about rysnc developers or filesystem developers? For a filesystem the developers are free to innovate on any form of filesystem-wide block checksumming and content-addressed deduplication approach. This kind of global block-level content awareness is entirely decoupled from filenames and paths. Using rsync to underpin a filesystem strikes me as a gross hack done by developers punting on the real challenges of modern filesystem development. WRT rsync doing better than --fuzzy, the options are more limited since it's a userspace tool working within the confines of filesystem APIs like POSIX.
    - catmanjan 6y ago
      So basically there is no good solution at the moment, and a good solution would probably involve enhancing the underlying filesystem - this was the same conclusion I came to
ysleepy 6y ago
This is a cluster mgmt system that also has a file sync feature built in, probably to deploy binaries and data to run jobs. https://helix.apache.org https://helix.apache.org The link here is to the docs of version 0.6.8 while 1.0.1 is current (after 0.7, 0.8 and 0.9). This also explains the heavy weight architecture, you probably are running a ZooKeeper anyway in that environment.
tashbarg 6y ago
> To facilitate this, the master logs each transaction in a file and each transaction is associated with an 64 bit ID in which the 32 LSB represents a sequence number and MSB represents the generation number The sequence number gets incremented on every transaction and the generation is incremented when a new master is elected So, what happens after 32bit transactions without re-election?
fulafel 6y ago
How do I mount this? Or is it like s3, inaccessible as a normal FS?
mperham 6y ago
I use Resilio Sync in production with my servers. I like it more than this solution because it does not require a primary and replica design but instead uses bit torrent where all servers are equal. I can add a new server and drop an old server and it handles everything gracefully.
- GordonS 6y ago
  Haven't come across this before. At a glance, it looks like it's a file sync tool for small scale use, rather than a "serious" tool for replicating a file system. Curious to know more about your production use case?
minaguib 6y ago
It looks like a more distributed alternative to lsyncd ( https://axkibe.github.io/lsyncd/ https://axkibe.github.io/lsyncd/ )
thamer 6y ago
[deleted]
- deleted 6y ago
  [deleted]
orasis 6y ago
What is the point of rsync if the files aren’t updated?
hedora 6y ago
It’d be nice if something like this was built atop unison.
- stephenr 6y ago
  Unison is only intended for two "points" isn't it? i.e. it's meant for pretty narrow range of "client : server" sync jobs, and is essentially unusable without exactly the same patch version on both ends. For purely server-side "cluster" sync I've found csync2 to be a reasonable solution.
chinachin 6y ago
Thanks for picking out the time to discuss this. It is extremely helpful for me. Thanks for such a valuable help again. https://wordcounter.tools https://wordcounter.tools
riejo 6y ago
When it's 2020 and you label your boxes master/slave.
- butterisgood 6y ago
  Yeah a lot of people are moving away from this terminology thankfully!
  - anaganisk 6y ago
    Im not sure if you're being sarcastic or if it really matters that we call it matters and slave. Are we so easy to get triggered?
stephenr 6y ago
As I haven't seen it mentioned yet, if you want bi-directional sync of changes with relatively low overhead, I've found csync2 to be reasonably good for this task.

296 points by GordonS 6y ago

How is this using rsync? Does it invoke an rsync process, or implement the rsync wire protocol? One of my gripes with rsync is I've not found another way to integrate it into other systems. librsync doesn't implement the wire protocol and parsing the output of rsync itself is fraught with peril (for instance the formatting of --no-h has changed before.) Edit: it seems to invoke an a rsync process but doesn't parse stdout for progress. A bit disappointing but I suppose that's all which is needed for this application.

pengaru 6y ago
librsync has absolutely no connection to the rsync utility, it's a distinct implementation of the underlying differential algorithm. The rsync utility doesn't use it, and they don't even share a common data format at any level.
- bigbubba 6y ago
  Yes I'm aware, but usually when I voice my gripes with integrating rsync somebody unfamiliar with the matter quickly googles 'rsync library', finds librsync, and assumes it must be the answer to my problem. Then I have to explain to them that it's not. I thought to head that off this time by mentioning upfront that librsync is not the answer.
  - pengaru 6y ago
    I see, yeah, librsync's misleading name has clearly caused a lot of wasted time.
  - techcode 6y ago
    Just to be sure I'm getting the question - you want to use rsync(1), and you're wondering if there's a good way to guard your code that's using/calling it from underlying changes in rsync options and output, how to read status/progress and such? Here's an example of library that wraps rsync https://metacpan.org/pod/File::Rsync https://metacpan.org/pod/File::Rsync - you can provide callback functions that will be called for each line of stdout/stderr. Granted it's in Perl - and chances are your code isn't written in Perl. So if you can't find something similar for your language of choice, and if you can't be bothered with checking out implementation in Perl and rewriting it... I would suggest searching & checking out "Inline::" namespace (e.g.: https://metacpan.org/pod/distribution/Inline-Java/lib/Inline/Java.pod https://metacpan.org/pod/distribution/Inline-Java/lib/Inline..., and there's also JS, C ...etc). And next to those - Perlito can "Compile Perl to Java/JavaScript/Python/Ruby/Go/etc" https://github.com/fglock/Perlito https://github.com/fglock/Perlito. I think we had/have some Perl code running in production Hadoop/JVM.
  - danmur 6y ago
    No way to win :)
strbean 6y ago
rclone might hold some promise as an alternative, if you don't mind using Go.
rakoo 6y ago
Indeed there is no widespread documentation on how to play with rsync itself. A long time ago I played with the idea of replicating rproxy (https://rproxy.samba.org/ https://rproxy.samba.org/), which was just a cool idea. The results are here (https://rproxy.samba.org/ https://rproxy.samba.org/) but are very very basic, it was just a toy. I did have to invent my own protocol, but I did it over HTTP.

varispeed 6y ago

I used to run distributed system based on a few bash scripts and rsync to work in real time. It has become more complex when each "cell" exceeded the node capacity. The file naming system was not flexible enough to split these cells into smaller nodes and I abandoned writing a router for Nginx to direct requests to right server based on the filenames. I just decided to throw a couple of big machines at the problem instead and that removed all complexity. It wouldn't be possible just a couple of years ago though. For a couple of years the rsync solution worked very well.

3np 6y ago

I'm using a distributed filesystem for persisted files (config, assets and resources, etc - anything that'd need to be in a docker volume and persisted between restarts) in my container workloads. This mostly works fine, except for sqlite applications that insist on using WAL - this completely breaks and results in corrupted database files. Maybe this could be an alternative for those!

teraflop 6y ago
"Near-realtime" isn't good enough for SQLite. It has a strong dependency on the consistency guarantees provided by a POSIX-like filesystem. The approach described in this page does make some attempt to provide consistency. The master generates a single stream of file updates, and all of the replicas consistently apply those changes in the same order. But rsync is not guaranteed to observe changes to files on the master in the same order that they were written. In particular, SQLite (by default) uses a rollback journal to recover from failures. While a transaction is in process and some parts of the database might have been partially updated, the journal stores the old contents of the modified pages. Since the old data is fsync'ed to the journal before the new data is written to the database file, the data on disk at any moment in time is recoverable. But there's no clean way for rsync to read an atomic snapshot of both the database file and the journal at the same instant in time. If it reads them at different times, a database change might be partially applied or partially rolled back, which has a high likelihood of corrupting the database.
jlokier 6y ago
This won't work for database files, and it won't work for any other application that requires a filesystem to behave consistently over a network either. The property you are looking for in a distributed filesystem is called strong consistency, or cache coherent (different name, same effect). If a distributed filesystem does genuinely provide that property, it should be ok for SQLite and other applications, because it means the filesystem behaves the same as if it were multiple processes on the same local system accessing it. Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down. There is another property called durability, which you might also care about. That affects whether you get corruption when devices fail or are rebooted suddenly.

traceroute66 6y ago

Minor rant: The trouble with all these lovely Apache projects is their dependency on Java. If these things were written in C, Go or Rust then I'd leap at the opportunity to explore them and maybe use them in production. But Java dependency just brings so much baggage with it. Its all well and good if you've got a Java pro or two on your team. But if you don't, you usually end up spending hours and days troubleshooting obscure JVM problems or somesuch.

ktpsns 6y ago
I second this so much! I have the feeling that the ASF tries to be so enterprisy with Java. I have no idea. (Note that there are a lot of very high quality Apache java projects.)
- pjmlp 6y ago
  One of their best features.
- jaaron 6y ago
  You have to go back almost 20 years to understand the relationship between the ASF and the Java community. Java at the time was positioned to be a core web technology (originally as applets, then as as backend tech via J2EE) and Apache was interested in ensuring the web ecosystem as a whole wasn't locked up in proprietary tech. The ASF was rather influential in working with Sun at the time to foster was eventually became a vibrant open source community. I know the ASF isn't as trendy these days, but that's never really bothered those of us who volunteer. The focus has always been about a stable, open source ecosystem, primarily, but not exclusively for the internet and its underlying infrastructure. The goal is measured in decades. It wasn't about being "enterprisey" or anything like that. The ASF as a foundation doesn't really care about the language or tech stack. Being a more mature non-profit, many corporations have found it easier to interface with the ASF than many other open source organizations out there, which does lend itself to seeing a lot of enterprisey donations.
buryat 6y ago
java is very easy to troubleshoot compared to c, go, and rust

redm 6y ago

I wonder how near-realtime this is. It seems that will dictate the types of applications this will be applicable for. I currently use Gluster which has its downsides, but is what I consider near-realtime, but with full filesystem features in a nice fuse wrapper. I'd love it if someone would contrast the performance and more of the features.

late2part 6y ago
In the mid-2000s (the "oughts?") I went to my doctor at PAMF with a sore throat and he said he would do a strep test. I asked him if we would get "real time results?" He stopped, and looked at me puzzled with a tilted head... "What other kind of time is there?" Your question is highly reasonable, but I heard a colleague today say that the issue could happen only in a very, very short time period of, say, 2-3 minutes. We routinely work with corner cases of 1-2 ms; so the context is relative :-)
- jjtheblunt 6y ago
  "context is relative" made me smile, thinking "PAMF" is Palo Alto Medical Foundation, but why would anyone in general have the context to understand that
StavrosK 6y ago
What are the downsides of Gluster?
- rodgerd 6y ago
  Gluster's performance with metadata heavy workloads is very poor. If you're running files with a high ratio of data:metadata, it works well; if you're running something like a git repo on Gluster it is not a great experience. It's also thrashing a bit development wise; there's a lot of features being pruned, and I'm a little concerned what its future looks like as Red Hat push harder down the Ceph route, since they're the main source of code contributions to Gluster.

par 6y ago

Looks like a great way to keep my photos, videos and documents backed up :)

coffeeri 6y ago
I wouldn't consider this as a backup solution, more as one for high-availability of data.
mceachen 6y ago
Please don't use this as your only backup. Consider it an experimental cross-machine RAID. You'll adopt the oft-repeated "RAID isn't a backup" mantra as soon as any of these issues hit you (like several have hit me): https://photostructure.com/faq/raid-is-not-a-backup/#why-isnt-raid-a-backup https://photostructure.com/faq/raid-is-not-a-backup/#why-isn...

gfodor 6y ago

This might be an appropriate thread: does anyone have a OSS alternative to EFS they have run in production? (A NFS-mountable, elastic storage system.)

cyberpunk 6y ago
Leofs
JoshTriplett 6y ago
Does it specifically need to be NFS, or just mountable?
- gfodor 6y ago
  Just mountable
gluteousminimus 6y ago
In the HPC space, Lustre.

pengaru 6y ago

A major problem with rsync is it relies on filenames matching at the source and destination for an accelerated delta transfer to occur. If a file gets renamed or copied, modified or not, rsync will transfer the whole file. They added --fuzzy to try improve on this situation, but it generally only helps with unmodified files copied/renamed within the same directory. You can build robust solutions around rsync that work efficiently most of the time, but pathological cases such as users having a huge file regularly regenerated using something like a $(date --iso-8601=seconds) filename (think mysqldump backups) are surprisingly common once you have enough users. Even if you can convince them to uncompress such files for delta transfer sake, something as common as a versioned filename prevents rsync from automatically finding the relationship for use in the delta transfer.

catmanjan 6y ago
How else could they do it? All systems I've seen that have relied on files on a file system use the file name as the "primary key" I believe you can use journalling to detect file name change events but I don't think it scales well
- pengaru 6y ago
  Who is "they"? Are you asking about rysnc developers or filesystem developers? For a filesystem the developers are free to innovate on any form of filesystem-wide block checksumming and content-addressed deduplication approach. This kind of global block-level content awareness is entirely decoupled from filenames and paths. Using rsync to underpin a filesystem strikes me as a gross hack done by developers punting on the real challenges of modern filesystem development. WRT rsync doing better than --fuzzy, the options are more limited since it's a userspace tool working within the confines of filesystem APIs like POSIX.
  - catmanjan 6y ago
    So basically there is no good solution at the moment, and a good solution would probably involve enhancing the underlying filesystem - this was the same conclusion I came to

ysleepy 6y ago

This is a cluster mgmt system that also has a file sync feature built in, probably to deploy binaries and data to run jobs. https://helix.apache.org https://helix.apache.org The link here is to the docs of version 0.6.8 while 1.0.1 is current (after 0.7, 0.8 and 0.9). This also explains the heavy weight architecture, you probably are running a ZooKeeper anyway in that environment.

tashbarg 6y ago

> To facilitate this, the master logs each transaction in a file and each transaction is associated with an 64 bit ID in which the 32 LSB represents a sequence number and MSB represents the generation number The sequence number gets incremented on every transaction and the generation is incremented when a new master is elected So, what happens after 32bit transactions without re-election?

fulafel 6y ago

How do I mount this? Or is it like s3, inaccessible as a normal FS?

mperham 6y ago

I use Resilio Sync in production with my servers. I like it more than this solution because it does not require a primary and replica design but instead uses bit torrent where all servers are equal. I can add a new server and drop an old server and it handles everything gracefully.

GordonS 6y ago
Haven't come across this before. At a glance, it looks like it's a file sync tool for small scale use, rather than a "serious" tool for replicating a file system. Curious to know more about your production use case?

minaguib 6y ago

It looks like a more distributed alternative to lsyncd ( https://axkibe.github.io/lsyncd/ https://axkibe.github.io/lsyncd/ )

thamer 6y ago

[deleted]

deleted 6y ago
[deleted]

orasis 6y ago

What is the point of rsync if the files aren’t updated?

hedora 6y ago

It’d be nice if something like this was built atop unison.

stephenr 6y ago
Unison is only intended for two "points" isn't it? i.e. it's meant for pretty narrow range of "client : server" sync jobs, and is essentially unusable without exactly the same patch version on both ends. For purely server-side "cluster" sync I've found csync2 to be a reasonable solution.

chinachin 6y ago

Thanks for picking out the time to discuss this. It is extremely helpful for me. Thanks for such a valuable help again. https://wordcounter.tools https://wordcounter.tools

riejo 6y ago

When it's 2020 and you label your boxes master/slave.

butterisgood 6y ago
Yeah a lot of people are moving away from this terminology thankfully!
- anaganisk 6y ago
  Im not sure if you're being sarcastic or if it really matters that we call it matters and slave. Are we so easy to get triggered?

stephenr 6y ago

As I haven't seen it mentioned yet, if you want bi-directional sync of changes with relatively low overhead, I've found csync2 to be reasonably good for this task.