3 ms·

Very interesting. I always wanted a solution to asynchronously replicate 100's of GBs over long distances. I've longingly looked at Intermezzo/Coda a long time

by nona 12y ago

Very interesting. I always wanted a solution to asynchronously replicate 100's of GBs over long distances. I've longingly looked at Intermezzo/Coda a long time ago but it never went anywhere; played with block level replication (lvmsync), but it doesn't allow concurrent use; in the end, the only solution I had to fall back on was rsync (which needs to iterate over the entire directory structure, crazy expensive) and git and/or unison (both of which can't cope with many GBs). I'm going to give orifs a try.

jewel 12y ago
Just in case you're not aware, here are some other options: btsync (http://getsync.com http://getsync.com) works well with at least 500GB of photo-sized files. git-annex works well for large collections. I am using this for my own photo collection, which is about 450GB. I like it because you can do partial checkout. I haven't tried it with a large collection but git-annex assistant should work well too, if you're interested in automatic sync.
- icebraining 12y ago
  Another git-annex user with a few hundred GBs. I have my laptop, VPS, nexus tablet and an S3 account being synced without any issues (with a different subset of files in each).
mindslight 12y ago
Unison works great on my >2TB collection. Granted, most files are static and my update bandwidth is a tiny fraction of that, doable over DSL. The only real issues I've found with unison are: Works exclusively in terms of replica pairs. Still, I'd rather have a limited solid tool instead of a flaky perfect one. Lack of inotify support (hacky half/embedded python stuff after the original author lost interest doesn't count). 3x disk access time for copying new files (hash source, transfer, hash dest). This could be reduced to 2x for similar consistency guarantees, but is really only an issue when creating new replicas, and can be fixed by copying the data first. And for use as a 'filesystem', having to come up with file organization / wrapper scripts to choose what is replicated where, while still being sure what is a "full replica".

by nona 12y ago

jewel 12y ago

Just in case you're not aware, here are some other options: btsync (http://getsync.com http://getsync.com) works well with at least 500GB of photo-sized files. git-annex works well for large collections. I am using this for my own photo collection, which is about 450GB. I like it because you can do partial checkout. I haven't tried it with a large collection but git-annex assistant should work well too, if you're interested in automatic sync.

icebraining 12y ago
Another git-annex user with a few hundred GBs. I have my laptop, VPS, nexus tablet and an S3 account being synced without any issues (with a different subset of files in each).

mindslight 12y ago

Unison works great on my >2TB collection. Granted, most files are static and my update bandwidth is a tiny fraction of that, doable over DSL. The only real issues I've found with unison are: Works exclusively in terms of replica pairs. Still, I'd rather have a limited solid tool instead of a flaky perfect one. Lack of inotify support (hacky half/embedded python stuff after the original author lost interest doesn't count). 3x disk access time for copying new files (hash source, transfer, hash dest). This could be reduced to 2x for similar consistency guarantees, but is really only an issue when creating new replicas, and can be fixed by copying the data first. And for use as a 'filesystem', having to come up with file organization / wrapper scripts to choose what is replicated where, while still being sure what is a "full replica".