7 ms·

> This whole NSA thing should be enough to push things like named data networking/content centric networking You're damn right. It wasn't even a few days after

> This whole NSA thing should be enough to push things like named data networking/content centric networking You're damn right. It wasn't even a few days after this whole thing broke that I started working on a simple, distributed and automatic file synchronization program. No third parties and open source. I was absolutely stunned to find out that nothing out there meets those criteria (despite my best efforts). The only thing that was close was git-annex assistant, but I don't want to use git for simple file synchronization (particularly for large files). There's also Bittorrent Sync, but it's completely locked down. :-/

pwnna 13y ago
This is all good but the biggest thing right now is that there is no coordinated effort to build a system that's easy to use, secure, and open source. Everyone can spend their time working on their own solution but no one would use it as everyone's work will be incompatible with each other. This means no one (or rather, not enough people) will use any solutions that are developed and revert back to something like Skype, FB message, etc. I think what we need is to establish a foundation that coordinates the development efforts for these secure and reliable software packages with a few centralized projects that become a standard (here's hope for WebRTC).
Fixnum 13y ago
Out of curiosity, what's wrong with Unison?
- burntsushi 13y ago
  That's what I use now. But it's not automatic and is designed for use between two end points. I want it to be automatic across N end points. (And I intend to use a bittorrent-like protocol to achieve that, rather than an rsync-like protocol that unison uses.) [EDIT] I forgot one of my most frustrating nits with unison: you need to have the same version on both ends. It's solvable by building the right version yourself, but it's damn annoying.
  - erikpukinskis 13y ago
    Wouldn't it be easier to contribute those changes to Unison instead of starting over?
    - burntsushi 13y ago
      Dunno. They feel like pretty big changes to me. I imagine it'd have to be completely gutted since the entire protocol would have to change. Plus, I am not fond at all of its interface/configuration. Very complex.
  - VLM 13y ago
    "I want it to be automatic across N end points." I solved that problem, using unison, I guess near 15 years ago the unix way, not the windows way. Its been awhile but with a little symlink fun you can make a little shell script that builds a partial mesh and syncs both the script itself, its list of possible mesh members (as opposed to the 3 or so I used) and all the other stuff, more or less silently run out of .bash_profile or .bashrc or whatever it was. There's already excellent tools in unix for scripting, automation, batch, etc, so no need to include them all into one tool. The windows way, would be to put all those tools into one (buggy) executable with a gui and an installer that also installs browser toolbars or whatever. I switched to git away from unison and never looked back. Eventually it got to be too big of a PITA to install 2 or 3 debian packages of different versions of unison all of which can only talk to certain other versions. So then my shell script had to detect which unison versions are installed on this node, and match up to the master list of other nodes and their versions, ... oh forget all this garbage just try this new subversion svn thing. And later, forget this whole script, just put everything important thats small in git and everything thats big in multiple AFS servers running on RAID.
    - burntsushi 13y ago
      Yes, most of my stuff is kept in sync with git. But there are many things I want to keep in sync that I don't want to check into git. I'd rather they just sync automatically without intervention from me.
endgame 13y ago
Your synchronisation project sounds intersting, do you have source code online yet?
- burntsushi 13y ago
  Not yet. I have a piece or two on my github, but otherwise it hasn't been written yet. I'll be sure to post here when I'm done, but I've never had much luck with that in the past.
lukifer 13y ago
> There's also Bittorrent Sync, but it's completely locked down. This really surprised me; for some reason, I assumed it was open source. Since the storage is entirely peer-to-peer, I wonder what their business model is?
- burntsushi 13y ago
  Who knows. It's completely opaque to me. I just don't want to be left high-and-dry if the maintainers decide to abandon the project. Plus, I want to be able to tinker, of course. :-) [EDIT] This was all I could find.[1] Which isn't much. [1] - http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-source/ http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-...
- mtgx 13y ago
  Not open source, probably because the bittorrent protocol isn't open source.
  - lukifer 13y ago
    How so? Several open-source clients exist. Were they reverse-engineered?
  - goblin89 13y ago
    I believe that's not correct. First a nitpick, I think it's only software that can be open or closed source. Protocol specification might be kept secret, though. Regarding specifically BitTorrent protocol specification[0], it's free to use. Anyone can build their open or closed source BitTorrent client without reverse engineering. [0] http://www.bittorrent.org/beps/bep_0003.html http://www.bittorrent.org/beps/bep_0003.html
  - joshaidan 13y ago
    I believe the first bittorrent client was written in python. http://en.wikipedia.org/wiki/Bram_Cohen http://en.wikipedia.org/wiki/Bram_Cohen
icebraining 13y ago
Just for checking: git-annex doesn't actually check the files into the git repository, so whether the file is small or large is irrelevant to it. Git is only used to keep track of the filenames and their locations.
- burntsushi 13y ago
  Interesting. The web site really makes it sound otherwise:[1] Then any changes you make to its folder will automatically be committed to git, and synced to repositories on other computers. Nevertheless, I checked it out myself and you are indeed correct. I guess that means I'm out of legitimate excuses? :P [1] - http://git-annex.branchable.com/assistant/quickstart/ http://git-annex.branchable.com/assistant/quickstart/
  - gioele 13y ago
    > Then any changes you make to its folder will automatically be committed to git The _changes_ are tracked (addition, removals, etc.) not the content.
    - burntsushi 13y ago
      Well, yes...... I know that now. But my initial reading did not catch that. Hence the confusion.
whadar 13y ago
You can try the open sourced project, webrtc based, Sharefest.me for filesharing p2p with no cloud storage...
davedx 13y ago
There's a WebRTC demo that does something like this. I can't seem to find the link, but it was linked from one of the official/reference implementation sites.
pthatcherg 13y ago
FYI, Years ago I worked at a startup which made a distributed, p2p automatic file synchronization system. The startup didn't work out, but the system (mostly) worked. We spent a lot of time trying to solve things that are not largely solved, so if I were to start over today, a lot of the hard work would be done. To be more specific, if I were you, I would start by using the p2p connectivity and data channel code built into WebRTC (at webrtc.org) or libjingle (at https://code.google.com/p/libjingle/ https://code.google.com/p/libjingle/). This is what AeroFS (at https://aerofs.com/ https://aerofs.com/) is doing. By the way, writing a distributed file sync system is not that simple, even though it might seem simple at first. Things get complicated really fast. I think that ones reason one of these hasn't taken off is because it's really hard to get right. But, I still hope it will happen one day. Oh, and there is http://camlistore.org/ http://camlistore.org/, which is an open source project run by a really smart Googler. It's not specific to file syncing, but it has a powerful foundation. A combination of the p2p bits from WebRTC and the distributed data bits from camlistore would be pretty powerful. It makes me wish I could find some more 20% time :). (Full disclosure: I work at Google, often on the WebRTC code base, and am the maintainer of libjingle)
- burntsushi 13y ago
  I can hardly consider myself an amateur with respect to distributed computing, so I am ignorant enough to believe that it is not that complex. Even worse, I won't believe it until I find out for myself. :-) I've looked into camlistore before (particularly since I'm doing this project in Go), but I've been overwhelmed fairly quickly.[1] From the description on the web site though, I do get the sense that it would be incredibly useful---but it feels so heavyweight. My problem is that I think I can get file synchronization working in just a few thousand lines, which is a considerable impetus to chug along without resorting to such a huge dependency. Re libjingle: it looks like an amazing project. But to use it, I would have to violate my so-far-lifelong-commitment not to use C++ (I'm at 11 years now). I will readily admit that whatever I come up with won't even be a fraction as capable as libjingle. And if that doesn't end up being enough, my project will fail. Thanks for your comments though. At the very least, it may provoke me to switch tactics once I become a little less ignorant. [1] - http://camlistore.org/pkg/ http://camlistore.org/pkg/
Scaevolus 13y ago
Have you looked into CurveCP? It would provide security and automatic bandwidth throttling with minimal effort.

pwnna 13y ago

This is all good but the biggest thing right now is that there is no coordinated effort to build a system that's easy to use, secure, and open source. Everyone can spend their time working on their own solution but no one would use it as everyone's work will be incompatible with each other. This means no one (or rather, not enough people) will use any solutions that are developed and revert back to something like Skype, FB message, etc. I think what we need is to establish a foundation that coordinates the development efforts for these secure and reliable software packages with a few centralized projects that become a standard (here's hope for WebRTC).

Fixnum 13y ago

Out of curiosity, what's wrong with Unison?

burntsushi 13y ago
That's what I use now. But it's not automatic and is designed for use between two end points. I want it to be automatic across N end points. (And I intend to use a bittorrent-like protocol to achieve that, rather than an rsync-like protocol that unison uses.) [EDIT] I forgot one of my most frustrating nits with unison: you need to have the same version on both ends. It's solvable by building the right version yourself, but it's damn annoying.
- erikpukinskis 13y ago
  Wouldn't it be easier to contribute those changes to Unison instead of starting over?
  - burntsushi 13y ago
    Dunno. They feel like pretty big changes to me. I imagine it'd have to be completely gutted since the entire protocol would have to change. Plus, I am not fond at all of its interface/configuration. Very complex.
- VLM 13y ago
  "I want it to be automatic across N end points." I solved that problem, using unison, I guess near 15 years ago the unix way, not the windows way. Its been awhile but with a little symlink fun you can make a little shell script that builds a partial mesh and syncs both the script itself, its list of possible mesh members (as opposed to the 3 or so I used) and all the other stuff, more or less silently run out of .bash_profile or .bashrc or whatever it was. There's already excellent tools in unix for scripting, automation, batch, etc, so no need to include them all into one tool. The windows way, would be to put all those tools into one (buggy) executable with a gui and an installer that also installs browser toolbars or whatever. I switched to git away from unison and never looked back. Eventually it got to be too big of a PITA to install 2 or 3 debian packages of different versions of unison all of which can only talk to certain other versions. So then my shell script had to detect which unison versions are installed on this node, and match up to the master list of other nodes and their versions, ... oh forget all this garbage just try this new subversion svn thing. And later, forget this whole script, just put everything important thats small in git and everything thats big in multiple AFS servers running on RAID.
  - burntsushi 13y ago
    Yes, most of my stuff is kept in sync with git. But there are many things I want to keep in sync that I don't want to check into git. I'd rather they just sync automatically without intervention from me.

endgame 13y ago

Your synchronisation project sounds intersting, do you have source code online yet?

burntsushi 13y ago
Not yet. I have a piece or two on my github, but otherwise it hasn't been written yet. I'll be sure to post here when I'm done, but I've never had much luck with that in the past.

lukifer 13y ago

> There's also Bittorrent Sync, but it's completely locked down. This really surprised me; for some reason, I assumed it was open source. Since the storage is entirely peer-to-peer, I wonder what their business model is?

burntsushi 13y ago
Who knows. It's completely opaque to me. I just don't want to be left high-and-dry if the maintainers decide to abandon the project. Plus, I want to be able to tinker, of course. :-) [EDIT] This was all I could find.[1] Which isn't much. [1] - http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-source/ http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-...
mtgx 13y ago
Not open source, probably because the bittorrent protocol isn't open source.
- lukifer 13y ago
  How so? Several open-source clients exist. Were they reverse-engineered?
- goblin89 13y ago
  I believe that's not correct. First a nitpick, I think it's only software that can be open or closed source. Protocol specification might be kept secret, though. Regarding specifically BitTorrent protocol specification[0], it's free to use. Anyone can build their open or closed source BitTorrent client without reverse engineering. [0] http://www.bittorrent.org/beps/bep_0003.html http://www.bittorrent.org/beps/bep_0003.html
- joshaidan 13y ago
  I believe the first bittorrent client was written in python. http://en.wikipedia.org/wiki/Bram_Cohen http://en.wikipedia.org/wiki/Bram_Cohen

icebraining 13y ago

Just for checking: git-annex doesn't actually check the files into the git repository, so whether the file is small or large is irrelevant to it. Git is only used to keep track of the filenames and their locations.

burntsushi 13y ago
Interesting. The web site really makes it sound otherwise:[1] Then any changes you make to its folder will automatically be committed to git, and synced to repositories on other computers. Nevertheless, I checked it out myself and you are indeed correct. I guess that means I'm out of legitimate excuses? :P [1] - http://git-annex.branchable.com/assistant/quickstart/ http://git-annex.branchable.com/assistant/quickstart/
- gioele 13y ago
  > Then any changes you make to its folder will automatically be committed to git The _changes_ are tracked (addition, removals, etc.) not the content.
  - burntsushi 13y ago
    Well, yes...... I know that now. But my initial reading did not catch that. Hence the confusion.

whadar 13y ago

You can try the open sourced project, webrtc based, Sharefest.me for filesharing p2p with no cloud storage...

davedx 13y ago

There's a WebRTC demo that does something like this. I can't seem to find the link, but it was linked from one of the official/reference implementation sites.

pthatcherg 13y ago

FYI, Years ago I worked at a startup which made a distributed, p2p automatic file synchronization system. The startup didn't work out, but the system (mostly) worked. We spent a lot of time trying to solve things that are not largely solved, so if I were to start over today, a lot of the hard work would be done. To be more specific, if I were you, I would start by using the p2p connectivity and data channel code built into WebRTC (at webrtc.org) or libjingle (at https://code.google.com/p/libjingle/ https://code.google.com/p/libjingle/). This is what AeroFS (at https://aerofs.com/ https://aerofs.com/) is doing. By the way, writing a distributed file sync system is not that simple, even though it might seem simple at first. Things get complicated really fast. I think that ones reason one of these hasn't taken off is because it's really hard to get right. But, I still hope it will happen one day. Oh, and there is http://camlistore.org/ http://camlistore.org/, which is an open source project run by a really smart Googler. It's not specific to file syncing, but it has a powerful foundation. A combination of the p2p bits from WebRTC and the distributed data bits from camlistore would be pretty powerful. It makes me wish I could find some more 20% time :). (Full disclosure: I work at Google, often on the WebRTC code base, and am the maintainer of libjingle)

burntsushi 13y ago
I can hardly consider myself an amateur with respect to distributed computing, so I am ignorant enough to believe that it is not that complex. Even worse, I won't believe it until I find out for myself. :-) I've looked into camlistore before (particularly since I'm doing this project in Go), but I've been overwhelmed fairly quickly.[1] From the description on the web site though, I do get the sense that it would be incredibly useful---but it feels so heavyweight. My problem is that I think I can get file synchronization working in just a few thousand lines, which is a considerable impetus to chug along without resorting to such a huge dependency. Re libjingle: it looks like an amazing project. But to use it, I would have to violate my so-far-lifelong-commitment not to use C++ (I'm at 11 years now). I will readily admit that whatever I come up with won't even be a fraction as capable as libjingle. And if that doesn't end up being enough, my project will fail. Thanks for your comments though. At the very least, it may provoke me to switch tactics once I become a little less ignorant. [1] - http://camlistore.org/pkg/ http://camlistore.org/pkg/

Scaevolus 13y ago

Have you looked into CurveCP? It would provide security and automatic bandwidth throttling with minimal effort.