4 ms·

FYI, Years ago I worked at a startup which made a distributed, p2p automatic file synchronization system. The startup didn't work out, but the system (mostly)

FYI, Years ago I worked at a startup which made a distributed, p2p automatic file synchronization system. The startup didn't work out, but the system (mostly) worked. We spent a lot of time trying to solve things that are not largely solved, so if I were to start over today, a lot of the hard work would be done. To be more specific, if I were you, I would start by using the p2p connectivity and data channel code built into WebRTC (at webrtc.org) or libjingle (at https://code.google.com/p/libjingle/ https://code.google.com/p/libjingle/). This is what AeroFS (at https://aerofs.com/ https://aerofs.com/) is doing. By the way, writing a distributed file sync system is not that simple, even though it might seem simple at first. Things get complicated really fast. I think that ones reason one of these hasn't taken off is because it's really hard to get right. But, I still hope it will happen one day. Oh, and there is http://camlistore.org/ http://camlistore.org/, which is an open source project run by a really smart Googler. It's not specific to file syncing, but it has a powerful foundation. A combination of the p2p bits from WebRTC and the distributed data bits from camlistore would be pretty powerful. It makes me wish I could find some more 20% time :). (Full disclosure: I work at Google, often on the WebRTC code base, and am the maintainer of libjingle)

burntsushi 13y ago
I can hardly consider myself an amateur with respect to distributed computing, so I am ignorant enough to believe that it is not that complex. Even worse, I won't believe it until I find out for myself. :-) I've looked into camlistore before (particularly since I'm doing this project in Go), but I've been overwhelmed fairly quickly.[1] From the description on the web site though, I do get the sense that it would be incredibly useful---but it feels so heavyweight. My problem is that I think I can get file synchronization working in just a few thousand lines, which is a considerable impetus to chug along without resorting to such a huge dependency. Re libjingle: it looks like an amazing project. But to use it, I would have to violate my so-far-lifelong-commitment not to use C++ (I'm at 11 years now). I will readily admit that whatever I come up with won't even be a fraction as capable as libjingle. And if that doesn't end up being enough, my project will fail. Thanks for your comments though. At the very least, it may provoke me to switch tactics once I become a little less ignorant. [1] - http://camlistore.org/pkg/ http://camlistore.org/pkg/
- pthatcherg 13y ago
  To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly. Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide. By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.
  - burntsushi 13y ago
    > To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly. Those are exactly the things I intend to leave out in my initial work. If it happens to catch on, hopefully I'll get some help. If not, it will still be very useful to me without those features. > Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide. Yup. I think this is one of the things that Bittorrent Sync gets right: strong symmetric AES encrpytion. > By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work. Not to write in it. In the Go community, there's a heavy bias to avoid linking to external libraries if it can be avoided. There is a certain amount of cachet that comes from being able to do `go get ...` and have it work without having to make sure another library is installed. I'm at the infancy stages of my understanding, so it's certainly possible that I'll come to my senses and build on top of something else. But for now, I'm going to blaze my own trail. At least partially to learn. (Two of the most important pieces---encryption and p2p---are things I have very little experience with.)

burntsushi 13y ago

I can hardly consider myself an amateur with respect to distributed computing, so I am ignorant enough to believe that it is not that complex. Even worse, I won't believe it until I find out for myself. :-) I've looked into camlistore before (particularly since I'm doing this project in Go), but I've been overwhelmed fairly quickly.[1] From the description on the web site though, I do get the sense that it would be incredibly useful---but it feels so heavyweight. My problem is that I think I can get file synchronization working in just a few thousand lines, which is a considerable impetus to chug along without resorting to such a huge dependency. Re libjingle: it looks like an amazing project. But to use it, I would have to violate my so-far-lifelong-commitment not to use C++ (I'm at 11 years now). I will readily admit that whatever I come up with won't even be a fraction as capable as libjingle. And if that doesn't end up being enough, my project will fail. Thanks for your comments though. At the very least, it may provoke me to switch tactics once I become a little less ignorant. [1] - http://camlistore.org/pkg/ http://camlistore.org/pkg/

pthatcherg 13y ago
To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly. Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide. By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work.
- burntsushi 13y ago
  > To do p2p without C++, you have to implement ICE or the equivalent. Sounds like a fun thing to do, especially but it's going to take a lot of work. Then once you're done with that, you'll need to implement reliable delivery with congestion control on top, which also would be a lot of fun but a fair amount of work to do properly. Those are exactly the things I intend to leave out in my initial work. If it happens to catch on, hopefully I'll get some help. If not, it will still be very useful to me without those features. > Finally, you'll want to add encryption in there; luckily, Go has good crypto libraries, but getting crypto right is still really hard and it's easy to mess up and have an insecure system. At the very least, you could use libjingle as your guide. Yup. I think this is one of the things that Bittorrent Sync gets right: strong symmetric AES encrpytion. > By the way, what do you mean by "not to use C++"? Do you mean write it in it, or do you mean use libraries written in it? I can understand the former, but the latter seems a bit extreme. Go makes it pretty easy to link to C libraries. Why not use libjingle from Go? That sounds fun and a lot less work. Not to write in it. In the Go community, there's a heavy bias to avoid linking to external libraries if it can be avoided. There is a certain amount of cachet that comes from being able to do `go get ...` and have it work without having to make sure another library is installed. I'm at the infancy stages of my understanding, so it's certainly possible that I'll come to my senses and build on top of something else. But for now, I'm going to blaze my own trail. At least partially to learn. (Two of the most important pieces---encryption and p2p---are things I have very little experience with.)