8 ms·

Build An Opensource Dropbox Clone

86 points by tommynazareth 16y ago

res0nat0r 16y ago
I've looked at this project before, it looks like a nice implementation of rsync with inotify, but it seems to be just for a typical use case for rsync, ie, push your local changes to a remote host. I think if you try to use this like a real dropbox, ie changes occurring on multiple hosts at or around the same time you are going to run in to issues. Also does lysnc handle deleting data on the remote end and propagting those changes out (rsync --del). Also I'm wondering how it will handle conflicts and open files (which it says isn't recommended). I see the .dropbox dir using some type of uuid's or sha1 sigs so I'm sure its doing something more sophisticated to keep things in sync. I've noticed my daemon has died on my linux box every once in a while, and it somehow tracks multiple changes from multiple hosts and replays those transactions correctly and everything rolls up fine.
tommynazareth 16y ago
I don't plan on ditching Dropbox, but I'm definitely interested in having a similar sort of automatic backup with diffs that I can host myself. This looks like it might be an option. Does anyone else handle this in a different way?
- patrickaljord 16y ago
  I just use http://rdiff-backup.nongnu.org/ http://rdiff-backup.nongnu.org/
- gvb 16y ago
  I use rsnapshot (also built on top of rsync/ssh) http://www.rsnapshot.org http://www.rsnapshot.org
- aidanf 16y ago
  I've used Unison (http://www.cis.upenn.edu/~bcpierce/unison/ http://www.cis.upenn.edu/~bcpierce/unison/) to get a dropbox type setup running across several machines.
  - larelli 16y ago
    I've thought about that as well. Could you give some details on the topology used? Did you run unison in automatic mode via cron?
    - durin42 16y ago
      I use unison for my code repositories (I use dropbox for other things) and I don't bother with it in cron. It's been fast enough in my experience that I don't mind waiting, and it sometimes (rarely) requires manual conflict resolution.
dododo 16y ago
lsyncd does the brunt of the work. it uses inotify, which is a facility of the linux kernel, to tell which files should be copied. but i think there is a problem. inotify is not guaranteed to see every file system access. in fact, it often misses them when the file system is busy because there is an upper bound upon the number of inotify events that can be queued, set in /proc/sys/fs/inotify/max_queued_events. i.e., you could get data loss using this mechanism alone. you need to also run rsync every so often, i think. the lsyncd guys are aware of this: http://code.google.com/p/lsyncd/ http://code.google.com/p/lsyncd/ (maybe this is how dropbox works under linux too, if so it presumably has the same problem..)
- beza1e1 16y ago
  Well, Ubuntu One seems to rely on checking every so often alone ...
- noja 16y ago
  Any files not moved elsewhere within X seconds can be handled by a cleanup daemon.
- rw- 16y ago
  inotify is also unable to inherit handlers. after a "mkdir -p a/b/c/d" it is very likely that "d" is untracked. therefore lsyncd is useless on real systems. :(
- brlewis 16y ago
  I would guess Dropbox has this same problem on Linux, though I can't say with certainty. What I can say with certainty is that they're thinking about the problem. As is often the case, you learn a lot about a company through its jobs page. Their latest challenge for applicants looks very much like this problem.
xxpor 16y ago
I don't understand why dropbox doesn't open source their client. Their business isn't built off of their technology, which, while nice, isn't exactly novel. They make their money from selling storage. Open sourcing their client doesn't change the fundamental fact people need the space to store stuff.
- edanm 16y ago
  Showing it doesn't necessarily hurt Dropbox isn't enough. You have to show that it actually helps Dropbox to open-source the client. Otherwise, why should they?
  - xxpor 16y ago
    2 things that I can think of immediately 1. Would increase adoption on Linux clients 2. Increased security (the whole security though obscurity doesn't work thing)
    - edanm 16y ago
      Well, I don't think the Linux market is big enough to make it a compelling argument (but I really don't know; I could be dead wrong). As for security, I'm not sure how big a problem Dropbox has with security. And I would't open-source a client just to get better security for it, but maybe that's just me. Open sourcing the client is betting on the fact that the client doesn't matter. Maybe it doesn't matter, but maybe it does. More importantly, maybe we think it doesn't matter right now, but we'll find out that we were wrong in 2 years, after it's too late.
      - michaelbuckbee 16y ago
        Linux as a desktop client is potentially too small a market to address, I'm not sure the same could be said about the Linux server market.
        edanm 16y ago
please 16y ago
except that it misses 90% of what dropbox does. like multi client sync, cross platform, shared folders, web interface and so on
- fluidcruft 16y ago
  except that it covers 99% of what I actually use dropbox for.
- kgo 16y ago
  Don't forget thirty days of backups and redundant hardware...
gvb 16y ago
What about using unison http://www.cis.upenn.edu/~bcpierce/unison/ http://www.cis.upenn.edu/~bcpierce/unison/ instead of lsyncd? It seems like it would be more resilient. It does look like unison would require a cron job or human trigger where lsyncd is triggered by a file/directory change. On the plus side, it seems like unison would work better in an offline mode.
- res0nat0r 16y ago
  I've tried to use this on syncing my mp3 directory between two boxes and it seemed to be horribly slow. It needs to stat each file for a second or two before it even begins to sync, this takes way too long.
  - dagar 16y ago
    -fastcheck do fast (and slightly unsafe) update detection on windows
    - res0nat0r 16y ago
      I think I remember doing that and it didn't seem to help, I was using my Ubuntu box if that matters.
mikeyg 16y ago
Zimbra's open source edition's WebDAV sharing works well once you get the URLs figured out. You get a lot more fine grained permissions. It's cross platform, there are shared folders, etc.. if that's what you're looking for. You can make folders public to share with others, and so on. With bandwidth as fat as it is these days DAV's heavyness doesn't feel so heavy anymore. Am I the only one that shares these sentiments? Did I mention it works on MacOS 10.6, Linux, and Windows 7? If you're using Dropbox to back up some of these other solutions might make more sense. Crafting up a script using hard links and rsync can give you daily snapshots of any filesystem at only the storage cost of the delta (which can be easily turned into a cron job) .. here's a decent resource for that: http://www.mikerubel.org/computers/rsync_snapshots/ http://www.mikerubel.org/computers/rsync_snapshots/
khingebjerg 16y ago
Old news. http://news.ycombinator.com/item?id=1196421 http://news.ycombinator.com/item?id=1196421
telemachos 16y ago
This is an off-topic short rant. Vote as you like (obvious enough), but skip reading this if you want on-topic discussion. Shoot me if this is how the web is heading. (Recent browsing suggests maybe it is...) We get not only the insufferable "bottom bar" (is there a standard name for those yet?), but also a "top bar", a hover "Click Here to Share" box and a pop-up "Learn More, Instantly" box (not to mention the general clutter and overdense design). Man, that's foul.
- stevelosh 16y ago
  I activated Safari Reader as soon as I saw those walls of 12px text, so I had no idea what you were talking about until I went back and let the page finish loading. "Foul" is not a strong enough word for that. Maybe "vile" is better?
  - deleted 16y ago
    [deleted]
  - mkramlich 16y ago
    Same here. And I love Safari Reader aka Readability! A good rule of thumb I have is that if you "have to put a bag over the head" of something in order to make that thing pleasant to digest or access (whether it's a UI or a bureaucracy or paperwork or a person or physical widget or whatever), then really the source thing itself should be designed that way in the first place. In this case, my own ideal style of web design is very close to what Readability/Reader produces. Thus making it redundant to the extent I've achieved my ideal.
- samstokes 16y ago
  Those bars are even more irritating on a mobile device. The fixed height and position means they obscure half of the actual content and you can't scroll away from them. Do not want.
  - jpr 16y ago
    They also break the "scroll one whole page at a time" paradigm that many are used to. EDIT: I just realized that Chrome devs have apparently already decided that it's not worth fighting the idiotic bars, so Chrome scrolls less than one page.
dochtman 16y ago
I thought http://www.sparkleshare.org/ http://www.sparkleshare.org/ was more interesting.

86 points by tommynazareth 16y ago

I've looked at this project before, it looks like a nice implementation of rsync with inotify, but it seems to be just for a typical use case for rsync, ie, push your local changes to a remote host. I think if you try to use this like a real dropbox, ie changes occurring on multiple hosts at or around the same time you are going to run in to issues. Also does lysnc handle deleting data on the remote end and propagting those changes out (rsync --del). Also I'm wondering how it will handle conflicts and open files (which it says isn't recommended). I see the .dropbox dir using some type of uuid's or sha1 sigs so I'm sure its doing something more sophisticated to keep things in sync. I've noticed my daemon has died on my linux box every once in a while, and it somehow tracks multiple changes from multiple hosts and replays those transactions correctly and everything rolls up fine.

tommynazareth 16y ago

I don't plan on ditching Dropbox, but I'm definitely interested in having a similar sort of automatic backup with diffs that I can host myself. This looks like it might be an option. Does anyone else handle this in a different way?

patrickaljord 16y ago
I just use http://rdiff-backup.nongnu.org/ http://rdiff-backup.nongnu.org/
gvb 16y ago
I use rsnapshot (also built on top of rsync/ssh) http://www.rsnapshot.org http://www.rsnapshot.org
aidanf 16y ago
I've used Unison (http://www.cis.upenn.edu/~bcpierce/unison/ http://www.cis.upenn.edu/~bcpierce/unison/) to get a dropbox type setup running across several machines.
- larelli 16y ago
  I've thought about that as well. Could you give some details on the topology used? Did you run unison in automatic mode via cron?
  - durin42 16y ago
    I use unison for my code repositories (I use dropbox for other things) and I don't bother with it in cron. It's been fast enough in my experience that I don't mind waiting, and it sometimes (rarely) requires manual conflict resolution.

dododo 16y ago

lsyncd does the brunt of the work. it uses inotify, which is a facility of the linux kernel, to tell which files should be copied. but i think there is a problem. inotify is not guaranteed to see every file system access. in fact, it often misses them when the file system is busy because there is an upper bound upon the number of inotify events that can be queued, set in /proc/sys/fs/inotify/max_queued_events. i.e., you could get data loss using this mechanism alone. you need to also run rsync every so often, i think. the lsyncd guys are aware of this: http://code.google.com/p/lsyncd/ http://code.google.com/p/lsyncd/ (maybe this is how dropbox works under linux too, if so it presumably has the same problem..)

beza1e1 16y ago
Well, Ubuntu One seems to rely on checking every so often alone ...
noja 16y ago
Any files not moved elsewhere within X seconds can be handled by a cleanup daemon.
rw- 16y ago
inotify is also unable to inherit handlers. after a "mkdir -p a/b/c/d" it is very likely that "d" is untracked. therefore lsyncd is useless on real systems. :(
brlewis 16y ago
I would guess Dropbox has this same problem on Linux, though I can't say with certainty. What I can say with certainty is that they're thinking about the problem. As is often the case, you learn a lot about a company through its jobs page. Their latest challenge for applicants looks very much like this problem.

xxpor 16y ago

I don't understand why dropbox doesn't open source their client. Their business isn't built off of their technology, which, while nice, isn't exactly novel. They make their money from selling storage. Open sourcing their client doesn't change the fundamental fact people need the space to store stuff.

edanm 16y ago
Showing it doesn't necessarily hurt Dropbox isn't enough. You have to show that it actually helps Dropbox to open-source the client. Otherwise, why should they?
- xxpor 16y ago
  2 things that I can think of immediately 1. Would increase adoption on Linux clients 2. Increased security (the whole security though obscurity doesn't work thing)
  - edanm 16y ago
    Well, I don't think the Linux market is big enough to make it a compelling argument (but I really don't know; I could be dead wrong). As for security, I'm not sure how big a problem Dropbox has with security. And I would't open-source a client just to get better security for it, but maybe that's just me. Open sourcing the client is betting on the fact that the client doesn't matter. Maybe it doesn't matter, but maybe it does. More importantly, maybe we think it doesn't matter right now, but we'll find out that we were wrong in 2 years, after it's too late.
    - michaelbuckbee 16y ago
      Linux as a desktop client is potentially too small a market to address, I'm not sure the same could be said about the Linux server market.
      - edanm 16y ago

please 16y ago

except that it misses 90% of what dropbox does. like multi client sync, cross platform, shared folders, web interface and so on

fluidcruft 16y ago
except that it covers 99% of what I actually use dropbox for.
kgo 16y ago
Don't forget thirty days of backups and redundant hardware...

gvb 16y ago

What about using unison http://www.cis.upenn.edu/~bcpierce/unison/ http://www.cis.upenn.edu/~bcpierce/unison/ instead of lsyncd? It seems like it would be more resilient. It does look like unison would require a cron job or human trigger where lsyncd is triggered by a file/directory change. On the plus side, it seems like unison would work better in an offline mode.

res0nat0r 16y ago
I've tried to use this on syncing my mp3 directory between two boxes and it seemed to be horribly slow. It needs to stat each file for a second or two before it even begins to sync, this takes way too long.
- dagar 16y ago
  -fastcheck do fast (and slightly unsafe) update detection on windows
  - res0nat0r 16y ago
    I think I remember doing that and it didn't seem to help, I was using my Ubuntu box if that matters.

mikeyg 16y ago

Zimbra's open source edition's WebDAV sharing works well once you get the URLs figured out. You get a lot more fine grained permissions. It's cross platform, there are shared folders, etc.. if that's what you're looking for. You can make folders public to share with others, and so on. With bandwidth as fat as it is these days DAV's heavyness doesn't feel so heavy anymore. Am I the only one that shares these sentiments? Did I mention it works on MacOS 10.6, Linux, and Windows 7? If you're using Dropbox to back up some of these other solutions might make more sense. Crafting up a script using hard links and rsync can give you daily snapshots of any filesystem at only the storage cost of the delta (which can be easily turned into a cron job) .. here's a decent resource for that: http://www.mikerubel.org/computers/rsync_snapshots/ http://www.mikerubel.org/computers/rsync_snapshots/

khingebjerg 16y ago

Old news. http://news.ycombinator.com/item?id=1196421 http://news.ycombinator.com/item?id=1196421

telemachos 16y ago

This is an off-topic short rant. Vote as you like (obvious enough), but skip reading this if you want on-topic discussion. Shoot me if this is how the web is heading. (Recent browsing suggests maybe it is...) We get not only the insufferable "bottom bar" (is there a standard name for those yet?), but also a "top bar", a hover "Click Here to Share" box and a pop-up "Learn More, Instantly" box (not to mention the general clutter and overdense design). Man, that's foul.

stevelosh 16y ago
I activated Safari Reader as soon as I saw those walls of 12px text, so I had no idea what you were talking about until I went back and let the page finish loading. "Foul" is not a strong enough word for that. Maybe "vile" is better?
- deleted 16y ago
  [deleted]
- mkramlich 16y ago
  Same here. And I love Safari Reader aka Readability! A good rule of thumb I have is that if you "have to put a bag over the head" of something in order to make that thing pleasant to digest or access (whether it's a UI or a bureaucracy or paperwork or a person or physical widget or whatever), then really the source thing itself should be designed that way in the first place. In this case, my own ideal style of web design is very close to what Readability/Reader produces. Thus making it redundant to the extent I've achieved my ideal.
samstokes 16y ago
Those bars are even more irritating on a mobile device. The fixed height and position means they obscure half of the actual content and you can't scroll away from them. Do not want.
- jpr 16y ago
  They also break the "scroll one whole page at a time" paradigm that many are used to. EDIT: I just realized that Chrome devs have apparently already decided that it's not worth fighting the idiotic bars, so Chrome scrolls less than one page.

dochtman 16y ago

I thought http://www.sparkleshare.org/ http://www.sparkleshare.org/ was more interesting.