7 ms·

Building Your Own Dropbox Equivalent

99 points by jnoller 15y ago

pieter 15y ago
This doesn't seem to be anything like dropbox, but more an NFS replacement.
- pmjordan 15y ago
  NFS doesn't mirror the remote data to the local disk, whereas glusterfs allows exactly that (plus some more stuff). http://en.wikipedia.org/wiki/GlusterFS http://en.wikipedia.org/wiki/GlusterFS
  - lloeki 15y ago
    So, is it usable even when offline? How does it handles conflicts once it comes back online? The only mention I seem to find about mirroring is mirroring across multiple servers, but not client<->server (unless you run a server locally on the client too).
- carmen 15y ago
  NFS and Git are exactly why i don't need dropbox. agglomerating large chunks of free space on all devices into universally-accessible paths without wasting any bytes on duplication or synch daemons, and a proper VCS for stuff that i want surgical branching/merging/history and redundancy
m0nty 15y ago
These things are a worthy effort (there have been several such posts recently) but there is no way it's equivalent to Dropbox. Dropbox is cross-platform and has web-based file access and many other benefits. Is Dropbox-like file storage destined to be one of those things (like anti-virus, for example) where there are some fairly good open-source implementations which are never quite as good as the commercial product(s)?
- ghj88 15y ago
  Calm down. Sheesh. He's only giving a tutorial on how to develop a very basic Dropbox. I thought any developer would know that.
  - cjeane 15y ago
    A more useful article would be a how to on setting up secure FTP. Edit: Looks like I was wrong.
  - wzdd 15y ago
    No, the gp has a good point. These articles seem to come up every week on HN. The point is that it's super easy to write something that syncs a folder. You can start with rsync and a cronjob, and just go up from there. Dropbox is fantastic because they haven't solved the easy problem (sync a folder on a single platform when nothing goes wrong), they've solved a very different hard problem -- sync a folder across multiple platforms, deal with corner cases, handle errors gracefully, and wrap the whole thing up in a very easy-to-use GUI. These things aren't dropbox replacements, they are reminders that dropbox is a unique product.
- CPlatypus 15y ago
  This is the difference between equality and equivalence. It's not cross-platform to the extent that Dropbox is, though the fundamentals are portable to any platform that supports FUSE and thus GlusterFS. There's also no web UI, though there are already plenty of web UIs that can be slapped on top of any POSIX directory on a remote server. If you're the sort of Dropbox user who primarily stores and retrieves individual files through the web UI, this is not equivalent. For what I believe to be the more common use case of using Dropbox as a mountable location to/from which to sync files, though, it's quite equivalent. The fact that it's open source, deployable under the user's own control, and actually secure even makes it a better fit for many current Dropbox users' needs. As for "never quite as good" that's just anti-open-source crap. It only works if you define a category by reference to a specific commercial alternative, and use that product's feature set as your checklist. That's no more valid than starting with the open-source alternative's feature set and using that checklist to find the commercial product inferior. It's marketing, not real life. If open source did exactly copy commercial products, then (a) technology wouldn't advance as much, (b) people would bitch about the lack of originality, and (c) the people who make the commercial products would sue. That's a totally unreasonable expectation.
mrleinad 15y ago
Thanks a lot. It's nice to see there are options, although they require a bit extra work, they're worth if you consider your privacy valuable.
deleted 15y ago
[deleted]
pmjordan 15y ago
Does anyone know of a similar setup that allows client-side encryption as well, with some nodes (those in the cloud) acting only as dumb storage that never sees the encryption keys? The only thing I'm aware of is drbd, which acts at the block level, so you can stack dm-crypt and any file system (including OCFS2) on top. As far as I know, drbd only supports 2 nodes, though, which kind of limits the appeal. (I'm specifically not talking about tunneling a normal cluster file system via SSL, which only provides end-to-end encryption)
- gary4gar 15y ago
  Try Wuala, all data is fully encrypted and acts as "dumb storage"
  - pmjordan 15y ago
    I already use that for some stuff, it gets rather expensive for vast amounts of data, and I've already got spare HDD space on servers.
hallman76 15y ago
These articles that describe how to "build your own dropbox" all ignore the client. For me, the beauty of dropbox is the simple, straightforward client. Backing up on my own hardware/cloud might be interesting, but not if I have to monkey around with shell scripts in a CLI. When someone reverse-engineers the dropbox protocol so I can use the official dropbox client on my own infrastructure THAT will be news.
- drdaeman 15y ago
  As I've understood it, it's already (partially) reverse-engineered by dropship. https://github.com/driverdan/dropship/blob/master/dropbox/metadataclient.py https://github.com/driverdan/dropship/blob/master/dropbox/me... Still, I see no reason to use Droxbox servers when you could setup your own.
danfitch 15y ago
One thing I like about products that work... They work, and I don't have to. Which is why I like dropbox.
MarketingMuppet 15y ago
For those interested in building cloud apps we at SpiderOak offer a 'Do it yourself' Storage API that is designed specifically towards backup, sync etc. You can find more info @ https://spideroak.com/blog/20100928101634-spideroak-diy-a-space-efficient-keyvalue-store-for-arbitrarily-large-values-now-in-beta https://spideroak.com/blog/20100928101634-spideroak-diy-a-sp...

99 points by jnoller 15y ago

pieter 15y ago

This doesn't seem to be anything like dropbox, but more an NFS replacement.

pmjordan 15y ago
NFS doesn't mirror the remote data to the local disk, whereas glusterfs allows exactly that (plus some more stuff). http://en.wikipedia.org/wiki/GlusterFS http://en.wikipedia.org/wiki/GlusterFS
- lloeki 15y ago
  So, is it usable even when offline? How does it handles conflicts once it comes back online? The only mention I seem to find about mirroring is mirroring across multiple servers, but not client<->server (unless you run a server locally on the client too).
carmen 15y ago
NFS and Git are exactly why i don't need dropbox. agglomerating large chunks of free space on all devices into universally-accessible paths without wasting any bytes on duplication or synch daemons, and a proper VCS for stuff that i want surgical branching/merging/history and redundancy

m0nty 15y ago

These things are a worthy effort (there have been several such posts recently) but there is no way it's equivalent to Dropbox. Dropbox is cross-platform and has web-based file access and many other benefits. Is Dropbox-like file storage destined to be one of those things (like anti-virus, for example) where there are some fairly good open-source implementations which are never quite as good as the commercial product(s)?

ghj88 15y ago
Calm down. Sheesh. He's only giving a tutorial on how to develop a very basic Dropbox. I thought any developer would know that.
- cjeane 15y ago
  A more useful article would be a how to on setting up secure FTP. Edit: Looks like I was wrong.
- wzdd 15y ago
  No, the gp has a good point. These articles seem to come up every week on HN. The point is that it's super easy to write something that syncs a folder. You can start with rsync and a cronjob, and just go up from there. Dropbox is fantastic because they haven't solved the easy problem (sync a folder on a single platform when nothing goes wrong), they've solved a very different hard problem -- sync a folder across multiple platforms, deal with corner cases, handle errors gracefully, and wrap the whole thing up in a very easy-to-use GUI. These things aren't dropbox replacements, they are reminders that dropbox is a unique product.
CPlatypus 15y ago
This is the difference between equality and equivalence. It's not cross-platform to the extent that Dropbox is, though the fundamentals are portable to any platform that supports FUSE and thus GlusterFS. There's also no web UI, though there are already plenty of web UIs that can be slapped on top of any POSIX directory on a remote server. If you're the sort of Dropbox user who primarily stores and retrieves individual files through the web UI, this is not equivalent. For what I believe to be the more common use case of using Dropbox as a mountable location to/from which to sync files, though, it's quite equivalent. The fact that it's open source, deployable under the user's own control, and actually secure even makes it a better fit for many current Dropbox users' needs. As for "never quite as good" that's just anti-open-source crap. It only works if you define a category by reference to a specific commercial alternative, and use that product's feature set as your checklist. That's no more valid than starting with the open-source alternative's feature set and using that checklist to find the commercial product inferior. It's marketing, not real life. If open source did exactly copy commercial products, then (a) technology wouldn't advance as much, (b) people would bitch about the lack of originality, and (c) the people who make the commercial products would sue. That's a totally unreasonable expectation.

mrleinad 15y ago

Thanks a lot. It's nice to see there are options, although they require a bit extra work, they're worth if you consider your privacy valuable.

deleted 15y ago

[deleted]

pmjordan 15y ago

Does anyone know of a similar setup that allows client-side encryption as well, with some nodes (those in the cloud) acting only as dumb storage that never sees the encryption keys? The only thing I'm aware of is drbd, which acts at the block level, so you can stack dm-crypt and any file system (including OCFS2) on top. As far as I know, drbd only supports 2 nodes, though, which kind of limits the appeal. (I'm specifically not talking about tunneling a normal cluster file system via SSL, which only provides end-to-end encryption)

gary4gar 15y ago
Try Wuala, all data is fully encrypted and acts as "dumb storage"
- pmjordan 15y ago
  I already use that for some stuff, it gets rather expensive for vast amounts of data, and I've already got spare HDD space on servers.

hallman76 15y ago

These articles that describe how to "build your own dropbox" all ignore the client. For me, the beauty of dropbox is the simple, straightforward client. Backing up on my own hardware/cloud might be interesting, but not if I have to monkey around with shell scripts in a CLI. When someone reverse-engineers the dropbox protocol so I can use the official dropbox client on my own infrastructure THAT will be news.

drdaeman 15y ago
As I've understood it, it's already (partially) reverse-engineered by dropship. https://github.com/driverdan/dropship/blob/master/dropbox/metadataclient.py https://github.com/driverdan/dropship/blob/master/dropbox/me... Still, I see no reason to use Droxbox servers when you could setup your own.

danfitch 15y ago

One thing I like about products that work... They work, and I don't have to. Which is why I like dropbox.

MarketingMuppet 15y ago

For those interested in building cloud apps we at SpiderOak offer a 'Do it yourself' Storage API that is designed specifically towards backup, sync etc. You can find more info @ https://spideroak.com/blog/20100928101634-spideroak-diy-a-space-efficient-keyvalue-store-for-arbitrarily-large-values-now-in-beta https://spideroak.com/blog/20100928101634-spideroak-diy-a-sp...