6 ms·

Dropship — successor to torrents?

127 points by herbatnic 15y ago

Stuk 15y ago
From "How does it work?" in the Readme: Dropbox its deduplication scheme works by breaking files into blocks. Each of these blocks is hashed with the SHA256__ algorithm and represented by the digest. Only blocks that are not yet known are uploaded to the server when syncing. By using the same API as the native client, Dropship pretends to sync a file to the dropbox folder without actually having the contents. This bluff succeeds because the only proof needed server-side is the hash of each 4MB block of the file, which is known. The server then adds the file metadata to the folder, which is, as usual, propagated to all clients. These will then start downloading the file. It looks like the Github repo was deleted a few hours ago, but the direct download link still works.
- lachyg 15y ago
  I still don't get it. Anyone willing to explain?
  - russell_h 15y ago
    Somewhere on a Dropbox server is a file that you want. Normally the file's owner would have to share that file with you in order for it to appear in your Dropbox. But if you know the hash of file you can trick Dropbox into thinking you already have the file and are just adding it to your Dropbox. Apparently Dropbox notices they already have that file, and instead of you uploading it they just make it appear in your account. Then you can download it. I wasn't impressed by the OP, but this is actually a really cool hack.
  - saulrh 15y ago
    Dropbox avoids having to store multiple copies of huge files by detecting duplicated files, storing only one copy, and letting every user that stores the file download from that one copy. Dropship exploits this system for filesharing by lying to the Dropbox servers and saying that it already owns a copy of the file. For example, Person A wants to distribute a copy of a CD or something. They upload the file to Dropbox normally. They then use Dropship to create something describing that file, which they then publish. Persons B and C download that descriptor and feed it to Dropship, which tricks Dropbox into thinking that they also own copies of the file. Dropbox then lets Person B and Person C download the file that Person A wanted to distribute, and mission accomplished. It's all very clever. I like it.
  - llimllib 15y ago
    I'll try. Let's say you want to upload files A and B to Dropbox from your computer. A is a 3mb file, and B is a 12mb file. The dropbox client first looks at A, sees that it's <4mb, and so hashes[1] the whole file. That means that it runs a function which turns the file into a 256-byte string (a "hash") which is unique[2] to that file. The client then sends that hash to the server, which checks to see if it has already seen that hash. If it has, then it assumes that it already has the file, and just copies it from the previous location where it stored the block with that hash. If it hasn't, then it goes ahead and uploads the file. The process for uploading file B is very similar, except that the client breaks it into three 4mb blocks, hashes each of those, and sends the hash to the server to see if it's already received those blocks. Phew. OK, now we can get to why Dropship is (was?) a neat hack. The idea is, if Alfred has uploaded file C, and Barbara wants to get a hold of file C, but doesn't want to download it, she can just send the dropbox server the hashes for each 4mb block of file C. The server will see each hash, say "ahha! I've already got the block represented by that hash, so I won't make you upload it!", and put the file in Barbara's Dropbox. Does that make sense? [1]: http://en.wikipedia.org/wiki/Hash_function http://en.wikipedia.org/wiki/Hash_function [2]: Not really unique, but the idea of hash functions is that we turn each input into a "hash" which is really really really likely to be unique, so likely that we can treat it as unique.
schrototo 15y ago
This isn't even remotely similar to bittorrent, it's more akin to rapidshare et al. You're completely at the whim of a 3rd party, Dropbox. And I'm pretty sure something like this would violate whatever contract you agree to when signing up. So no, not a successor to torrents.
- saulrh 15y ago
  Agreed. It's still a really neat hack, though.
- dbuxton 15y ago
  I'm pretty sure I've seen stuff suggesting that Dropbox occasionally purges copyrighted files from its system (which is made super-easy by the hash fingerprinting system that it uses to deduplicate storage) so I agree that this is not likely to meet most people's use case for torrents; i.e. stealing copyrighted music and films. I wonder why the github repo has been taken down.
  - ugh 15y ago
    How do they decide which copyrighted content to delete? The files in the Dropbox are by default not public. Merely having copyrighted files in your Dropbox is certainly no violation of copyright law. At which point does it become illegal? Is sharing it with one or two people ok? I would think that even putting it in your public folder is not necessarily illegal: What if you don't share the link publicly (or only with one or two people)? Services like Rapidshare thrive on those ambiguities. They let you upload any file and give you a link, only after this link really becomes public will they take down copyrighted content (which introduces a time delay). I have actually never seen that happen with Dropbox links (which, I think, is the right strategy for them: It would be bad for their brand if they were to become "that piracy website"), so they must be doing something different.
    - ZoFreX 15y ago
      > Merely having copyrighted files in your Dropbox is certainly no violation of copyright law. Actually, it could be. Copyright means exactly that: the right to copy.
tlrobinson 15y ago
From the README, in case it wasn't obvious: "These utilities make use of the deduplication scheme of Dropbox__ to allow for "teleporting" files into your Dropbox account given only a list of hashes, provided of course that the files already exist on their servers. This enables arbitrary, anonymous transfers of files between Dropbox accounts." Between this and the minor information leakage issue I suspect Dropbox will be making changes to their deduplication scheme. A simple way to fix both of these issues is to require each user to upload the complete file once, regardless of whether Dropbox already has it stored. Deduplication in storage and per-user uploading is still possible. Also interesting to note is the Github repo for this has been deleted. Tarball of the source is still available.
- limmeau 15y ago
  Napkin-cryptographic way how Dropbox could fix this while still getting full deduplication: currently, when the client discovers that a file has been added locally, it sends hashes of 4MB blocks, and the server considers the file added. Additional measure at that point: the server could challenge the client to provide the values of bytes at a couple of arbitrarily chosen byte offsets of the original file. (Could precompute that, provided the queries don't repeat often).
  - alex1 15y ago
    What would stop pirates from querying each other (maybe on some P2P network) for those random bytes? Client A wants the file that Client B has so when Dropbox asks Client A for some random offset, Client A asks Client B in the background and relays the result to Dropbox. It really depends on how far pirates would be willing to go.
    - limmeau 15y ago
      Of course, Dropbox can't prevent people from sharing content out of band. But if Client A and Client B are offering arbitrary byte ranges to complete strangers, they are effectively playing BitTorrent again.
steipete 15y ago
Here's a mirror: https://github.com/steipete/dropship https://github.com/steipete/dropship
- driverdan 15y ago
  And here's a tar mirror on Dropbox: http://dl.dropbox.com/u/7562426/laanwj-dropship-464e1c4.tar.gz http://dl.dropbox.com/u/7562426/laanwj-dropship-464e1c4.tar....
  - drdaeman 15y ago
    > Restricted Content > This file is no longer available. For additional information contact Dropbox Support. So, Dropbox has censorship? Ni-i-ice.
    - driverdan 15y ago
      Dear Dropbox User: We have received a notification under the Digital Millennium Copyright Act ("DMCA") from Dropbox that the following material is claimed to be infringing. /Public/laanwj-dropship-464e1c4.tar.gz Accordingly, pursuant to Section 512(c)(1)(C) of DMCA, we have removed or disabled access to the material that is claimed to be infringing or to be the subject of infringing activity. ----------------- This is BULLSHIT! Dropbox is censoring this because they don't want it to get out there. What will they censor next?
      - drdaeman 15y ago
        Wow. Either Dropbox have some copyright issues with Dropship's code, or this is just a blatant misuse of DMCA to take down the content. Won't speculate, but I personally suspect the latter. Wladimir did release the software under FOSS Expat ("MIT") license so he can't really take it back. It's now up to good will of others. While I understand that this may put Dropbox in unfortunate situation, such methods to take down the problematic piece of software somehow feel wrong.
- w1ntermute 15y ago
joshbaptiste 15y ago
Indeed, great idea/hack but the only issue I can foresee is Dropship becoming very popular for illegal file duplication, bringing forth the attention of the RIAA/publishers etc.. causing legal headaches for our beloved Dropbox.
orofino 15y ago
forgive what is possibly a very ignorant question, but are there security concerns here? I understand that the key space is immensely huge and that for any file over 4MB in size it would be virtually impossible to guess, but what is to stop someone from just trying hashes for fun to see if they get interesting files? Like I said for file over 4MB it seems fine, guessing sequential hashes would be all but impossible. I assume the realistic solution is just to encrypt my files (preferably in a truecrypt volume over 4MB in size) if I'm truly concerned. On a side note, it would be interesting to see if this could be modified to tell me how unique my overall file set is.
- power78 15y ago
  I see your point here. I hope non-public files are protected from Dropbox's deduplication
  - valindar 15y ago
    They aren't. A colleague copied a whole bunch of documentation from his private Dropbox onto my computer; when I then copied it into my Dropbox it took around half a minute to sync and it was a couple hundred MB.
antimatter15 15y ago
It's a novel exploit of deduplication, but I don't see how it's practically any better than moving a file into the /Public directory and handing them a URL.
- orofino 15y ago
  In theory, developing this a bit more would lead to searchable directories of files and an easy way to retrieve them, very similar to a torrent tracker. That has significant benefit over shared files and I have to think would scare the heck out of dropbox because of the ire it might bring upon them. This would have to be a worst nightmare for them. Although removing deduplication would solve it for them (with significant increase in what has to be stored).
- limmeau 15y ago
  Consider 1) User buys a file from the rights owner and explorers it into his Dropbox 2) User obtains a blockwise hash of the file and runs dropship on it 3) User obtains a public URL from somebody who has the file and downloads the file from the Dropbox web server The point is that the Dropbox server cannot distinguish 1 from 2, but both from 3. Therefore, 2 should be more robust against takedown notices than 3.
- Pahalial 15y ago
  Dropbox has bandwidth limits on /Public URLs, particularly low for free accounts. This wholly circumvents that, and I suspect that's the real reason Arash asked for a takedown, not so much the loose association with piracy.
  - dr_win 15y ago
    Dropbox could do the same for hashes. For example internet cannot download more than 400MB from a hash (4MB) per day.
EGreg 15y ago
It's centralized, not distributed. You are at the whim of dropbox. This is not a successor to torrents at all.
Jarred 15y ago
I think the real successor to torrents was actually its predecessor, and that's binary usenet files. Download speeds are bottleknecked at your own downstream, most providers have SSL support for encryption of everything you download, and there's a plethora of content. People don't really know about it though
- kin 15y ago
  absolutely. instead of leaving a movie download overnight i leave it for a shower. i hope it never hits mainstream, especially since it's subscription based
- ippisl 15y ago
  From my experience rapidshare like sites have much more content than usenet. is there a some hidden way to to access vast content in usenet ?
deleted 15y ago
[deleted]
JeremyBanks 15y ago
Does anybody have another mirror?
- deleted 15y ago
  [deleted]
power78 15y ago
So who is game for setting up a repository site of json hash files?
Aissen 15y ago
At last. I'm surprised nobody thought of implementing this earlier. I thought about it, but this being a direct attack on Dropbox, I don't see much value in it. Apart from being unethical, it will only force Dropbox to either remove this very useful feature, or implement a challenge-like system which will render this useless. This will be short-lived code if it spreads. In fact, I think this "feature" is one of the (many) reasons why Dropbox doesn't have an opensource client. And it isn't exposed it in its so-called "API". Edit: I just saw that they killed the feature: http://news.ycombinator.com/item?id=2483053 http://news.ycombinator.com/item?id=2483053
gbraad 15y ago
A lot to do about this posting: http://razorfast.com/2011/04/25/dropbox-attempts-to-kill-open-source-project/ http://razorfast.com/2011/04/25/dropbox-attempts-to-kill-ope...

127 points by herbatnic 15y ago

Stuk 15y ago

From "How does it work?" in the Readme: Dropbox its deduplication scheme works by breaking files into blocks. Each of these blocks is hashed with the SHA256__ algorithm and represented by the digest. Only blocks that are not yet known are uploaded to the server when syncing. By using the same API as the native client, Dropship pretends to sync a file to the dropbox folder without actually having the contents. This bluff succeeds because the only proof needed server-side is the hash of each 4MB block of the file, which is known. The server then adds the file metadata to the folder, which is, as usual, propagated to all clients. These will then start downloading the file. It looks like the Github repo was deleted a few hours ago, but the direct download link still works.

lachyg 15y ago
I still don't get it. Anyone willing to explain?
- russell_h 15y ago
  Somewhere on a Dropbox server is a file that you want. Normally the file's owner would have to share that file with you in order for it to appear in your Dropbox. But if you know the hash of file you can trick Dropbox into thinking you already have the file and are just adding it to your Dropbox. Apparently Dropbox notices they already have that file, and instead of you uploading it they just make it appear in your account. Then you can download it. I wasn't impressed by the OP, but this is actually a really cool hack.
- saulrh 15y ago
  Dropbox avoids having to store multiple copies of huge files by detecting duplicated files, storing only one copy, and letting every user that stores the file download from that one copy. Dropship exploits this system for filesharing by lying to the Dropbox servers and saying that it already owns a copy of the file. For example, Person A wants to distribute a copy of a CD or something. They upload the file to Dropbox normally. They then use Dropship to create something describing that file, which they then publish. Persons B and C download that descriptor and feed it to Dropship, which tricks Dropbox into thinking that they also own copies of the file. Dropbox then lets Person B and Person C download the file that Person A wanted to distribute, and mission accomplished. It's all very clever. I like it.
- llimllib 15y ago
  I'll try. Let's say you want to upload files A and B to Dropbox from your computer. A is a 3mb file, and B is a 12mb file. The dropbox client first looks at A, sees that it's <4mb, and so hashes[1] the whole file. That means that it runs a function which turns the file into a 256-byte string (a "hash") which is unique[2] to that file. The client then sends that hash to the server, which checks to see if it has already seen that hash. If it has, then it assumes that it already has the file, and just copies it from the previous location where it stored the block with that hash. If it hasn't, then it goes ahead and uploads the file. The process for uploading file B is very similar, except that the client breaks it into three 4mb blocks, hashes each of those, and sends the hash to the server to see if it's already received those blocks. Phew. OK, now we can get to why Dropship is (was?) a neat hack. The idea is, if Alfred has uploaded file C, and Barbara wants to get a hold of file C, but doesn't want to download it, she can just send the dropbox server the hashes for each 4mb block of file C. The server will see each hash, say "ahha! I've already got the block represented by that hash, so I won't make you upload it!", and put the file in Barbara's Dropbox. Does that make sense? [1]: http://en.wikipedia.org/wiki/Hash_function http://en.wikipedia.org/wiki/Hash_function [2]: Not really unique, but the idea of hash functions is that we turn each input into a "hash" which is really really really likely to be unique, so likely that we can treat it as unique.

schrototo 15y ago

This isn't even remotely similar to bittorrent, it's more akin to rapidshare et al. You're completely at the whim of a 3rd party, Dropbox. And I'm pretty sure something like this would violate whatever contract you agree to when signing up. So no, not a successor to torrents.

saulrh 15y ago
Agreed. It's still a really neat hack, though.
dbuxton 15y ago
I'm pretty sure I've seen stuff suggesting that Dropbox occasionally purges copyrighted files from its system (which is made super-easy by the hash fingerprinting system that it uses to deduplicate storage) so I agree that this is not likely to meet most people's use case for torrents; i.e. stealing copyrighted music and films. I wonder why the github repo has been taken down.
- ugh 15y ago
  How do they decide which copyrighted content to delete? The files in the Dropbox are by default not public. Merely having copyrighted files in your Dropbox is certainly no violation of copyright law. At which point does it become illegal? Is sharing it with one or two people ok? I would think that even putting it in your public folder is not necessarily illegal: What if you don't share the link publicly (or only with one or two people)? Services like Rapidshare thrive on those ambiguities. They let you upload any file and give you a link, only after this link really becomes public will they take down copyrighted content (which introduces a time delay). I have actually never seen that happen with Dropbox links (which, I think, is the right strategy for them: It would be bad for their brand if they were to become "that piracy website"), so they must be doing something different.
  - ZoFreX 15y ago
    > Merely having copyrighted files in your Dropbox is certainly no violation of copyright law. Actually, it could be. Copyright means exactly that: the right to copy.

tlrobinson 15y ago

From the README, in case it wasn't obvious: "These utilities make use of the deduplication scheme of Dropbox__ to allow for "teleporting" files into your Dropbox account given only a list of hashes, provided of course that the files already exist on their servers. This enables arbitrary, anonymous transfers of files between Dropbox accounts." Between this and the minor information leakage issue I suspect Dropbox will be making changes to their deduplication scheme. A simple way to fix both of these issues is to require each user to upload the complete file once, regardless of whether Dropbox already has it stored. Deduplication in storage and per-user uploading is still possible. Also interesting to note is the Github repo for this has been deleted. Tarball of the source is still available.

limmeau 15y ago
Napkin-cryptographic way how Dropbox could fix this while still getting full deduplication: currently, when the client discovers that a file has been added locally, it sends hashes of 4MB blocks, and the server considers the file added. Additional measure at that point: the server could challenge the client to provide the values of bytes at a couple of arbitrarily chosen byte offsets of the original file. (Could precompute that, provided the queries don't repeat often).
- alex1 15y ago
  What would stop pirates from querying each other (maybe on some P2P network) for those random bytes? Client A wants the file that Client B has so when Dropbox asks Client A for some random offset, Client A asks Client B in the background and relays the result to Dropbox. It really depends on how far pirates would be willing to go.
  - limmeau 15y ago
    Of course, Dropbox can't prevent people from sharing content out of band. But if Client A and Client B are offering arbitrary byte ranges to complete strangers, they are effectively playing BitTorrent again.

steipete 15y ago

Here's a mirror: https://github.com/steipete/dropship https://github.com/steipete/dropship

driverdan 15y ago
And here's a tar mirror on Dropbox: http://dl.dropbox.com/u/7562426/laanwj-dropship-464e1c4.tar.gz http://dl.dropbox.com/u/7562426/laanwj-dropship-464e1c4.tar....
- drdaeman 15y ago
  > Restricted Content > This file is no longer available. For additional information contact Dropbox Support. So, Dropbox has censorship? Ni-i-ice.
  - driverdan 15y ago
    Dear Dropbox User: We have received a notification under the Digital Millennium Copyright Act ("DMCA") from Dropbox that the following material is claimed to be infringing. /Public/laanwj-dropship-464e1c4.tar.gz Accordingly, pursuant to Section 512(c)(1)(C) of DMCA, we have removed or disabled access to the material that is claimed to be infringing or to be the subject of infringing activity. ----------------- This is BULLSHIT! Dropbox is censoring this because they don't want it to get out there. What will they censor next?
    - drdaeman 15y ago
      Wow. Either Dropbox have some copyright issues with Dropship's code, or this is just a blatant misuse of DMCA to take down the content. Won't speculate, but I personally suspect the latter. Wladimir did release the software under FOSS Expat ("MIT") license so he can't really take it back. It's now up to good will of others. While I understand that this may put Dropbox in unfortunate situation, such methods to take down the problematic piece of software somehow feel wrong.
w1ntermute 15y ago

joshbaptiste 15y ago

Indeed, great idea/hack but the only issue I can foresee is Dropship becoming very popular for illegal file duplication, bringing forth the attention of the RIAA/publishers etc.. causing legal headaches for our beloved Dropbox.

orofino 15y ago

forgive what is possibly a very ignorant question, but are there security concerns here? I understand that the key space is immensely huge and that for any file over 4MB in size it would be virtually impossible to guess, but what is to stop someone from just trying hashes for fun to see if they get interesting files? Like I said for file over 4MB it seems fine, guessing sequential hashes would be all but impossible. I assume the realistic solution is just to encrypt my files (preferably in a truecrypt volume over 4MB in size) if I'm truly concerned. On a side note, it would be interesting to see if this could be modified to tell me how unique my overall file set is.

power78 15y ago
I see your point here. I hope non-public files are protected from Dropbox's deduplication
- valindar 15y ago
  They aren't. A colleague copied a whole bunch of documentation from his private Dropbox onto my computer; when I then copied it into my Dropbox it took around half a minute to sync and it was a couple hundred MB.

antimatter15 15y ago

It's a novel exploit of deduplication, but I don't see how it's practically any better than moving a file into the /Public directory and handing them a URL.

orofino 15y ago
In theory, developing this a bit more would lead to searchable directories of files and an easy way to retrieve them, very similar to a torrent tracker. That has significant benefit over shared files and I have to think would scare the heck out of dropbox because of the ire it might bring upon them. This would have to be a worst nightmare for them. Although removing deduplication would solve it for them (with significant increase in what has to be stored).
limmeau 15y ago
Consider 1) User buys a file from the rights owner and explorers it into his Dropbox 2) User obtains a blockwise hash of the file and runs dropship on it 3) User obtains a public URL from somebody who has the file and downloads the file from the Dropbox web server The point is that the Dropbox server cannot distinguish 1 from 2, but both from 3. Therefore, 2 should be more robust against takedown notices than 3.
Pahalial 15y ago
Dropbox has bandwidth limits on /Public URLs, particularly low for free accounts. This wholly circumvents that, and I suspect that's the real reason Arash asked for a takedown, not so much the loose association with piracy.
- dr_win 15y ago
  Dropbox could do the same for hashes. For example internet cannot download more than 400MB from a hash (4MB) per day.

EGreg 15y ago

It's centralized, not distributed. You are at the whim of dropbox. This is not a successor to torrents at all.

Jarred 15y ago

I think the real successor to torrents was actually its predecessor, and that's binary usenet files. Download speeds are bottleknecked at your own downstream, most providers have SSL support for encryption of everything you download, and there's a plethora of content. People don't really know about it though

kin 15y ago
absolutely. instead of leaving a movie download overnight i leave it for a shower. i hope it never hits mainstream, especially since it's subscription based
ippisl 15y ago
From my experience rapidshare like sites have much more content than usenet. is there a some hidden way to to access vast content in usenet ?

deleted 15y ago

[deleted]

JeremyBanks 15y ago

Does anybody have another mirror?

deleted 15y ago
[deleted]

power78 15y ago

So who is game for setting up a repository site of json hash files?

Aissen 15y ago

At last. I'm surprised nobody thought of implementing this earlier. I thought about it, but this being a direct attack on Dropbox, I don't see much value in it. Apart from being unethical, it will only force Dropbox to either remove this very useful feature, or implement a challenge-like system which will render this useless. This will be short-lived code if it spreads. In fact, I think this "feature" is one of the (many) reasons why Dropbox doesn't have an opensource client. And it isn't exposed it in its so-called "API". Edit: I just saw that they killed the feature: http://news.ycombinator.com/item?id=2483053 http://news.ycombinator.com/item?id=2483053

gbraad 15y ago

A lot to do about this posting: http://razorfast.com/2011/04/25/dropbox-attempts-to-kill-open-source-project/ http://razorfast.com/2011/04/25/dropbox-attempts-to-kill-ope...