5 ms·

Catching Cheating Servers in Decentralized Storage Networks

90 points by siavosh 8y ago

runeks 8y ago
> To do it properly, you need to store the file in multiple places since you don't trust any individual stranger's computer. But how do you differentiate between three honest servers with one copy each and three cheating servers with one copy total? Anything you ask one server about the file it can get from its collaborator. How about just storing three copies of the same data, where each copy is encrypted with a unique private key (before sending it off to storage)? This way the servers essentially don’t know that they’re storing the same data three times.
- a1369209993 8y ago
  As TFA mentions, this works fine for private storage, but we're trying to incentivise people to back up publicly accessable data like the blockchain or chunks of the internet archive. We want a way for possibly-malicious servers to prove that they're storing distinct bottom-layer encodings of data they do know is the same.
  - irq-1 8y ago
    So encrypt large files not blocks. Then you ask for random segments and either the file is stored, or the cheater would need to re-encrypt a large file over and over, which would be much more expensive than storage. (and if it's not more expensive, make the files larger, the requests more often, etc...)
- mannykannot 8y ago
  In the beginning, each participant had her own copy of the blockchain, and this decentralization was put forward as an important feature of the system (in practice, we have moved away from that, but anyone can get a copy if they want.) If we were to simply replace each of today's copies of the blockchain 1-to-1 with a distinct distributed copy, then encryption would be fine, but it would not reduce the amount of storage needed globally. This proposal addresses one problem in reducing the the global storage required without creating the situation where, in principle, there might only be one copy of the blockchain, vulnerable to corruption and controlled by one entity, and with that situation not being detectable until too late. Looked at in this light, the use of distributed storage would appear, at least at first sight, to create another hierarchical relationship, this time between storage providers and ordinary users, analogous perhaps to the hierarchical relationship between miners and ordinary users. I don't know if this would create another 51% issue.
Drdrdrq 8y ago
Incentives are not aligned here. As a user, I'm not paying for file to be stored - I'm paying so it can be retrieved anytime. As a space owner, I want to maximize profits per space, with cheapest (not reliable) disks possible. This system would work much better if payment was tied to retrieval, not storage.
- hamandcheese 8y ago
  This system has its own set of issues though: - if I am storing data, how do I know I’ll get paid? What if the data is never retrieved? - if I want data stored for later retrieval, how do I know it will continue to be stored? - if my data is stored with only one remaining party, how do I make sure they don’t learn this and hold my data hostage? - how are retrievals public ally verified?
  - Drdrdrq 8y ago
    True, the idea would need to be developed further. That said, I think periodic retreival of data (and thus periodic payment to storage providers) would solve most of these issues.
  - niyikiza 8y ago
    My team -- and I'm sure a number of other teams -- are looking into different approaches of implementing a "Proof of Retrievability". What we need is a cost-effective way for a data owner to verify that their data would be available intact should they need it. And that's where we want to tie our incentive policy for storage nodes owners. And yes, a storage owner does not need to know whose data that is, or which other nodes are storing replicas for the same data.
- woah 8y ago
  You are paying for the option of retrieval. That is what you want as a user. That’s what these systems are designed for. The words “paying for storage” are just easier to say.
ghthor 8y ago
I've been trying to figure this out as well. A few of my ideas. Payouts could be based on having the largest unique set of hashes. So this should make a server excited to see something it hasn't seen before, and would discourage it from dumping unique data, as it's worth the most.
wyldfire 8y ago
> What happens if it's a public good like blockchain state? Can't these be nodes use pruning such that this state is only needed for "full nodes"? (yes, this does mean less decentralization). Aren't full nodes already incentivized to store and backup the 75GB somewhat by the coin's stakers? > And worse, we don't trust anyone to encrypt it properly. We need something totally different that anyone can check and anyone can decode. We don't trust anyone but we might trust "everyone". If you had a backup that was signed by several devs or community leaders, it would probably be "good enough." I think the solution is interesting but the problem less so.

90 points by siavosh 8y ago

> To do it properly, you need to store the file in multiple places since you don't trust any individual stranger's computer. But how do you differentiate between three honest servers with one copy each and three cheating servers with one copy total? Anything you ask one server about the file it can get from its collaborator. How about just storing three copies of the same data, where each copy is encrypted with a unique private key (before sending it off to storage)? This way the servers essentially don’t know that they’re storing the same data three times.

a1369209993 8y ago
As TFA mentions, this works fine for private storage, but we're trying to incentivise people to back up publicly accessable data like the blockchain or chunks of the internet archive. We want a way for possibly-malicious servers to prove that they're storing distinct bottom-layer encodings of data they do know is the same.
- irq-1 8y ago
  So encrypt large files not blocks. Then you ask for random segments and either the file is stored, or the cheater would need to re-encrypt a large file over and over, which would be much more expensive than storage. (and if it's not more expensive, make the files larger, the requests more often, etc...)
mannykannot 8y ago
In the beginning, each participant had her own copy of the blockchain, and this decentralization was put forward as an important feature of the system (in practice, we have moved away from that, but anyone can get a copy if they want.) If we were to simply replace each of today's copies of the blockchain 1-to-1 with a distinct distributed copy, then encryption would be fine, but it would not reduce the amount of storage needed globally. This proposal addresses one problem in reducing the the global storage required without creating the situation where, in principle, there might only be one copy of the blockchain, vulnerable to corruption and controlled by one entity, and with that situation not being detectable until too late. Looked at in this light, the use of distributed storage would appear, at least at first sight, to create another hierarchical relationship, this time between storage providers and ordinary users, analogous perhaps to the hierarchical relationship between miners and ordinary users. I don't know if this would create another 51% issue.

Drdrdrq 8y ago

Incentives are not aligned here. As a user, I'm not paying for file to be stored - I'm paying so it can be retrieved anytime. As a space owner, I want to maximize profits per space, with cheapest (not reliable) disks possible. This system would work much better if payment was tied to retrieval, not storage.

hamandcheese 8y ago
This system has its own set of issues though: - if I am storing data, how do I know I’ll get paid? What if the data is never retrieved? - if I want data stored for later retrieval, how do I know it will continue to be stored? - if my data is stored with only one remaining party, how do I make sure they don’t learn this and hold my data hostage? - how are retrievals public ally verified?
- Drdrdrq 8y ago
  True, the idea would need to be developed further. That said, I think periodic retreival of data (and thus periodic payment to storage providers) would solve most of these issues.
- niyikiza 8y ago
  My team -- and I'm sure a number of other teams -- are looking into different approaches of implementing a "Proof of Retrievability". What we need is a cost-effective way for a data owner to verify that their data would be available intact should they need it. And that's where we want to tie our incentive policy for storage nodes owners. And yes, a storage owner does not need to know whose data that is, or which other nodes are storing replicas for the same data.
woah 8y ago
You are paying for the option of retrieval. That is what you want as a user. That’s what these systems are designed for. The words “paying for storage” are just easier to say.

ghthor 8y ago

I've been trying to figure this out as well. A few of my ideas. Payouts could be based on having the largest unique set of hashes. So this should make a server excited to see something it hasn't seen before, and would discourage it from dumping unique data, as it's worth the most.

wyldfire 8y ago

> What happens if it's a public good like blockchain state? Can't these be nodes use pruning such that this state is only needed for "full nodes"? (yes, this does mean less decentralization). Aren't full nodes already incentivized to store and backup the 75GB somewhat by the coin's stakers? > And worse, we don't trust anyone to encrypt it properly. We need something totally different that anyone can check and anyone can decode. We don't trust anyone but we might trust "everyone". If you had a backup that was signed by several devs or community leaders, it would probably be "good enough." I think the solution is interesting but the problem less so.