8 ms·

Filecoin's Cryptoeconomic Constructions

83 points by containerds 6y ago

drcode 6y ago
Some initial questions about this plan: 1. How does this system deal with the "data withholding" problem? In other words, when people provide "storage power" their data will be repeatedly sampled to make sure it is available... but when an entity claims that samples aren't being provided as required by the protocol, how does the system determine that that person wasn't lying, if the sampled data is still provided correctly in a followup request? If the answer is "through arbitration", what prevent the arbitration system from being DDOSed? 2. The "verified clients" are certified by "a decentralized network of verifiers". How does this system prevent a sibyl attack, i.e how does it prevent verifiers from repeatedly verifying themselves using multiple accounts? 3. I notice this system doesn't mention the use of erasure coding, which is usually a common feature of similar schemes by other projects. Why is it that erasure coding isn't necessary in this system? In other words, if data is randomly sampled, how does a client make sure 0.001% of the data isn't missing if only 99.999% or less of the data has been sampled so far? 4. The filecoin organization has a ton of funds due to their successful ICO. This makes it hard for users of the filecoin network to know if it is truly scalable (since the filecoin org could just run a bunch of anonymous server farms with their funds that provide free storage to paper over flaws in the cryptoeconomic incentives) How can a user of filecoin get some assurance that the files they are storing aren't just sitting on a server run by the filecoin organization & are truly running on a decentralized system functioning through the specified cryptoeconomic mechanism?
- Wishery2 6y ago
  I don't think there's a good solution to arbitration DoS, I believe that's a theoretically unsolvable issue. isn't most of ipfs running on some combination of ipfs.io and infura? Will Filecoin end up in the same situation?
  - carlosdp 6y ago
    Well even if people use ipfs.io or infura gateways, the content served via IPFS is not being "served" from those central servers, just cached, so it's not exactly the same situation.
- Acrobatic_Road 6y ago
  To answer 1.), the data is not sampled to prove its there. Instead, the miner must provide a continua; cryptographic proof (called proof of replication) to prove he has stored the data over a given period of time. https://filecoin.io/blog/filecoin-proof-system/ https://filecoin.io/blog/filecoin-proof-system/
- emagdnim2100 6y ago
  Hey, great questions! I'm on the Filecoin team (but not the proofs/cryptography team, where the root answers to some of your questions lie). Let me try a first pass and tap out for someone deeper on the cryptography side if needed. > How does this system deal with the "data withholding" problem? In other words, when people provide "storage power" their data will be repeatedly sampled to make sure it is available... but when an entity claims that samples aren't being provided as required by the protocol, how does the system determine that that person wasn't lying, if the sampled data is still provided correctly in a followup request? Filecoin sort of splits this problem into two parts – "data withholding" from Filecoin's proof-of-spacetime consensus mechanism (a "storage fault" in Filecoin terminology, yes I know there's a lot of new terminology here!), and "data withholding" from a client that's requesting stored data. Storage miners are required to prove to the network itself, not to any specific challenger entity, that they're storing files. Each storage miner is (basically) randomly challenged once per [short interval] to provide a compressed cryptographic proof in response to a challenge. The proof conclusively confirms that, during that period, the miner was storing the data being they'd previously promised to store. You can ctrl-f "if a miner goes offline" in the linked post for a surface-level description of how the network deals with storage faults. Ditching the data and recovering it later is economically irrational for pretty involved reasons – basically, recovery is more expensive than just storing the data over the short-ish intervals during which faults are recoverable. When it comes to "withholding data" from clients – retrieval on Filecoin is just a market-based system for bandwidth. The solution to holding data "hostage," i.e. refusing to serve it at reasonable prices, is to store a few replicated copies (just like centralized storage services do for you today behind the scenes). There's really no upside to miners refusing to profitably serve you a file when they know or suspect you can get it from another source. > The "verified clients" are certified by "a decentralized network of verifiers". How does this system prevent a sibyl attack, i.e how does it prevent verifiers from repeatedly verifying themselves using multiple accounts? The short answer here – with apologies for the brevity; details forthcoming – is that verified data isn't meant to be scarce, and some degree of over-verification is expected. There will be a decentralized group of folks responsible for (quite permissively) verifying and renewing clients for fixed amounts of data, and declining to renew allocations for clients who seem to be abusively verifying data. We're optimistic that this will dramatically decrease the rate at which "fake" data is stored and (most importantly) ensure that there's always storage available for client data. > Why is it that erasure coding isn't necessary in this system? Basically: cool, novel cryptography! In particular, this is where proofs-of-replication and proofs-of-spacetime kick in. Check out this podcast with Juan to learn much more: https://filecoin.io/blog/filecoin-proof-system/ https://filecoin.io/blog/filecoin-proof-system/ (Also – if you like erasure coding, it is totally compatible with Filecoin whether you're a miner or a client! I would be surprised if this feature isn't developed by the community in Filecoin's early days.) > How can a user of filecoin get some assurance that the files they are storing aren't just sitting on a server run by the filecoin organization? Really fair question. First and foremost, as a client, you get to choose your storage miner if you want to. You then have to solve another problem, of course, which is how to map a Filecoin peer ID to a real-world actor (or prove that it's not being run by Filecoin, or whatever). This is solvable in a bunch of different ways, which I won't get into here, but the high-level takeaway is that you're not just throwing your data at an undifferentiated storage interface with obscure inner workings. More fundamentally – Filecoin is part of a huge ecosystem of open source projects. Transparency is a key value – highlighting the success of the community, including the many decentralized storage miners participating in Filecoin, is really important to us and the only way the network can succeed. You can hop on our Slack any time (https://join.slack.com/t/filecoinproject/shared_invite/zt-dj58b7fq-weyaTEvjHoYF_ENkQHR6Ig https://join.slack.com/t/filecoinproject/shared_invite/zt-dj...) to chat with the many folks already building on Filecoin. If you have other ideas on how we can establish that there are lots of groups operating on the network, not just us, let us know and I'll see what we can do :)
lalaland1125 6y ago
This really looks like a solution in search of a problem. The cloud storage market is already pretty efficient and we are expected to believe that adding a bunch of inefficiency and complexity will result in a better product?
- drcode 6y ago
  This system is going after the dream where everyone can earn extra money by letting other people use spare space on their hard drives. In theory, I think it could offer "cloud storage" at orders of magnitude lower cost than other offerings- This would certainly be desirable. The problem is, people have been trying to build this sort of system for many years, so any proposed system is likely to fail when put into operation & therefore has an enormous burden to prove they've overcome the many fatal flaws that existed in previous systems.
  - ajayyy 6y ago
    Sia has been around for years doing this for backups
    - api 6y ago
      Sia is one of the few ICO projects that seems to actually work, but it gets zero attention.
      - nemo1618 6y ago
        Yep, we fell for the "good products sell themselves" meme. :/ As engineers, we felt uncomfortable promising the moon like so many other ICOs did (and still do). Blockchain-wise, the system we built was actually very conservative: aside from the file contract payment channels, our consensus design doesn't stray far from Bitcoin. As it turns out, the blockchain is the easy part! The hard part is, once you've got your network of hosts, how do you turn that into a truly great product? Sia is crazy cheap, it streams video at 1Gbps, it's got plenty of 9s, but those metrics don't mean anything in a vacuum. We've finally hit upon a more user-friendly product in Skynet, and we're doing a lot more PR this time, but we've still got a long way to go.
jhoechtl 6y ago
Can someone shed light on the relation between filecoin and IPFS? My initial thinking was that filecoin is the incentive layer to pin IPFS content but I see not mention of IPFS on the provided link. Did Filecoin and IPFS separate or did filecoin generalize their "offering"?
- aboodman 6y ago
  There's no direct relationship between the two currently. They share a lot of underlying technology but Filecoin doesn't directly incentivize IPFS, and IPFS itself isn't directly used by Filecoin either. The basic reason why is because the protocol to request a file from Filecoin is a lot more complicated than simply fetching a URL, so IPFS can't simply sit atop, at least not without other changes.
jhoechtl 6y ago
Honest question: Did a quick lookup on filecoins price. Filecoin is not real yet, it is FUD or in financial terms a future, right?
- edude03 6y ago
  It's essentially in closed beta right now. The tech exists, but testing and tuning is still happening.
- aspenmayer 6y ago
  It’s not real in that it’s not able to be mined because the network isn’t functional on devnet yet until more work is done on specs from what I could figure out on the forum and the github. I was directed to the page below to learn more and to refer to their github specs repo to participate. https://filecoin-project.github.io/specs/ https://filecoin-project.github.io/specs/
- Proven 6y ago
  That was exactly the purpose of the post, to make you think if you should get some just in case it becomes the next Bitcoin.

83 points by containerds 6y ago

Some initial questions about this plan: 1. How does this system deal with the "data withholding" problem? In other words, when people provide "storage power" their data will be repeatedly sampled to make sure it is available... but when an entity claims that samples aren't being provided as required by the protocol, how does the system determine that that person wasn't lying, if the sampled data is still provided correctly in a followup request? If the answer is "through arbitration", what prevent the arbitration system from being DDOSed? 2. The "verified clients" are certified by "a decentralized network of verifiers". How does this system prevent a sibyl attack, i.e how does it prevent verifiers from repeatedly verifying themselves using multiple accounts? 3. I notice this system doesn't mention the use of erasure coding, which is usually a common feature of similar schemes by other projects. Why is it that erasure coding isn't necessary in this system? In other words, if data is randomly sampled, how does a client make sure 0.001% of the data isn't missing if only 99.999% or less of the data has been sampled so far? 4. The filecoin organization has a ton of funds due to their successful ICO. This makes it hard for users of the filecoin network to know if it is truly scalable (since the filecoin org could just run a bunch of anonymous server farms with their funds that provide free storage to paper over flaws in the cryptoeconomic incentives) How can a user of filecoin get some assurance that the files they are storing aren't just sitting on a server run by the filecoin organization & are truly running on a decentralized system functioning through the specified cryptoeconomic mechanism?

Wishery2 6y ago
I don't think there's a good solution to arbitration DoS, I believe that's a theoretically unsolvable issue. isn't most of ipfs running on some combination of ipfs.io and infura? Will Filecoin end up in the same situation?
- carlosdp 6y ago
  Well even if people use ipfs.io or infura gateways, the content served via IPFS is not being "served" from those central servers, just cached, so it's not exactly the same situation.
Acrobatic_Road 6y ago
To answer 1.), the data is not sampled to prove its there. Instead, the miner must provide a continua; cryptographic proof (called proof of replication) to prove he has stored the data over a given period of time. https://filecoin.io/blog/filecoin-proof-system/ https://filecoin.io/blog/filecoin-proof-system/
emagdnim2100 6y ago
Hey, great questions! I'm on the Filecoin team (but not the proofs/cryptography team, where the root answers to some of your questions lie). Let me try a first pass and tap out for someone deeper on the cryptography side if needed. > How does this system deal with the "data withholding" problem? In other words, when people provide "storage power" their data will be repeatedly sampled to make sure it is available... but when an entity claims that samples aren't being provided as required by the protocol, how does the system determine that that person wasn't lying, if the sampled data is still provided correctly in a followup request? Filecoin sort of splits this problem into two parts – "data withholding" from Filecoin's proof-of-spacetime consensus mechanism (a "storage fault" in Filecoin terminology, yes I know there's a lot of new terminology here!), and "data withholding" from a client that's requesting stored data. Storage miners are required to prove to the network itself, not to any specific challenger entity, that they're storing files. Each storage miner is (basically) randomly challenged once per [short interval] to provide a compressed cryptographic proof in response to a challenge. The proof conclusively confirms that, during that period, the miner was storing the data being they'd previously promised to store. You can ctrl-f "if a miner goes offline" in the linked post for a surface-level description of how the network deals with storage faults. Ditching the data and recovering it later is economically irrational for pretty involved reasons – basically, recovery is more expensive than just storing the data over the short-ish intervals during which faults are recoverable. When it comes to "withholding data" from clients – retrieval on Filecoin is just a market-based system for bandwidth. The solution to holding data "hostage," i.e. refusing to serve it at reasonable prices, is to store a few replicated copies (just like centralized storage services do for you today behind the scenes). There's really no upside to miners refusing to profitably serve you a file when they know or suspect you can get it from another source. > The "verified clients" are certified by "a decentralized network of verifiers". How does this system prevent a sibyl attack, i.e how does it prevent verifiers from repeatedly verifying themselves using multiple accounts? The short answer here – with apologies for the brevity; details forthcoming – is that verified data isn't meant to be scarce, and some degree of over-verification is expected. There will be a decentralized group of folks responsible for (quite permissively) verifying and renewing clients for fixed amounts of data, and declining to renew allocations for clients who seem to be abusively verifying data. We're optimistic that this will dramatically decrease the rate at which "fake" data is stored and (most importantly) ensure that there's always storage available for client data. > Why is it that erasure coding isn't necessary in this system? Basically: cool, novel cryptography! In particular, this is where proofs-of-replication and proofs-of-spacetime kick in. Check out this podcast with Juan to learn much more: https://filecoin.io/blog/filecoin-proof-system/ https://filecoin.io/blog/filecoin-proof-system/ (Also – if you like erasure coding, it is totally compatible with Filecoin whether you're a miner or a client! I would be surprised if this feature isn't developed by the community in Filecoin's early days.) > How can a user of filecoin get some assurance that the files they are storing aren't just sitting on a server run by the filecoin organization? Really fair question. First and foremost, as a client, you get to choose your storage miner if you want to. You then have to solve another problem, of course, which is how to map a Filecoin peer ID to a real-world actor (or prove that it's not being run by Filecoin, or whatever). This is solvable in a bunch of different ways, which I won't get into here, but the high-level takeaway is that you're not just throwing your data at an undifferentiated storage interface with obscure inner workings. More fundamentally – Filecoin is part of a huge ecosystem of open source projects. Transparency is a key value – highlighting the success of the community, including the many decentralized storage miners participating in Filecoin, is really important to us and the only way the network can succeed. You can hop on our Slack any time (https://join.slack.com/t/filecoinproject/shared_invite/zt-dj58b7fq-weyaTEvjHoYF_ENkQHR6Ig https://join.slack.com/t/filecoinproject/shared_invite/zt-dj...) to chat with the many folks already building on Filecoin. If you have other ideas on how we can establish that there are lots of groups operating on the network, not just us, let us know and I'll see what we can do :)

lalaland1125 6y ago

This really looks like a solution in search of a problem. The cloud storage market is already pretty efficient and we are expected to believe that adding a bunch of inefficiency and complexity will result in a better product?

drcode 6y ago
This system is going after the dream where everyone can earn extra money by letting other people use spare space on their hard drives. In theory, I think it could offer "cloud storage" at orders of magnitude lower cost than other offerings- This would certainly be desirable. The problem is, people have been trying to build this sort of system for many years, so any proposed system is likely to fail when put into operation & therefore has an enormous burden to prove they've overcome the many fatal flaws that existed in previous systems.
- ajayyy 6y ago
  Sia has been around for years doing this for backups
  - api 6y ago
    Sia is one of the few ICO projects that seems to actually work, but it gets zero attention.
    - nemo1618 6y ago
      Yep, we fell for the "good products sell themselves" meme. :/ As engineers, we felt uncomfortable promising the moon like so many other ICOs did (and still do). Blockchain-wise, the system we built was actually very conservative: aside from the file contract payment channels, our consensus design doesn't stray far from Bitcoin. As it turns out, the blockchain is the easy part! The hard part is, once you've got your network of hosts, how do you turn that into a truly great product? Sia is crazy cheap, it streams video at 1Gbps, it's got plenty of 9s, but those metrics don't mean anything in a vacuum. We've finally hit upon a more user-friendly product in Skynet, and we're doing a lot more PR this time, but we've still got a long way to go.

jhoechtl 6y ago

Can someone shed light on the relation between filecoin and IPFS? My initial thinking was that filecoin is the incentive layer to pin IPFS content but I see not mention of IPFS on the provided link. Did Filecoin and IPFS separate or did filecoin generalize their "offering"?

aboodman 6y ago
There's no direct relationship between the two currently. They share a lot of underlying technology but Filecoin doesn't directly incentivize IPFS, and IPFS itself isn't directly used by Filecoin either. The basic reason why is because the protocol to request a file from Filecoin is a lot more complicated than simply fetching a URL, so IPFS can't simply sit atop, at least not without other changes.

jhoechtl 6y ago

Honest question: Did a quick lookup on filecoins price. Filecoin is not real yet, it is FUD or in financial terms a future, right?

edude03 6y ago
It's essentially in closed beta right now. The tech exists, but testing and tuning is still happening.
aspenmayer 6y ago
It’s not real in that it’s not able to be mined because the network isn’t functional on devnet yet until more work is done on specs from what I could figure out on the forum and the github. I was directed to the page below to learn more and to refer to their github specs repo to participate. https://filecoin-project.github.io/specs/ https://filecoin-project.github.io/specs/
Proven 6y ago
That was exactly the purpose of the post, to make you think if you should get some just in case it becomes the next Bitcoin.