7 ms·
Filecoin: Proof of Storage Systems
- simonebrunozzi 6y ago> We created Filecoin because the amount of data humans generate is exponentially increasing, and we need more efficient ways to store and access it. I don't think Filecoin can be MORE efficient than some of the larger, current systems. I actually think this statement is misleading. It should say that Filecoin's goal is to offer storage options that are distributed (hence redundant), protected from censorship, and possibly removed from absolute control of a single large corporation.
- RL_Quine 6y agoAlmost by definition, it simply can not be more efficient.
- rjeli 6y agoAssuming the FIL storage market is efficient, if the cost of CPU time and energy to run PoRep is less than the premium charged by commercial providers, it's more efficient. That seems plausible to me.
- TylerE 6y agoThat's an awfully tall and teetering stack of assumptions.
- geek_at 6y agoit can, if you're using microsoft storage spaces as software raid
- rudolph9 6y agoThere are a lot of nuanced ways a trustless distributed system can be more efficient. The article is pretty light on details but consider the content addressing mechanism. The data can be securely served by any device and independently verified by any recipient. The minimum latency of reads is theoretically much lower than the sever/client infrastructure commonly used today. It’s like edge computing with any device that can connect to one another. Obviously a lot goes into ensuring the data existing on a low latency connection to the distributed-client but the there are a number of somewhat nuanced theoretical efficiencies that can be achieved by making content addressing the core of a distributed system (caching, [as mentioned] fault-tolerance, parallel computing, etc).
- acdha 6y ago> The minimum latency of reads is theoretically much lower than the sever/client infrastructure commonly used today. This is an oversimplification: it assumes that discovery is very low cost and that there's a peer with a copy enough closer on the network that it's faster than talking to server-class hardware in a data center with a high quality network connection. Given the number of assumptions which need to be true for that to be a net-positive I'm skeptical that it'd be easy to hit anywhere close to the best-case theoretical scenario.
- rudolph9 6y ago> Obviously a lot goes into ensuring the data existing on a low latency connection to the distributed-client This is admittedly a non-trivial task but perhaps an example can shed light on way it can be more efficient in practice. Consider the Apple Photos app. It's on your phone, it's on your desktop, it's on the cloud (iCloud storage). Say you have so many photos they don't fit on your phone. When you try to access a photo that is not on your phone it gets it from icloud (these are literally features that are currently offered by photos/iCloud). Now suppose your desktop has lots of storage, you usually look at your photos while connected to your local network, why cant you get the data from there? The key to the efficiency of file coin is not some magic cryptographic token, it's not even "blockchain", the key is the underlying protocols being unified for synchronizing devices across multiple tiers of networking. Filecoin mostly just makes sure the data doesn't disappear and gives you an optional substitute for cloud storage. Content addressing, p2p communication by default, flexible/future-proof standards for negotiated what data is and how it is formatted, and in modular plug-and-play utilization of many different communication/synchronization techniques is the key to applications being built in more effective and efficient way than conventional client/server architecture. The word that comes to mind is "flexibility". Want you store to be S3 buckets? Great! Want to host a private network, only your devices can access? Great! Want to joint the public network? Great! Want to ensure persistence of data via a token? Great! Want to just use the address mechanism? Great! Want to run the software in the cloud, on you local machine, in a sandboxed browser tab and have them all seamlessly forming a p2p network over a multitude of network standards (tcp, udp, web-sockets, etc)? Great! I worked extensively last year with various ipfs techs utilized in a private p2p network consisting of ARM nodes capture time-series measurement devices and cloud based persistent nodes managing the pinset of data and persisting long-term storage. There are definitely some rough edges but again it's open-source built is such a modular way it's almost difficult to develop your self into a corner. The software coming out of protocol labs is overwhelmingly not new technology. Protocol lab large focuses on utilization of battle-tested existing standards and tech (many of which are relatively ancient like TCP, ssh, git, json, etc) unified, without breaking existing stands, via a common addressing mechanism. tl;dr Filecoin/crypto-tokens just a side note! The vast majority of software coming out of protocol labs is independently useful and largely focuses on bridging together widely-used battle-tested tech.
- fsflover 6y agoIt can be more efficient if you are using thousands/millions of hard drives which otherwise would just be unused.
- Taek 6y agoThis story falls apart a bit with Filecoin because you need to attach them to 128 GB of RAM and a high end graphics card to do their storage proofs. Secondary markets for drives may appear to close the gap a bit though.
- hddherman 6y agoA bit unrelated, but a pyramid scheme called Kairos had this exact same pitch. That didn't work out, but that was due to the fact that it was a scam, not due to any technical issues.
- mdaniel 6y agoThat is not what matches my mental model of a "deep dive," but was interesting enough to warrant an upvote For example, the phrase "decentralized marketplace of storage providers and choose the one" was just glossed over -- through what mechanism does one choose a storage provider? Is there an API that (for example) a container storage interface provider could consume? like that kind of "choose a provider"? I also would have enjoyed more hyperlinks for terms such as zk-SNARK, and "a storage miner" which is similarly glossed over
- momack2 6y agoYou may enjoy this HackFS Workshop from Juan Benet (creator of Filecoin) that starts pretty high level but gets much more into the details of how the Filecoin network works: https://www.youtube.com/watch?v=P28aNAdZDi4 https://www.youtube.com/watch?v=P28aNAdZDi4
- dang 6y agoOk, we'll swap a prefix for a suffix of the title above. (Submitted title was "Deep Dive into Filecoin".)
- simonebrunozzi 6y agoGood call. I should have thought of a better (submission) title in the first place.
- ironchief 6y agoThis hardly meets the definition of brief description, let alone a "Deep Dive".
- hacker_newz 6y agoThat's par for the course working with filecoin.
- cgb223 6y agoHow is Filecoin different from Sia coin? I remember a while ago hearing about both trying to decentralize storage but never kept up with it enough to really suss out the difference
- Sargos 6y agoSia seems like a Dropbox/OneDrive replacement whereas Filecoin is an incentivized storage layer infrastructure for apps to use, especially dapps. If you wanted to build a decentralized SoundCloud you couldn't use Sia but IPFS/Filecoin is the primary building block of such a dapp.
- olah_1 6y agoSia released a new version of their product called Skynet[1] which provides exactly the functionality of IPFS+Filecoin today. Part of Skynet is a group of SDKs[2] which you can use to integrate skynet storage as part of your application. I believe that they are even adding the ability for developers to get kickbacks if they build on Sia[3]. If you do want something like Dropbox that is built on Sia, there is Filebase[4]. I am not associated with Sia at all. I've just been following their twitter for a while. [1]: https://siasky.net/ https://siasky.net/ [2]: https://nebulouslabs.github.io/skynet-docs/#introduction https://nebulouslabs.github.io/skynet-docs/#introduction [3]: https://twitter.com/SiaTechHQ/status/1291441690008592384 https://twitter.com/SiaTechHQ/status/1291441690008592384 [4]: https://docs.filebase.com/ https://docs.filebase.com/
- xur17 6y agoI'm curious to look at filecoin - last time I tried to use sia, it was still fairly rough around the edge (minimum file size, seed based recovery isn't automatic, etc).
- olah_1 6y agoMy 2 cents is that they just have different methodologies driving them. Sia is shipping a working network early and updates are arriving often. Filecoin seems to be taking an approach more like Holochain stuff. Meaning they are shipping early and often but only internally and you'll get to see it when its ready™.
- mdaniel 6y agoBy chance, I saw a relevant article over on r/ipfs: https://www.axios.com/filecoin-blockchain-delay-3b5e6b9a-bcc8-41cf-81cf-563f6cebb2c4.html https://www.axios.com/filecoin-blockchain-delay-3b5e6b9a-bcc...
- wmf 6y agoDiscussed yesterday: https://news.ycombinator.com/item?id=24199595 https://news.ycombinator.com/item?id=24199595
- mifeng 6y agoThis article seems like it was written in 2017, not 2020. Filecoin raised over $200m and hasn't shipped anything that I'm aware of in 3 years. Maybe it will be revolutionary, but the crypto industry has long moved on from white whale projects that don't ship.
- wmf 6y agohttps://github.com/filecoin-project/lotus https://github.com/filecoin-project/lotus https://docs.filecoin.io/how-to/install-filecoin/ https://docs.filecoin.io/how-to/install-filecoin/
- mifeng 6y agoThis is what I'm talking about. It's been 3 years, and they're still doing testnets? What a joke.
- hinkley 6y agoGoing from zero to a positive velocity takes a certain kind of person who doesn't let doubts stop them. But to build a team around a project? You can hack a lot of stuff together than only makes sense to you, crippling any of your would-be collaborators. Certain kinds of doubts should in fact stop you, or at least slow you down. It's a difficult thing to balance, and most of us struggle to pull it off. I tried to help out with Freenet years ago. The code was in a language I had quite a lot of deep knowledge about, I should have been able to help quite a bit, but I ended up sticking to specs and architecture. Bluntly, the code was not the output of an organized mind. Even though most of it was written by one person. It ping-ponged all over. The prototype was already full of esoteric optimizations that sacrificed clarity (and thus correctness) for a little more speed. Major rafts of functionality were still being designed, and it was already work-hardened. When I see a team ramp up and get wedged, my first questions are about code quality, practices, and team dynamics. About cohesion. Too often the old guard can't delegate, even if they want to. With enough tribal knowledge, your coworkers just start to get interesting, and then they move on to a new job, and you replace them with another person who struggles to accomplish even simple things. Or even worse, they made a promise that Information Theory says they can't keep, and they've been bargaining the entire time. It's a terrible place to be.
- zepearl 6y agoGeneral question: when somebody will push into Filecoin some "bad" files (anything, from docs needed by the mafia to children being sexually abused, to fake videos/claims/whatever about you, to any other bad things you can think of), how can that be detected and then deleted? (ignoring here the theme related to accountability) Question based on https://docs.filecoin.io/introduction/why-filecoin/ https://docs.filecoin.io/introduction/why-filecoin/ : Filecoin resists censorship because there is no central provider that can be coerced into deleting files or withholding service. The network is made up of many different computers run by many different people and organizations. Faulty or malicious actors are noticed by the network and removed automatically. In this context I'm not a believer of "automatically".
- yorwba 6y agoIn this context, "faulty and malicious actors" would be those who interfere with the operation of the network, e.g. by deleting files they consider "bad".
- acdha 6y agoI generally phrase this as “How is this safer than running a Tor exit node?”. If someone uploads illicit material to Amazon, they have enough lawyers & clout to have the FBI send them a letter demanding client information. If this is a peer-to-peer service, you have a non-zero risk that the SWAT team is going to show up at your home/office and demand you prove that you weren't aware of that content. That's going to deter a LOT of people.
- simonebrunozzi 6y agoI think this is a serious problem - but probably not for the exact use case you list (mafia, sexually abused kids). It should be solved by allowing an "expensive" (a) "vote" (b) to override, or delete, or hide for a number of years. (a): expensive should discourage the option in general (b): proof-of-stake, proof-of-ownership, proof-of-work are some of the many ways in which you can define voting rights. Unfortunately, it doesn't work easily in practice. Example: You created a file (say, a CV, or a newspaper article in PDF) to discredit me, but it's a lie. To me, not having that file publicly accessible would be worth $1,000. You could blackmail me - give me $1,000 or I'll upload the file! But I think you might do it anyway, so I don't negotiate (never negotiate!!). The file is 1 MB, and it costs $1/year to keep it up and running on Filecoin. I should have an option to ask a voting pool to remove that file for $100. The pool agrees, and the money gets burned (I shouldn't create any incentive for someone to cash in the amount). However, you can upload the file again; or a slightly different version, and we are back to square one. It then goes to the "discoverability" of the file. That part is probably going to be decentralized too, so no court could "order" Filecoin to make a file not discoverable. How do we solve it? I think there will emerge a search system that provides services to censor, obfuscate, etc, a number of files, based on reasonable requests. Whoever builds that company, I bet it will not be poor.
- filleokus 6y agoI don't really get the point of decentralised _storage_ solutions, where the stated goal is to compete with ≈ S3 [0]. S3 storage pricing, for most users is just a rounding error compared to e.g egress or compute cost. And it has proven to be tremendously reliable, both in durability and access. No sane person would point a CDN against a Filecoin backed origin, right? For kinda-censorship resistant distribution of stuff like Wikileak dumps we already have torrents right now, where the seeders are aware of the content they spread. For uses where S3 is not competitive, like storage of my media rips, spinning rust at places like Hetzner exist, or Backblaze et. al for more casual backup uses. The only point I can see for decentralised "storage" of this kind is for ultra high durability, small data size, almost never read, almost some sort of "stakes". One could imagine something like storing contracts/land deeds, similar to Gwern's timestamping of URL's. The files should be small, the storage model optimised for maximal durability (every node store's almost everything?), and prices high enough to ensure durability. [0]: Just look at this PR page https://filecoin.io/store/ https://filecoin.io/store/ [1]: https://www.gwern.net/Timestamping https://www.gwern.net/Timestamping
- macklemoreshair 6y agoAgreed the #1 issue with file coin from day one has been the idea that the value of a coin is tied to anything at all. It puts a limit on the imaginative value because it’s tied to something as commoditized as bulk storage
- thefounder 6y agoFuck S3 and all the lords gating the internet. I think that's what decentralised web wants to say and I'm pretty much on its side.
- Proven 6y agoYou can run S3 at home and mirror data to encrypted container running the same in public cloud, genius.
- Dylan16807 6y agoI understand your overall point, but when it comes to > S3 storage pricing, for most users is just a rounding error compared to e.g egress or compute cost. And it has proven to be tremendously reliable, both in durability and access. It's not like you can separate S3 storage costs from egress costs, though. I wish I could use S3 for storage but pay for my own bandwidth. Even if you'll to go all the way to their network and plug a network cable in, well... They charge $7.40 a day for a gigabit port, and $54 a day for 10gig. That's on par with what many companies charge for transit to the actual internet, but fair enough. Oh, wait, I neglected the per-byte cost on top. The per byte cost that, for a mostly saturated port, is $210 or $2100 per day respectively. And that's the cheap option.
- opportune 6y agoOne thing I'd like to see decentralized storage protocols guarantee is that not only does the data exist on the backend, but is it available enough to be useful, and if so, are the latency and bandwidth good. PoST might be able to solve this if it enforces a strict deadline on the answer and tests availability sufficiently frequently. The other is some sort of authentication. Encryption of the data itself, IMO, is not good enough. In 10 years time I don't want a nation state or megacorporation with enough entangled qubits to be able to pwn all of my data. But distributed authentication as I understand it is a hard problem, because you probably need to make your authentication proof public, which creates a similar set of problems on the authentication itself.
- foota 6y agoTwo thoughts: One is that frequently verifying availability is going to get expensive quick with iops for coldish storage systems. Secondly, iirc symmetric encryption isn't really vulnerable to quantum algorithms, as far as we know (and maybe even proven not to be?). It's asymmetric crypto that's at risk.
- Taek 6y agoCan address this with probabilistic checks. Download random pieces of your data and ensure it has acceptable latencies. If not, move the data to a different provider. By erasure coding your data over many providers, you minimize the amount of harm that a couple of bad apples can cause.