25 ms·

Filecoin, StorJ and the problem with decentralized storage (2019)

126 points by arthur2e5 5y ago

yosito 5y ago
As someone who just set up Storj to backup my Nextcloud installation, I am very interested in this topic. Storj was a cheap alternative to AWS S3, and, at least out of the box, zero-knowledge encryption was easier, though I know there are ways to achieve zero knowledge encryption with any S3 provider. I hadn't considered Glacier, but I may look into it. Edit: One thing I liked about Storj was that geographic redundancy is built in. I know Amazon has that too, but if three data centers caught fire on the same day, Amazon could lose your data. Super unlikely with Amazon, but virtually impossible with Storj, and most S3 providers have less geographic redundancy than Amazon. > knowing your data is there does not mean that you can get it back when needed. This is the subject of the next blog post. This is something I'd love to hear more about.
- GTP 5y ago
  The next blog post it's already there, have a look. It's basically about the incentives that could make in certain scenarios more profitable to never give back data to the one who purchased the storage.
- nemo1618 5y ago
  Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer. Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.
  - GoldenMonkey 5y ago
    Sure it can provide an SLA. The code is the SLA. And is available for all to examine. Via opensource. And governance allows the community to change terms/accept code... moving forward.
    - prepend 5y ago
      I can run the code behind a network that prevents outbound file transfers for large data. So the code shows file, all my proof of storage is correct and users are hosed. There’s no “proof of bandwidth” (yet)
      - opnitro 5y ago
        And again, even if we restrict ourselves to honest parties, hardware failures happen. Auditing the code tells you nothing about their infrastructure.
    - mediocregopher 5y ago
      That's pretty hand-wavy. Anyone can read the published code, but you can't read the code being run by each individual entity in the network. Any individual might change their server to not make the stored data available to those paying for the storage. It's not _likely_, as that would negate the usefulness of the whole system if it became an issue, and therefore negate the cash flow into the system. Ultimately what's keeping the storage providers honest, whether they're centralized or decentralized, is that their future profit stream is based on their current behavior, and they presumably care about future profits. Code and SLAs ain't nothing but words.
      - yjftsjthsd-h 5y ago
        In fairness, real-company SLAs also tend to be hand-wavy. People like to pretend it's "they'll never be down more than X" or "if they're ever down more than X they'll give us a refund", but it's usually more like "the service provider said that it didn't count because only one part of their service was 'degraded', and after we threatened to take them to court they threw us a $13 discount'".
- thebean11 5y ago
  I looked into glacier pricing a while back it's..tricky. It looks super cheap up front, but gets extremely expensive if you ever need to get the data out of glacier.
  - prepend 5y ago
    Extremely expensive in terms of storage, but still much cheaper than S3. I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
  - teraflop 5y ago
    Depending on how long ago you checked it out (relative to the ~9 years since Glacier was first announced) it might be worth another look. It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
    - gruez 5y ago
      >It's still quite expensive if you ever need to retrieve data in a hurry The real killer is the $0.09/GB egress fees. That dwarfs the $0.00099/GB/month storage fee and the $0.0025/GB bulk retrieval fee.
      - thebean11 5y ago
        yep, this is what I was talking about, didn't have the exact terminology
  - deleted 5y ago
    [deleted]
twostorytower 5y ago
Sia has been operating at a large scale for years and has a running decentralized CDN service (Skynet).
msgilligan 5y ago
Article looks like it is from December 2019. Shouldn't (2019) be in the title?
Geee 5y ago
One important property of decentralized storage is that it prevents data monopolies. For example, if you host your web app on Sia Skynet, then users own their data instead of being locked in by the app developer. It also enables a model where different apps can easily access the same data. The data is in the "cloud", but still owned by the user.
- tymekpavel 5y ago
  Do you know why Sia is overlooked so often? I see a lot of shills on forums and the founder seems to tweet bitterly about Filecoin, but folks seem optimistic about the tech. So what am I missing that makes Sia not get adoption vs. Filecoin?
  - shepardrtc 5y ago
    They spent very little on marketing and focused on building something. That's starting to change now.
  - tracedddd 5y ago
    The Sia team has let down customers for other tangential projects, and Sia has weaker guarantees than Filecoin. It’s been out a long time now and they have failed to address some core scaling and redundancy concerns. I wouldn’t say it’s bad, but it’s seeming more like maidsafe - never got enough traction and now the interest is on filecoin and arweave.
  - yupyup54133 5y ago
    Likewise with ScPrime (https://scpri.me https://scpri.me) which is like Sia but focused on B2B data storage and is changing to PoS instead of PoW to be more environmentally friendly.
  - Geee 5y ago
    Filecoin got a lot of funding from prominent investors before they even started. I don't think there's much actual adoption in there.
- wmf 5y ago
paulpauper 5y ago
You can use the data field in etheruem for decentralized and permaenent storage (convert the file to hex ad then paste it) but very, very expensive .
- coolspot 5y ago
  Not with middle-out compression.
de6u99er 5y ago
Just for everybody to be in the same page. Blockchain is not used for distributed storage itself. Existing concepts like IPFS are used for this. Blockchain is used to confirm that someone is storing certain data and to verify it's integrity through an additional protocol. From ipfs.io about Blockchain >With IPFS, you can address large amounts of data and put immutable, permanent links in transactions — timestamping and securing content without having to put the data itself on-chain. Please correct me if I am wrong.
- omginternets 5y ago
  You are corrent; I think the article is just clumsily worded, though. I think what he's saying is that IPFS gives you references to immutable data, which you are then free to commit to your favorite blockchain.
omginternets 5y ago
I've noticed the debate around (de)centralization always seems to focus on purely technical issues, or on economic issues (which are technicalities of another sort). In other words, it's always the following two points that are debated: 1. Economies of Scale 2. Network Properties (robustness, scalability, latency, etc.) It seems to me that a crucial element is missing from these analyses: organizational policy. All other things being equal (which they may not be), it seems like decentralization appeals to those trying to restrict top-down control. Historically, this has been an ideological thing (e.g. bittorrent's copyleft slant), but more recently I've seen mainstream endorsement of concepts such as "flat hierarchy", "distributed workforce", "collaborative work", etc. Interestingly, I don't see this discourse being picked up by decentralization evangelists, and I'm not sure what to make of that. Off the cuff, it seems like decentralized storage tech has a compelling story to tell about small distributed teams, working in a loose, peer-to-peer organizational structure. In this context, my mind immediately goes to the functional centralization of cloud storage. In general, the Cloud tries to concentrate control over computing resources in a single department (i.e. the devops team). But what happens when a bunch of freelancers collaborate on a project-by-project basis? From what I've seen, either (1) they use cloud providers, but the account is under the client's name or (2) they use some kind of SaaS solution with support for sharing or other forms of collaboration. With decentralized storage, it seems like a "bring your own cloud" approach is possible in principle, where these freelancers would pool resources (storage, compute, etc.). Are there any pain-points here? Is there perhaps a business problem that needs solving? The question for me is whether these new patterns of collaboration are really becoming a thing, or whether it's a fad amongst business-school graduates. At any rate, this comment is an invitation for y'all to opine on the subject. I feel like a good analysis of organizational policy could help us decide whether a decentralized web is economically viable, but I'm kinda stumped.
- omginternets 5y ago
  Addendum: I think I could get excited about FileCoin if it allowed me to pool storage with friends and colleagues. I've been wanting to have a distributed hacker-garage for a while [0]. Importantly, I want it to be independent of any cloud provider because (1) it's hard to freely experiment/prototype when you have to keep track of billing and (2) functional centralization makes the AWS/GCE admin de facto responsible for everything running under the account. This is great for large enterprises that want to virtualize their IT department, but it really goes against the grain for hackers, freelancers, and -- IMHO -- startup founders. Returning to Filecoin, I feel like all the *coin ventures make the same fundamental mistake of over-estimating how much we care about money. I really don't care about pimping out my hard-drive for a couple of cents. I'm motivated by gains on the order of several hundred to a couple of thousand dollars. The counter-argument I usually hear is that there will be more interest in developing nations, but then I don't want to store my data in e.g. Pakistan. I want data for pet projects in my own hacker-garage, and I want production data close to the end-user. /rant [0] Incidentally, I'm working on a project to do exactly this. It's a way of clustering a set of computers over the internet to produce a virtual cloud. It would be super cool to use FileCoin as a block storage layer in this context, but restricted to the hardware we own.
iR5ugfXGAE 5y ago
Is there any sensible study on those system's energy consumption?
- wmf 5y ago
  Given that public clouds are ruthlessly optimized for efficiency and they don't have to perform trustless proof of anything, it's virtually certain that decentralized storage is less efficient in every way than centralized.
  - capableweb 5y ago
    What does "virtually certain" mean here anyway? And on what scale are you imagining things? Let's say the entirety of YouTube was decentralized in a way that every internet connected device plugged in to the wall starts sharing watched content to other viewers who want to watch that content. If it's your neighbor, they'll download it straight from you, via the closest router/hub. Suddenly the company's hosting is more providing a "core archive" of data, and every user becomes a edge node of that content. Internet providers can, if they chose, pre-share content they think will be popular, in order to speed up their own infrastructure, and so on. I'm not saying this is bullet proof or anything, I just think you're too quick to dismiss it without really saying why you think it's impossible (or "virtually certain").
    - wmf 5y ago
      I love P2P and I have been into it since the Napster days but it isn't feasible any more. The value of a random Internet device is actually negative because the complexity of wrangling it outweighs what little resources it could contribute. (See Spotify and Joost for examples.) I assume all these decentralized storage systems are datacenter-based so I'm comparing random Chinese datacenters vs. FAANG datacenters.
    - benlivengood 5y ago
      Youtube (and most other streaming services) is already decentralized into thousands of edge caches. Decentralizing it further doesn't make much sense because the network cost is about the same if individual homes or their nearest ISP DC has the cache. Most home networks are wireless only at this point which is pretty bad for throughput, latency, and reliability if serving other homes. Maybe ISPs could start building some storage and IPFS into modems? I think handling the metadata of a p2p system might still be too costly. Imagine every neighborhood having to run a torrent tracker for every media file produced in the last ~year.
SavantIdiot 5y ago
Economies of scale in the cloud have one thing going for them: massive reliability. As someone responsible for my company's data, I cannot make a sound argument to convince management (or myself) to use anything but Amazon S3 (or Google or Microsoft) cloud. The data is simply too important to trust to a smaller entity. Maybe coin storage can boost Storj's reputation. And I certainly favor decentralization for Crypto.
- capableweb 5y ago
  So granted the durability and availability is better than centralized solutions, you'd go with the distributed/decentralized solution? If that's the case, I think you could argue that open source and distributed solutions are much easier to make more reliable than centralized solutions. With a centralized solution, you're stuck with trusting their reliability or implement some reusable API (or use some 3rd party's API that abstract that away, introducing more error surface) so you can also store the data in multiple places with the same API. Or, you can just use a open source decentralized solutions and run it locally geographically, hosted on dedicated servers and "in the cloud" (meaning other users who host it) with the same API without really caring about _where_ the files are, just _which_ files you need. And that's the beauty of content-addressable networks like many of these are. It doesn't really matter where the data is as long as you know what data you need. That makes it easier to provide reliability for yourself, not harder. Once you dig down in the details and compare them both carefully, I think you can make sound arguments both ways, it's not as black & white as you make it. Ultimately it depends on one's use case.
  - SavantIdiot 5y ago
    I don't see how this addresses the reliability claim. There's a big difference between explaining that the data is stored in a Google cloud with certified 11-nines reliability, versus stored on a N number of machines in multiple clouds with unknown specs. How do I communicate the reliability of a distributed cloud that is essentially a collection of random servers? Is there a way that each of these server's reliability is accounted for by the high-level service?
fwip 5y ago
IPFS is mostly fine technology for storing data in a content-addressable way. FileCoin is, at best, an inefficient way to incentivize people to store your data in an imaginary trustless world. (At worst, it's an obvious scam.) You can imagine any hosting provider offering regular IPFS hosting. Sign up on our website, send us a list of data to store, pay us money, and we'll guarantee you can get it back or you can sue us - same as the guarantees by any conventional hosting provider. This works fine and well without gluing Filecoin to it.
- pessimizer 5y ago
  The advantage would be to have a program on my computer that would negotiate prices with providers and manage latency and redundancy. This could of course also be solved if storage providers, as a group, standardized on a single protocol (creating a commodity industry.) I'm not sure there's a huge difference between that and this, other than anonymity for providers (which is the major critique of this blogpost.)
- kevincox 5y ago
  IPFS is not a storage technology. IPFS is a retrieval technology. In fact the two technologies would complement each other really well. Imagine that you could pay someone to make content available on IPFS, basically a distributed pinning service. However this isn't trivial because IPFS generally assumes that people with files would like to share them, however a for-profit storage node would be incentivized to use as little bandwidth as possible.
  - fwip 5y ago
    IPFS includes both storage and retrieval. I outlined in my post a specific, trivial way to pay somebody to make data available on IPFS.
    - kevincox 5y ago
      I guess we are arguing semantics but your last paragraph sounds like you are asking someone to do storage and make it available over IPFS, I still see the storage part as quite disparate. This is like saying HTTP provides storage because you can pay someone to host your files and allow you to access them over HTTP.
brutaltruth 5y ago
Not very interesting. There are ways to solve all of these concerns--no fees are paid during the first X months, nodes periodically tested for bandwidth, etc. etc. Just because the author didn't bother to think it through doesn't mean it's not straightforward. The whole idea of a blockchain is reputation CAN be built up over time, because every transaction is recorded and there might even be incentives to associate nodes to show X nines of pool reliability. Is it a bad idea to use this as your only source of storage? Of course. Is it less useful than AWS if you do large local burst queries? Of course. But none of the concerns mentioned in the post are relevant. This can be cheaper than AWS because they charge an enormous markup, and this automatically provides worldwide replication and accessibility. Amazon already erases books from Kindles, Google scans your Docs for blasphemy against Fauci, and both have been known to throw businesses off their services. In a few years it will seem crazy not to have an extra copy of your data on here.
- 300bps 5y ago
  What do you base AWS storage having "an enormous markup" on? AWS has a type of storage to meet any access need / budget. S3 Glacier Deep Dive for example is perfect for storing archival items and costs $0.00099 per GB-month. So you can store a TB for about $1 per month. There are even options like S3 Intelligent Tiering that enable you to store objects on S3 Standard and it will intelligently move them to lower-cost options based on the usage patterns of the data. There are six different pricing options of S3 storage. There are EBS volumes, EFS volumes, FSx for Lustre and many other options for storage. I used to mine BTC, currently mine Ethereum and looked into mining Filecoin and was left wondering if the people buying into it have any idea what cloud storage options are actually out there. There is so much competition and the prices are sooooo low that I don't see any viable use case for Filecoin outside of the one you mentioned about being concerned about being shut down by a single vendor.
  - deleted 5y ago
    [deleted]
  - admax88q 5y ago
    > What do you base AWS storage having "an enormous markup" on? Probably on the basis that backblaze costs an order of magnitude less for the same product.
jtolds 5y ago
Hi! CTO of Storj here. This article makes some reasonable points, and it makes some unreasonable ones. First, on the reasonable: absolutely, privacy and security should be layered on top of whatever storage platform you use using your own encryption. No objection! That's the right approach. It is always better to bring your own encryption, and having it layered so your storage provider can't have access is great. I have much love for @cperciva's Tarsnap. It's awesome. The downside with a product that punts encryption to the user is that users /typically don't do this/. In fact, users like to share things and serve content to others. Bringing your own encryption that the sharing infrastructure doesn't understand means it is much harder to actually use the product for sharing. This is why Storj includes hierarchical encryption for delegated sharing. It's not that embedding encryption into the framework is better encryption by any means, but it is a better default. Users are by default protected by their own keys that we don't have access to, which is a better default than the cloud. Should people use something else also? Sure! If it suits them. On the unreasonable: The author says: > [I]t is very unlikely that a decentralized storage market can offer an alternative that can compete against centralized providers— AWS, Google, Azure— when measured on any of these [cost, speed, reliability] dimensions. This is a testable assumption, and the author didn't test it (post is from December of 2019, we were in late beta then, working very well). Storj is cheaper ($4/TB/mo, was $10/TB/mo in December of 2019), provides 11 9s of durability (no SLA offered until March 2020, but we had and still have never lost a single object out of billions), and is equivalent in speed to providers like Backblaze (and getting faster). It's also multiregion by default (obviously). This is a screaming deal on all dimensions. Try it out for yourself. Our metrics are delivering on all three dimensions. This author is picking and choosing between architectures of Filecoin and Storj, which is fine, but each should be evaluated in isolation. Filecoin and Storj make very different design decisions with very different payoffs. The author argues Storj can't deliver on its promises with appeals to Filecoin's architecture. Filecoin absolutely uses blockchains and proof of storage and whatever else. In fact, Filecoin does fall victim to requiring lots of resources (see https://docs.filecoin.io/mine/hardware-requirements/ https://docs.filecoin.io/mine/hardware-requirements/). But this is just true of Filecoin, this isn't a problem with decentralized storage in general, and Storj is an excellent counter example. Storj does not require lots of resources (and in fact many of our node operators use Raspberry Pis). Storj is not a blockchain, uses statistical audits that are low-effort and low-CPU for storage node operators, and works great for idle capacity. If you're interested in how we can achieve these things, you might take a look at an older blog post I wrote explaining (without naming) why Filecoin's architecture is fundamentally flawed: https://www.storj.io/blog/replication-is-bad-for-decentralized-storage-part-1-erasure-codes-for-fun-and-profit https://www.storj.io/blog/replication-is-bad-for-decentraliz... Storj and Filecoin are both decentralized storage products, but they are really fundamentally very different, and their differences are worth understanding.
hanklazard 5y ago
As someone who's been interested in filecoin mining for a while, my main question is around legal liability for miners. For instance, if I am storing other people's data and it turns out someone was storing something illegal (unbeknownst to me), could I be held liable under US law?
- wmf 5y ago
  This is where the DMCA and CDA safe harbors help you.
- qyi 5y ago
  Why do you think this is different than hosting a normal website with file uploads? Literally millions of people have made such websites (especially around 2000-2010).
  - hanklazard 5y ago
    That’s true but when you host a website, you can decide what types of files can be uploaded. With this sort of decentralized storage network, people will include whatever files they want. And to put a finer point on it, no doubt some significant portion of the file content will be horrifying video content, some of which will be illegal. My question is more around the liability for that, even if it is encrypted and the miner doesn’t know what it is. A small filecoin operation would not have the legal funds to defend against an aggressive prosecutor trying to “take down a network of dangerous pornographers” and would get little sympathy from a general public that doesn’t understand the tech. Until there is strong legal precedent, I wouldn’t entertain a mining operation. Maybe I’m paranoid but it just doesn’t seem worth the risk.
    - qyi 5y ago
      It's infeasible to make sure a server is only accepting "legal" content. I've never heard of a case of someone being jailed because his system involuntarily received some data that happens to be illegal. Normally what happens if there is illegal data that anyone cares about, the hoster is contacted and told to remove it. >it just doesn’t seem worth the risk. Well, the concern is only yours. This does raise the point that if police contact you to remove some data you are hosting for someone, you lose money, and they will likely not give you back whatever money you lost (all the more reason it should be fully anonymous like Freenet but IPFS and co are obsessed with trying to solve every use case in the world including legal compliance).
ruggeri 5y ago
The article raises a question I've wondered about before: how do you know that anyone will transfer you your data when you need it? You can require people to prove that they stored it, but how can you require people to transfer it out to you? You presumably agree with the node to pay for the transfer out, but if both counterparties are anonymous and do not trust each other, how is this accomplished?
- wmf 5y ago
  The answer is incentives: if you're paying to retrieve data, why shouldn't they take your business?
  - ruggeri 5y ago
    But do they send you the data first, or do you send them the payment first? If they send you the data first, then why should you pay them? If you send them the payment first, then why should they send it? I'm sure there's a piece I'm missing, but have I missed something obvious?
    - wmf 5y ago
      Data retrieval is broken into blocks so it's a repeated game with incentives to cooperate. AFAIK Filecoin uses payment channels so you probably lock some money, retrieve data, then unlock the payment.
      - ruggeri 5y ago
        Incremental transfer seems like it should give an incentive to cooperate, but I don't think so? If I pay you upfront for the first block, what stops you from then saying: actually I still want more money to start transfering the first block?
        wmf 5y ago
        I doubt any of the protocols have an extortion response code.
liamsargent 5y ago
Why is Sia overlooked every single time this topic is brought up? It is shocking to me that Filecoin, an example of an ICO darling with no working product, is always at the top of the list. I have used Sia and Skynet to build several useful decentralized media sharing applications with zero loss of data for over a year.
- ranguna 5y ago
  Sia currently costs 8$ TB/month, storj costs a fixed forever* 4$ TB/month. * not counting usd inflation, because they don't lock storage price to coin value (with crazy volatility), they fix it against usd and covert it to their coin when paying their nodes.
qyi 5y ago
Interesting article. While reading about Filecoin the first time, I had to stop every paragraph and think "but what if the node just doesn't give you your data back when you finally ask for it?". And as pointed out by part 2 of the article, not only can they choose not to give it back, but they are incentivized not to. edit: Wouldn't erasure codes just completely fix it (as described by the Storj guy)?
jfrunyon 5y ago
> Cloud services are ruled by a ruthless economy of scales. This is where Amazon, Google, MSFT and a host of other cloud providers shine, reaping the benefits of investment in data-centers and petabytes of storage capacity. I dunno about that. Amazon and Google, at least, charge significantly more than I could go out and buy a few drives for on sale. (AND, of course, they charge it perpetually. Dunno how StorJ works.)
deleted 5y ago
[deleted]
kfprt 5y ago
A better solution would be payment based on a minimum GB and bandwidth where the data is constantly being streamed back over the network with a payout period of ~5min. This way you can guarantee that you will get all your data back at least once a week. You can also guarantee via network latency that your data is not being collocated. Sounds crazy but it would be more energy efficient than bitcoin.
wallacoloo 5y ago
> Another option is requiring service providers to put up a surety bond. Before they are allow to participate in the ecosystem, they must set aside a lump sum on the blockchain held in escrow. These funds would be used to compensate any customers harmed by failure to honor storage contracts. But this has the effect of creating additional barriers to entry and locking away capital in a very unproductive way. If bonds solve the bad actor problem, then this use of capital would directly enable safe data storage. I guess it depends on your definition of “productive”, but that definitely feels like a good use of capital to me.
rmtech 5y ago
Since this was written, Filecoin is up about 10,000% or something stupid. If you pay attention to the naysayers, you're gonna have fun staying poor.

126 points by arthur2e5 5y ago

As someone who just set up Storj to backup my Nextcloud installation, I am very interested in this topic. Storj was a cheap alternative to AWS S3, and, at least out of the box, zero-knowledge encryption was easier, though I know there are ways to achieve zero knowledge encryption with any S3 provider. I hadn't considered Glacier, but I may look into it. Edit: One thing I liked about Storj was that geographic redundancy is built in. I know Amazon has that too, but if three data centers caught fire on the same day, Amazon could lose your data. Super unlikely with Amazon, but virtually impossible with Storj, and most S3 providers have less geographic redundancy than Amazon. > knowing your data is there does not mean that you can get it back when needed. This is the subject of the next blog post. This is something I'd love to hear more about.

GTP 5y ago
The next blog post it's already there, have a look. It's basically about the incentives that could make in certain scenarios more profitable to never give back data to the one who purchased the storage.
nemo1618 5y ago
Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer. Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.
- GoldenMonkey 5y ago
  Sure it can provide an SLA. The code is the SLA. And is available for all to examine. Via opensource. And governance allows the community to change terms/accept code... moving forward.
  - prepend 5y ago
    I can run the code behind a network that prevents outbound file transfers for large data. So the code shows file, all my proof of storage is correct and users are hosed. There’s no “proof of bandwidth” (yet)
    - opnitro 5y ago
      And again, even if we restrict ourselves to honest parties, hardware failures happen. Auditing the code tells you nothing about their infrastructure.
  - mediocregopher 5y ago
    That's pretty hand-wavy. Anyone can read the published code, but you can't read the code being run by each individual entity in the network. Any individual might change their server to not make the stored data available to those paying for the storage. It's not _likely_, as that would negate the usefulness of the whole system if it became an issue, and therefore negate the cash flow into the system. Ultimately what's keeping the storage providers honest, whether they're centralized or decentralized, is that their future profit stream is based on their current behavior, and they presumably care about future profits. Code and SLAs ain't nothing but words.
    - yjftsjthsd-h 5y ago
      In fairness, real-company SLAs also tend to be hand-wavy. People like to pretend it's "they'll never be down more than X" or "if they're ever down more than X they'll give us a refund", but it's usually more like "the service provider said that it didn't count because only one part of their service was 'degraded', and after we threatened to take them to court they threw us a $13 discount'".
thebean11 5y ago
I looked into glacier pricing a while back it's..tricky. It looks super cheap up front, but gets extremely expensive if you ever need to get the data out of glacier.
- prepend 5y ago
  Extremely expensive in terms of storage, but still much cheaper than S3. I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
- teraflop 5y ago
  Depending on how long ago you checked it out (relative to the ~9 years since Glacier was first announced) it might be worth another look. It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
  - gruez 5y ago
    >It's still quite expensive if you ever need to retrieve data in a hurry The real killer is the $0.09/GB egress fees. That dwarfs the $0.00099/GB/month storage fee and the $0.0025/GB bulk retrieval fee.
    - thebean11 5y ago
      yep, this is what I was talking about, didn't have the exact terminology
- deleted 5y ago
  [deleted]

twostorytower 5y ago

Sia has been operating at a large scale for years and has a running decentralized CDN service (Skynet).

msgilligan 5y ago

Article looks like it is from December 2019. Shouldn't (2019) be in the title?

Geee 5y ago

One important property of decentralized storage is that it prevents data monopolies. For example, if you host your web app on Sia Skynet, then users own their data instead of being locked in by the app developer. It also enables a model where different apps can easily access the same data. The data is in the "cloud", but still owned by the user.

tymekpavel 5y ago
Do you know why Sia is overlooked so often? I see a lot of shills on forums and the founder seems to tweet bitterly about Filecoin, but folks seem optimistic about the tech. So what am I missing that makes Sia not get adoption vs. Filecoin?
- shepardrtc 5y ago
  They spent very little on marketing and focused on building something. That's starting to change now.
- tracedddd 5y ago
  The Sia team has let down customers for other tangential projects, and Sia has weaker guarantees than Filecoin. It’s been out a long time now and they have failed to address some core scaling and redundancy concerns. I wouldn’t say it’s bad, but it’s seeming more like maidsafe - never got enough traction and now the interest is on filecoin and arweave.
- yupyup54133 5y ago
  Likewise with ScPrime (https://scpri.me https://scpri.me) which is like Sia but focused on B2B data storage and is changing to PoS instead of PoW to be more environmentally friendly.
- Geee 5y ago
  Filecoin got a lot of funding from prominent investors before they even started. I don't think there's much actual adoption in there.
wmf 5y ago

paulpauper 5y ago

You can use the data field in etheruem for decentralized and permaenent storage (convert the file to hex ad then paste it) but very, very expensive .

coolspot 5y ago
Not with middle-out compression.

de6u99er 5y ago

Just for everybody to be in the same page. Blockchain is not used for distributed storage itself. Existing concepts like IPFS are used for this. Blockchain is used to confirm that someone is storing certain data and to verify it's integrity through an additional protocol. From ipfs.io about Blockchain >With IPFS, you can address large amounts of data and put immutable, permanent links in transactions — timestamping and securing content without having to put the data itself on-chain. Please correct me if I am wrong.

omginternets 5y ago
You are corrent; I think the article is just clumsily worded, though. I think what he's saying is that IPFS gives you references to immutable data, which you are then free to commit to your favorite blockchain.

omginternets 5y ago

I've noticed the debate around (de)centralization always seems to focus on purely technical issues, or on economic issues (which are technicalities of another sort). In other words, it's always the following two points that are debated: 1. Economies of Scale 2. Network Properties (robustness, scalability, latency, etc.) It seems to me that a crucial element is missing from these analyses: organizational policy. All other things being equal (which they may not be), it seems like decentralization appeals to those trying to restrict top-down control. Historically, this has been an ideological thing (e.g. bittorrent's copyleft slant), but more recently I've seen mainstream endorsement of concepts such as "flat hierarchy", "distributed workforce", "collaborative work", etc. Interestingly, I don't see this discourse being picked up by decentralization evangelists, and I'm not sure what to make of that. Off the cuff, it seems like decentralized storage tech has a compelling story to tell about small distributed teams, working in a loose, peer-to-peer organizational structure. In this context, my mind immediately goes to the functional centralization of cloud storage. In general, the Cloud tries to concentrate control over computing resources in a single department (i.e. the devops team). But what happens when a bunch of freelancers collaborate on a project-by-project basis? From what I've seen, either (1) they use cloud providers, but the account is under the client's name or (2) they use some kind of SaaS solution with support for sharing or other forms of collaboration. With decentralized storage, it seems like a "bring your own cloud" approach is possible in principle, where these freelancers would pool resources (storage, compute, etc.). Are there any pain-points here? Is there perhaps a business problem that needs solving? The question for me is whether these new patterns of collaboration are really becoming a thing, or whether it's a fad amongst business-school graduates. At any rate, this comment is an invitation for y'all to opine on the subject. I feel like a good analysis of organizational policy could help us decide whether a decentralized web is economically viable, but I'm kinda stumped.

omginternets 5y ago
Addendum: I think I could get excited about FileCoin if it allowed me to pool storage with friends and colleagues. I've been wanting to have a distributed hacker-garage for a while [0]. Importantly, I want it to be independent of any cloud provider because (1) it's hard to freely experiment/prototype when you have to keep track of billing and (2) functional centralization makes the AWS/GCE admin de facto responsible for everything running under the account. This is great for large enterprises that want to virtualize their IT department, but it really goes against the grain for hackers, freelancers, and -- IMHO -- startup founders. Returning to Filecoin, I feel like all the *coin ventures make the same fundamental mistake of over-estimating how much we care about money. I really don't care about pimping out my hard-drive for a couple of cents. I'm motivated by gains on the order of several hundred to a couple of thousand dollars. The counter-argument I usually hear is that there will be more interest in developing nations, but then I don't want to store my data in e.g. Pakistan. I want data for pet projects in my own hacker-garage, and I want production data close to the end-user. /rant [0] Incidentally, I'm working on a project to do exactly this. It's a way of clustering a set of computers over the internet to produce a virtual cloud. It would be super cool to use FileCoin as a block storage layer in this context, but restricted to the hardware we own.

iR5ugfXGAE 5y ago

Is there any sensible study on those system's energy consumption?

wmf 5y ago
Given that public clouds are ruthlessly optimized for efficiency and they don't have to perform trustless proof of anything, it's virtually certain that decentralized storage is less efficient in every way than centralized.
- capableweb 5y ago
  What does "virtually certain" mean here anyway? And on what scale are you imagining things? Let's say the entirety of YouTube was decentralized in a way that every internet connected device plugged in to the wall starts sharing watched content to other viewers who want to watch that content. If it's your neighbor, they'll download it straight from you, via the closest router/hub. Suddenly the company's hosting is more providing a "core archive" of data, and every user becomes a edge node of that content. Internet providers can, if they chose, pre-share content they think will be popular, in order to speed up their own infrastructure, and so on. I'm not saying this is bullet proof or anything, I just think you're too quick to dismiss it without really saying why you think it's impossible (or "virtually certain").
  - wmf 5y ago
    I love P2P and I have been into it since the Napster days but it isn't feasible any more. The value of a random Internet device is actually negative because the complexity of wrangling it outweighs what little resources it could contribute. (See Spotify and Joost for examples.) I assume all these decentralized storage systems are datacenter-based so I'm comparing random Chinese datacenters vs. FAANG datacenters.
  - benlivengood 5y ago
    Youtube (and most other streaming services) is already decentralized into thousands of edge caches. Decentralizing it further doesn't make much sense because the network cost is about the same if individual homes or their nearest ISP DC has the cache. Most home networks are wireless only at this point which is pretty bad for throughput, latency, and reliability if serving other homes. Maybe ISPs could start building some storage and IPFS into modems? I think handling the metadata of a p2p system might still be too costly. Imagine every neighborhood having to run a torrent tracker for every media file produced in the last ~year.

SavantIdiot 5y ago

Economies of scale in the cloud have one thing going for them: massive reliability. As someone responsible for my company's data, I cannot make a sound argument to convince management (or myself) to use anything but Amazon S3 (or Google or Microsoft) cloud. The data is simply too important to trust to a smaller entity. Maybe coin storage can boost Storj's reputation. And I certainly favor decentralization for Crypto.

capableweb 5y ago
So granted the durability and availability is better than centralized solutions, you'd go with the distributed/decentralized solution? If that's the case, I think you could argue that open source and distributed solutions are much easier to make more reliable than centralized solutions. With a centralized solution, you're stuck with trusting their reliability or implement some reusable API (or use some 3rd party's API that abstract that away, introducing more error surface) so you can also store the data in multiple places with the same API. Or, you can just use a open source decentralized solutions and run it locally geographically, hosted on dedicated servers and "in the cloud" (meaning other users who host it) with the same API without really caring about _where_ the files are, just _which_ files you need. And that's the beauty of content-addressable networks like many of these are. It doesn't really matter where the data is as long as you know what data you need. That makes it easier to provide reliability for yourself, not harder. Once you dig down in the details and compare them both carefully, I think you can make sound arguments both ways, it's not as black & white as you make it. Ultimately it depends on one's use case.
- SavantIdiot 5y ago
  I don't see how this addresses the reliability claim. There's a big difference between explaining that the data is stored in a Google cloud with certified 11-nines reliability, versus stored on a N number of machines in multiple clouds with unknown specs. How do I communicate the reliability of a distributed cloud that is essentially a collection of random servers? Is there a way that each of these server's reliability is accounted for by the high-level service?

fwip 5y ago

IPFS is mostly fine technology for storing data in a content-addressable way. FileCoin is, at best, an inefficient way to incentivize people to store your data in an imaginary trustless world. (At worst, it's an obvious scam.) You can imagine any hosting provider offering regular IPFS hosting. Sign up on our website, send us a list of data to store, pay us money, and we'll guarantee you can get it back or you can sue us - same as the guarantees by any conventional hosting provider. This works fine and well without gluing Filecoin to it.

pessimizer 5y ago
The advantage would be to have a program on my computer that would negotiate prices with providers and manage latency and redundancy. This could of course also be solved if storage providers, as a group, standardized on a single protocol (creating a commodity industry.) I'm not sure there's a huge difference between that and this, other than anonymity for providers (which is the major critique of this blogpost.)
kevincox 5y ago
IPFS is not a storage technology. IPFS is a retrieval technology. In fact the two technologies would complement each other really well. Imagine that you could pay someone to make content available on IPFS, basically a distributed pinning service. However this isn't trivial because IPFS generally assumes that people with files would like to share them, however a for-profit storage node would be incentivized to use as little bandwidth as possible.
- fwip 5y ago
  IPFS includes both storage and retrieval. I outlined in my post a specific, trivial way to pay somebody to make data available on IPFS.
  - kevincox 5y ago
    I guess we are arguing semantics but your last paragraph sounds like you are asking someone to do storage and make it available over IPFS, I still see the storage part as quite disparate. This is like saying HTTP provides storage because you can pay someone to host your files and allow you to access them over HTTP.

brutaltruth 5y ago

Not very interesting. There are ways to solve all of these concerns--no fees are paid during the first X months, nodes periodically tested for bandwidth, etc. etc. Just because the author didn't bother to think it through doesn't mean it's not straightforward. The whole idea of a blockchain is reputation CAN be built up over time, because every transaction is recorded and there might even be incentives to associate nodes to show X nines of pool reliability. Is it a bad idea to use this as your only source of storage? Of course. Is it less useful than AWS if you do large local burst queries? Of course. But none of the concerns mentioned in the post are relevant. This can be cheaper than AWS because they charge an enormous markup, and this automatically provides worldwide replication and accessibility. Amazon already erases books from Kindles, Google scans your Docs for blasphemy against Fauci, and both have been known to throw businesses off their services. In a few years it will seem crazy not to have an extra copy of your data on here.

300bps 5y ago
What do you base AWS storage having "an enormous markup" on? AWS has a type of storage to meet any access need / budget. S3 Glacier Deep Dive for example is perfect for storing archival items and costs $0.00099 per GB-month. So you can store a TB for about $1 per month. There are even options like S3 Intelligent Tiering that enable you to store objects on S3 Standard and it will intelligently move them to lower-cost options based on the usage patterns of the data. There are six different pricing options of S3 storage. There are EBS volumes, EFS volumes, FSx for Lustre and many other options for storage. I used to mine BTC, currently mine Ethereum and looked into mining Filecoin and was left wondering if the people buying into it have any idea what cloud storage options are actually out there. There is so much competition and the prices are sooooo low that I don't see any viable use case for Filecoin outside of the one you mentioned about being concerned about being shut down by a single vendor.
- deleted 5y ago
  [deleted]
- admax88q 5y ago
  > What do you base AWS storage having "an enormous markup" on? Probably on the basis that backblaze costs an order of magnitude less for the same product.

jtolds 5y ago

Hi! CTO of Storj here. This article makes some reasonable points, and it makes some unreasonable ones. First, on the reasonable: absolutely, privacy and security should be layered on top of whatever storage platform you use using your own encryption. No objection! That's the right approach. It is always better to bring your own encryption, and having it layered so your storage provider can't have access is great. I have much love for @cperciva's Tarsnap. It's awesome. The downside with a product that punts encryption to the user is that users /typically don't do this/. In fact, users like to share things and serve content to others. Bringing your own encryption that the sharing infrastructure doesn't understand means it is much harder to actually use the product for sharing. This is why Storj includes hierarchical encryption for delegated sharing. It's not that embedding encryption into the framework is better encryption by any means, but it is a better default. Users are by default protected by their own keys that we don't have access to, which is a better default than the cloud. Should people use something else also? Sure! If it suits them. On the unreasonable: The author says: > [I]t is very unlikely that a decentralized storage market can offer an alternative that can compete against centralized providers— AWS, Google, Azure— when measured on any of these [cost, speed, reliability] dimensions. This is a testable assumption, and the author didn't test it (post is from December of 2019, we were in late beta then, working very well). Storj is cheaper ($4/TB/mo, was $10/TB/mo in December of 2019), provides 11 9s of durability (no SLA offered until March 2020, but we had and still have never lost a single object out of billions), and is equivalent in speed to providers like Backblaze (and getting faster). It's also multiregion by default (obviously). This is a screaming deal on all dimensions. Try it out for yourself. Our metrics are delivering on all three dimensions. This author is picking and choosing between architectures of Filecoin and Storj, which is fine, but each should be evaluated in isolation. Filecoin and Storj make very different design decisions with very different payoffs. The author argues Storj can't deliver on its promises with appeals to Filecoin's architecture. Filecoin absolutely uses blockchains and proof of storage and whatever else. In fact, Filecoin does fall victim to requiring lots of resources (see https://docs.filecoin.io/mine/hardware-requirements/ https://docs.filecoin.io/mine/hardware-requirements/). But this is just true of Filecoin, this isn't a problem with decentralized storage in general, and Storj is an excellent counter example. Storj does not require lots of resources (and in fact many of our node operators use Raspberry Pis). Storj is not a blockchain, uses statistical audits that are low-effort and low-CPU for storage node operators, and works great for idle capacity. If you're interested in how we can achieve these things, you might take a look at an older blog post I wrote explaining (without naming) why Filecoin's architecture is fundamentally flawed: https://www.storj.io/blog/replication-is-bad-for-decentralized-storage-part-1-erasure-codes-for-fun-and-profit https://www.storj.io/blog/replication-is-bad-for-decentraliz... Storj and Filecoin are both decentralized storage products, but they are really fundamentally very different, and their differences are worth understanding.

hanklazard 5y ago

As someone who's been interested in filecoin mining for a while, my main question is around legal liability for miners. For instance, if I am storing other people's data and it turns out someone was storing something illegal (unbeknownst to me), could I be held liable under US law?

wmf 5y ago
This is where the DMCA and CDA safe harbors help you.
qyi 5y ago
Why do you think this is different than hosting a normal website with file uploads? Literally millions of people have made such websites (especially around 2000-2010).
- hanklazard 5y ago
  That’s true but when you host a website, you can decide what types of files can be uploaded. With this sort of decentralized storage network, people will include whatever files they want. And to put a finer point on it, no doubt some significant portion of the file content will be horrifying video content, some of which will be illegal. My question is more around the liability for that, even if it is encrypted and the miner doesn’t know what it is. A small filecoin operation would not have the legal funds to defend against an aggressive prosecutor trying to “take down a network of dangerous pornographers” and would get little sympathy from a general public that doesn’t understand the tech. Until there is strong legal precedent, I wouldn’t entertain a mining operation. Maybe I’m paranoid but it just doesn’t seem worth the risk.
  - qyi 5y ago
    It's infeasible to make sure a server is only accepting "legal" content. I've never heard of a case of someone being jailed because his system involuntarily received some data that happens to be illegal. Normally what happens if there is illegal data that anyone cares about, the hoster is contacted and told to remove it. >it just doesn’t seem worth the risk. Well, the concern is only yours. This does raise the point that if police contact you to remove some data you are hosting for someone, you lose money, and they will likely not give you back whatever money you lost (all the more reason it should be fully anonymous like Freenet but IPFS and co are obsessed with trying to solve every use case in the world including legal compliance).

ruggeri 5y ago

The article raises a question I've wondered about before: how do you know that anyone will transfer you your data when you need it? You can require people to prove that they stored it, but how can you require people to transfer it out to you? You presumably agree with the node to pay for the transfer out, but if both counterparties are anonymous and do not trust each other, how is this accomplished?

wmf 5y ago
The answer is incentives: if you're paying to retrieve data, why shouldn't they take your business?
- ruggeri 5y ago
  But do they send you the data first, or do you send them the payment first? If they send you the data first, then why should you pay them? If you send them the payment first, then why should they send it? I'm sure there's a piece I'm missing, but have I missed something obvious?
  - wmf 5y ago
    Data retrieval is broken into blocks so it's a repeated game with incentives to cooperate. AFAIK Filecoin uses payment channels so you probably lock some money, retrieve data, then unlock the payment.
    - ruggeri 5y ago
      Incremental transfer seems like it should give an incentive to cooperate, but I don't think so? If I pay you upfront for the first block, what stops you from then saying: actually I still want more money to start transfering the first block?
      - wmf 5y ago
        I doubt any of the protocols have an extortion response code.

liamsargent 5y ago

Why is Sia overlooked every single time this topic is brought up? It is shocking to me that Filecoin, an example of an ICO darling with no working product, is always at the top of the list. I have used Sia and Skynet to build several useful decentralized media sharing applications with zero loss of data for over a year.

ranguna 5y ago
Sia currently costs 8$ TB/month, storj costs a fixed forever* 4$ TB/month. * not counting usd inflation, because they don't lock storage price to coin value (with crazy volatility), they fix it against usd and covert it to their coin when paying their nodes.

qyi 5y ago

Interesting article. While reading about Filecoin the first time, I had to stop every paragraph and think "but what if the node just doesn't give you your data back when you finally ask for it?". And as pointed out by part 2 of the article, not only can they choose not to give it back, but they are incentivized not to. edit: Wouldn't erasure codes just completely fix it (as described by the Storj guy)?

jfrunyon 5y ago

> Cloud services are ruled by a ruthless economy of scales. This is where Amazon, Google, MSFT and a host of other cloud providers shine, reaping the benefits of investment in data-centers and petabytes of storage capacity. I dunno about that. Amazon and Google, at least, charge significantly more than I could go out and buy a few drives for on sale. (AND, of course, they charge it perpetually. Dunno how StorJ works.)

deleted 5y ago

[deleted]

kfprt 5y ago

A better solution would be payment based on a minimum GB and bandwidth where the data is constantly being streamed back over the network with a payout period of ~5min. This way you can guarantee that you will get all your data back at least once a week. You can also guarantee via network latency that your data is not being collocated. Sounds crazy but it would be more energy efficient than bitcoin.

wallacoloo 5y ago

> Another option is requiring service providers to put up a surety bond. Before they are allow to participate in the ecosystem, they must set aside a lump sum on the blockchain held in escrow. These funds would be used to compensate any customers harmed by failure to honor storage contracts. But this has the effect of creating additional barriers to entry and locking away capital in a very unproductive way. If bonds solve the bad actor problem, then this use of capital would directly enable safe data storage. I guess it depends on your definition of “productive”, but that definitely feels like a good use of capital to me.

rmtech 5y ago

Since this was written, Filecoin is up about 10,000% or something stupid. If you pay attention to the naysayers, you're gonna have fun staying poor.