4 ms·

As someone who just set up Storj to backup my Nextcloud installation, I am very interested in this topic. Storj was a cheap alternative to AWS S3, and, at least

by yosito 6y ago

As someone who just set up Storj to backup my Nextcloud installation, I am very interested in this topic. Storj was a cheap alternative to AWS S3, and, at least out of the box, zero-knowledge encryption was easier, though I know there are ways to achieve zero knowledge encryption with any S3 provider. I hadn't considered Glacier, but I may look into it. Edit: One thing I liked about Storj was that geographic redundancy is built in. I know Amazon has that too, but if three data centers caught fire on the same day, Amazon could lose your data. Super unlikely with Amazon, but virtually impossible with Storj, and most S3 providers have less geographic redundancy than Amazon. > knowing your data is there does not mean that you can get it back when needed. This is the subject of the next blog post. This is something I'd love to hear more about.

GTP 6y ago
The next blog post it's already there, have a look. It's basically about the incentives that could make in certain scenarios more profitable to never give back data to the one who purchased the storage.
nemo1618 6y ago
Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer. Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.
- GoldenMonkey 6y ago
  Sure it can provide an SLA. The code is the SLA. And is available for all to examine. Via opensource. And governance allows the community to change terms/accept code... moving forward.
  - prepend 6y ago
    I can run the code behind a network that prevents outbound file transfers for large data. So the code shows file, all my proof of storage is correct and users are hosed. There’s no “proof of bandwidth” (yet)
    - opnitro 6y ago
      And again, even if we restrict ourselves to honest parties, hardware failures happen. Auditing the code tells you nothing about their infrastructure.
  - mediocregopher 6y ago
    That's pretty hand-wavy. Anyone can read the published code, but you can't read the code being run by each individual entity in the network. Any individual might change their server to not make the stored data available to those paying for the storage. It's not _likely_, as that would negate the usefulness of the whole system if it became an issue, and therefore negate the cash flow into the system. Ultimately what's keeping the storage providers honest, whether they're centralized or decentralized, is that their future profit stream is based on their current behavior, and they presumably care about future profits. Code and SLAs ain't nothing but words.
    - yjftsjthsd-h 6y ago
      In fairness, real-company SLAs also tend to be hand-wavy. People like to pretend it's "they'll never be down more than X" or "if they're ever down more than X they'll give us a refund", but it's usually more like "the service provider said that it didn't count because only one part of their service was 'degraded', and after we threatened to take them to court they threw us a $13 discount'".
    - dntrkv 6y ago
      What if the system had 2 kinds of nodes, storage nodes, and then verification nodes. The storage nodes, store the data. The verification nodes, hit the storage nodes and verify the data is there and available. If the storage node fails a verification attempt, the storage node pays a fee. After a certain amount of failures, it is banned from the network. Disclaimer: I have no idea how these systems currently work.
- cle 6y ago
  S3 has no durability SLA, only an availability SLA. S3 also explicitly says it is not designed for 100% durability, but for a certian number of nines (99.999999999%).
  - karpierz 6y ago
    Blockchain has neither a durability or availability SLA.
    - chrisco255 6y ago
      An SLA just gives you the right to fee reductions or potential compensation for downtime. And not all SLAs are created equal. Meanwhile, for example, many blockchains such as Bitcoin and Ethereum, have 100% uptime. Not 99.99%...100%.
      - capableweb 6y ago
        "uptime" is such a weird concept when talking about decentralized networks as they are local-first and also build with many peers. And which "uptime" is being referred to here, read or write? If it's write, fees have definitely spiked at points to make it mostly useless for most people "using" it, but you can technically still write to it. And read is just always available as long as you have a copy of the data somewhere.
    - cle 6y ago
      I'm not sure that comparison makes much sense, it's like saying that HTTP has no SLA. AFAIU, Filecoin miners are free to offer SLAs for storage, and clients are free to enter into deals only with miners who offer such SLAs.
- ChainOfFools 6y ago
  > dozens of independent entities or one entity operating dozens of nodes? droplets are cheap, pwned windows boxes even cheaper. they don't even have to be cheap by regular Joe standards, they could be cheap by nation state, or ransomware gang standards. the point is there's no way to prove they are decentralized. this is an intrinsic, structural property of the cryptocurrency model, thus it is not a mere technical defect that will get solved "in the future" proving decentralization would require zero privacy, because you'd have to prove independent nodes are indeed independent and not part of a cartel, which means you'd have to somehow prove who operates each one. nothing but faith-based decentralization here, as with all cryptocurrencies.
- yosito 6y ago
  In the case of Storj, I believe that the redundancy between storage nodes is sufficient that something like only 20 out of 80 nodes need to respond in order to restore your data.
  - deleted 6y ago
    [deleted]
- dnr 6y ago
  The argument (of the second post) is that their behavior will be correlated because they all have (more or less) the same incentives and cost structure.
- px43 6y ago
  > SLAs -- something that a blockchain can't really offer. Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then their stake can get slashed, or burned completely. If a Filecoin node claims to be holding data for someone, but is unable to provide it in a timely manner when legitimately requested to do so, they're in for some serious penalties. The great thing is that the consumer doesn't need to worry about some big company hiring expensive lawyers to weasel out of their contractual obligations. The rules are all readily transparent and enshrined as code that anyone can read and verify the validity of before participating.
  - wmf 6y ago
    I think this speaks to the difference between what SLAs really are and what people wishfully think they are. People think an uptime SLA means "we will not go down much" but it really means "one day we will go down hard and we'll give you a refund (which will be insignificantly small compared to the pain we caused you)".
thebean11 6y ago
I looked into glacier pricing a while back it's..tricky. It looks super cheap up front, but gets extremely expensive if you ever need to get the data out of glacier.
- prepend 6y ago
  Extremely expensive in terms of storage, but still much cheaper than S3. I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
- teraflop 6y ago
  Depending on how long ago you checked it out (relative to the ~9 years since Glacier was first announced) it might be worth another look. It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
  - gruez 6y ago
    >It's still quite expensive if you ever need to retrieve data in a hurry The real killer is the $0.09/GB egress fees. That dwarfs the $0.00099/GB/month storage fee and the $0.0025/GB bulk retrieval fee.
    - thebean11 6y ago
      yep, this is what I was talking about, didn't have the exact terminology
- deleted 6y ago
  [deleted]

by yosito 6y ago

GTP 6y ago

The next blog post it's already there, have a look. It's basically about the incentives that could make in certain scenarios more profitable to never give back data to the one who purchased the storage.

nemo1618 6y ago

Presumably the argument is that a storage host might dutifully provide proofs of storage, but refuse to actually transfer your data to you later. This is true, but it's also true of centralized storage providers. The difference is that traditional providers are typically bound by SLAs -- something that a blockchain can't really offer. Instead, decentralized storage gives you the ability to store your data with dozens of independent entities; as long as the behavior of these entities is sufficiently uncorrelated, then it is highly likely that you will always be able to retrieve your data from some subset of them.

GoldenMonkey 6y ago
Sure it can provide an SLA. The code is the SLA. And is available for all to examine. Via opensource. And governance allows the community to change terms/accept code... moving forward.
- prepend 6y ago
  I can run the code behind a network that prevents outbound file transfers for large data. So the code shows file, all my proof of storage is correct and users are hosed. There’s no “proof of bandwidth” (yet)
  - opnitro 6y ago
    And again, even if we restrict ourselves to honest parties, hardware failures happen. Auditing the code tells you nothing about their infrastructure.
- mediocregopher 6y ago
  That's pretty hand-wavy. Anyone can read the published code, but you can't read the code being run by each individual entity in the network. Any individual might change their server to not make the stored data available to those paying for the storage. It's not _likely_, as that would negate the usefulness of the whole system if it became an issue, and therefore negate the cash flow into the system. Ultimately what's keeping the storage providers honest, whether they're centralized or decentralized, is that their future profit stream is based on their current behavior, and they presumably care about future profits. Code and SLAs ain't nothing but words.
  - yjftsjthsd-h 6y ago
    In fairness, real-company SLAs also tend to be hand-wavy. People like to pretend it's "they'll never be down more than X" or "if they're ever down more than X they'll give us a refund", but it's usually more like "the service provider said that it didn't count because only one part of their service was 'degraded', and after we threatened to take them to court they threw us a $13 discount'".
  - dntrkv 6y ago
    What if the system had 2 kinds of nodes, storage nodes, and then verification nodes. The storage nodes, store the data. The verification nodes, hit the storage nodes and verify the data is there and available. If the storage node fails a verification attempt, the storage node pays a fee. After a certain amount of failures, it is banned from the network. Disclaimer: I have no idea how these systems currently work.
cle 6y ago
S3 has no durability SLA, only an availability SLA. S3 also explicitly says it is not designed for 100% durability, but for a certian number of nines (99.999999999%).
- karpierz 6y ago
  Blockchain has neither a durability or availability SLA.
  - chrisco255 6y ago
    An SLA just gives you the right to fee reductions or potential compensation for downtime. And not all SLAs are created equal. Meanwhile, for example, many blockchains such as Bitcoin and Ethereum, have 100% uptime. Not 99.99%...100%.
    - capableweb 6y ago
      "uptime" is such a weird concept when talking about decentralized networks as they are local-first and also build with many peers. And which "uptime" is being referred to here, read or write? If it's write, fees have definitely spiked at points to make it mostly useless for most people "using" it, but you can technically still write to it. And read is just always available as long as you have a copy of the data somewhere.
  - cle 6y ago
    I'm not sure that comparison makes much sense, it's like saying that HTTP has no SLA. AFAIU, Filecoin miners are free to offer SLAs for storage, and clients are free to enter into deals only with miners who offer such SLAs.
ChainOfFools 6y ago
> dozens of independent entities or one entity operating dozens of nodes? droplets are cheap, pwned windows boxes even cheaper. they don't even have to be cheap by regular Joe standards, they could be cheap by nation state, or ransomware gang standards. the point is there's no way to prove they are decentralized. this is an intrinsic, structural property of the cryptocurrency model, thus it is not a mere technical defect that will get solved "in the future" proving decentralization would require zero privacy, because you'd have to prove independent nodes are indeed independent and not part of a cartel, which means you'd have to somehow prove who operates each one. nothing but faith-based decentralization here, as with all cryptocurrencies.
yosito 6y ago
In the case of Storj, I believe that the redundancy between storage nodes is sufficient that something like only 20 out of 80 nodes need to respond in order to restore your data.
- deleted 6y ago
  [deleted]
dnr 6y ago
The argument (of the second post) is that their behavior will be correlated because they all have (more or less) the same incentives and cost structure.
px43 6y ago
> SLAs -- something that a blockchain can't really offer. Hold on there, SLAs are actually something that blockchains are great at. Smart contracts provide cryptographic guarantees of service. In several of these types of systems, node runners must put up collateral as "stake" in exchange for collecting fees for their services. If a participant in the network identifies a misbehaving node (violation of SLA) then their stake can get slashed, or burned completely. If a Filecoin node claims to be holding data for someone, but is unable to provide it in a timely manner when legitimately requested to do so, they're in for some serious penalties. The great thing is that the consumer doesn't need to worry about some big company hiring expensive lawyers to weasel out of their contractual obligations. The rules are all readily transparent and enshrined as code that anyone can read and verify the validity of before participating.
- wmf 6y ago
  I think this speaks to the difference between what SLAs really are and what people wishfully think they are. People think an uptime SLA means "we will not go down much" but it really means "one day we will go down hard and we'll give you a refund (which will be insignificantly small compared to the pain we caused you)".

thebean11 6y ago

I looked into glacier pricing a while back it's..tricky. It looks super cheap up front, but gets extremely expensive if you ever need to get the data out of glacier.

prepend 6y ago
Extremely expensive in terms of storage, but still much cheaper than S3. I consider it as a mitigation against disaster and expect to never retrieve other than for testing the process. So it seems good for extremely low probability, high impact events. And I trust amazon to meet their SLA.
teraflop 6y ago
Depending on how long ago you checked it out (relative to the ~9 years since Glacier was first announced) it might be worth another look. It's still quite expensive if you ever need to retrieve data in a hurry, but it used to also have a complicated and unintuitive pricing model that made it extremely easy to accidentally spend orders of magnitude more money than you intended to. They redesigned the pricing structure in 2016 to make it a lot more straightforward.
- gruez 6y ago
  >It's still quite expensive if you ever need to retrieve data in a hurry The real killer is the $0.09/GB egress fees. That dwarfs the $0.00099/GB/month storage fee and the $0.0025/GB bulk retrieval fee.
  - thebean11 6y ago
    yep, this is what I was talking about, didn't have the exact terminology
deleted 6y ago
[deleted]