6 ms·

For certain workloads these storage pods are much, much cheaper than S3. Anything where you are storing files that rapidly become stale, but still need to be i

by jewel 12y ago

For certain workloads these storage pods are much, much cheaper than S3. Anything where you are storing files that rapidly become stale, but still need to be instantly accessible for the rare random request. I've only looked at it from the perspective of video files, though. Where I work we add a gigabyte of data per user per month. Eventually our S3 storage bill is going to be our largest cost due to compounding growth.

res0nat0r 12y ago
Any idea what the durability of Backblaze is? S3 is 99.999999999%, is Backblaze anywhere near that? I can't seem to find it in their FAQ.
- Retric 12y ago
  Amazon is no where near that. Software glitches have already distroyed s3 data. More importantly anything over 99.999% has to take into account vary low probability events like nuclear war.
  - res0nat0r 12y ago
    Their FAQ that has been in place for years needs to be updated: http://aws.amazon.com/s3/faqs/ http://aws.amazon.com/s3/faqs/
- jewel 12y ago
  S3's number is just marketing. It's going to be that durable until a black swan event wipes out 0.1% of all files. I imagine they keep multiple copies of every file, spread across multiple data centers. It's durable against hardware failure of all varieties, but it's still vulnerable to a software bug. It's also possibly vulnerable to certain types of natural disasters. IIRC backblaze employs a similar strategy. They have the advantage that your home computer is still storing a copy, so even if they have a minor catastrophe they can recover by simply having their client re-upload those files.
  - ghshephard 12y ago
    I think the S3 number is a floor, not a ceiling. That is, you should expect to be losing that much of data, but, given an unplanned event, obviously a great deal more data can be lost.
  - mikeash 12y ago
    Human disasters too. I'd wager the odds of a catastrophic massive nuclear war are higher than one in ten million per year. Not that I expect Amazon (or much of anyone) to protect against that, nor would the integrity of my S3 data be a priority afterwards, but it makes the claim kind of absurd. Edit: reading more closely, their durability number is per object. It's one object lost per 100 billion objects per year. The 10 million number comes from a hypothetical situation where you store 10,000 objects. I don't know how many objects S3 stores altogether, but if we say it's a billion (presumably a vast underestimate) then that would imply that the probability of losing all objects in the system over the coming year is one in 100 quadrillion. I don't think this planet is that safe.
    - res0nat0r 12y ago
      Not sure why it is absurd due to the redundancy they've implemented + data checksumming. Plus AWS hit two trillion objects a bit over a year ago. http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objec...
      - mikeash 12y ago
        As great as S3 is, it's still confined to Earth. Seems to me that the odds of a planet-wide disaster taking out all Amazon infrastructure (as well as certain less important things like human civilization) are higher than the odds they're giving of a catastrophic S3 failure.
        res0nat0r 12y ago
        Sure. The AFR of your Western Digital desktop HD doesn't factor in and lower it's reliability slightly because of the slim chance that a 747 is going to crash into your house and destroy your home. Barring unforeseen acts of God, the 9's listed above apply and you just have to personally weigh if you think the risk of S3 losing multiple datacenters is high enough for you to risk storing your data there.
dublinben 12y ago
Couldn't you save a ton of money by explicitly moving old data to 'cold storage,' and keeping it on Glacier? Users would probably understand if data older than a year takes a few minutes to be retrieved.
- travem 12y ago
  According to the Glacier docs "Most Amazon Glacier jobs take about four hours to complete" [1] so a few minutes is a bit of an underestimate. [1] http://docs.aws.amazon.com/amazonglacier/latest/dev/downloading-an-archive.html http://docs.aws.amazon.com/amazonglacier/latest/dev/download...

by jewel 12y ago

res0nat0r 12y ago

Any idea what the durability of Backblaze is? S3 is 99.999999999%, is Backblaze anywhere near that? I can't seem to find it in their FAQ.

Retric 12y ago
Amazon is no where near that. Software glitches have already distroyed s3 data. More importantly anything over 99.999% has to take into account vary low probability events like nuclear war.
- res0nat0r 12y ago
  Their FAQ that has been in place for years needs to be updated: http://aws.amazon.com/s3/faqs/ http://aws.amazon.com/s3/faqs/
jewel 12y ago
S3's number is just marketing. It's going to be that durable until a black swan event wipes out 0.1% of all files. I imagine they keep multiple copies of every file, spread across multiple data centers. It's durable against hardware failure of all varieties, but it's still vulnerable to a software bug. It's also possibly vulnerable to certain types of natural disasters. IIRC backblaze employs a similar strategy. They have the advantage that your home computer is still storing a copy, so even if they have a minor catastrophe they can recover by simply having their client re-upload those files.
- ghshephard 12y ago
  I think the S3 number is a floor, not a ceiling. That is, you should expect to be losing that much of data, but, given an unplanned event, obviously a great deal more data can be lost.
- mikeash 12y ago
  Human disasters too. I'd wager the odds of a catastrophic massive nuclear war are higher than one in ten million per year. Not that I expect Amazon (or much of anyone) to protect against that, nor would the integrity of my S3 data be a priority afterwards, but it makes the claim kind of absurd. Edit: reading more closely, their durability number is per object. It's one object lost per 100 billion objects per year. The 10 million number comes from a hypothetical situation where you store 10,000 objects. I don't know how many objects S3 stores altogether, but if we say it's a billion (presumably a vast underestimate) then that would imply that the probability of losing all objects in the system over the coming year is one in 100 quadrillion. I don't think this planet is that safe.
  - res0nat0r 12y ago
    Not sure why it is absurd due to the redundancy they've implemented + data checksumming. Plus AWS hit two trillion objects a bit over a year ago. http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objects-11-million-requests-second/ http://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-objec...
    - mikeash 12y ago
      As great as S3 is, it's still confined to Earth. Seems to me that the odds of a planet-wide disaster taking out all Amazon infrastructure (as well as certain less important things like human civilization) are higher than the odds they're giving of a catastrophic S3 failure.
      - res0nat0r 12y ago
        Sure. The AFR of your Western Digital desktop HD doesn't factor in and lower it's reliability slightly because of the slim chance that a 747 is going to crash into your house and destroy your home. Barring unforeseen acts of God, the 9's listed above apply and you just have to personally weigh if you think the risk of S3 losing multiple datacenters is high enough for you to risk storing your data there.

dublinben 12y ago

Couldn't you save a ton of money by explicitly moving old data to 'cold storage,' and keeping it on Glacier? Users would probably understand if data older than a year takes a few minutes to be retrieved.

travem 12y ago
According to the Glacier docs "Most Amazon Glacier jobs take about four hours to complete" [1] so a few minutes is a bit of an underestimate. [1] http://docs.aws.amazon.com/amazonglacier/latest/dev/downloading-an-archive.html http://docs.aws.amazon.com/amazonglacier/latest/dev/download...