3 ms·

incidentally, i have no idea why everyone treats s3 as so reliable. all these services offer backups to s3. well folks, i think that s3 is bound to fail some da

incidentally, i have no idea why everyone treats s3 as so reliable. all these services offer backups to s3. well folks, i think that s3 is bound to fail some day and some good peace of mind could be manufactured by mirroring s3 files to a few different mirrors.

ChuckN 16y ago
s3 is a representation of 3 geographically diverse mirrored stores. In addition, versioning is available to address the issue of a rogue credentialed service deleting objects. Your point is still valid. Vendor and application level redundancy is still desirable if one can afford it.
lsc 16y ago
They sound like they have a pretty good system, and if you can afford five petabytes and don't do much moving in and out, they are cheap, too. On the other hand, their system is non-standard. It's not used by or tested by anyone else. And it sounds like a fairly complex system. The fact that they haven't lost a whole lot of data yet means that they must be pretty good... but the more complex (and unusual) a system, the more I would fear a failure caused by a software bug.
- cperciva 16y ago
  [S3] sounds like a fairly complex system That depends on your definition of "complex", of course. In terms of the number of lines of code, I'd guess that S3 is significantly simpler than a typical filesystem -- key-blob CRUD is a much simpler thing to implement than directory trees, file metadata, memory-mapped files, and all the other horrible messes filesystems need to handle. As far as data loss specifically goes: S3 is in a good place. Most of the complexity of a system like S3 is in finding data -- that is, routing requests to the right node -- not in merely not losing data. I don't think it's a coincidence at all that despite several outages over the past few years which have made S3 unavailable, they've always been able to bring the service back online with no loss of data.
  - lsc 16y ago
    You'd know much more about these sorts of things than I would, but most filesystems largely ignore disk corruption. If amazon did that, we'd be seeing a lot more trouble than we have. This is where I'm seeing the complexity; detecting and recovering from subtle errors. I've gone a short way down that path and from where I stand it looks quite hairy. Besides, I've seen data loss caused by ext3 on my own systems that was not caused by disk corruption, so really, at s3's scale, it's got to function (and seems to have been functioning) much better than a regular filesystem.
    - cperciva 16y ago
      S3 has an advantage over regular filesystems there too. S3 doesn't need to provide single-node durability as long as data loss events on different nodes are uncorrelated; so they can protect data with cryptographic checksums and throw it away (from an individual node) at the first sign of corruption, knowing that they'll still have it on all the other nodes.

ChuckN 16y ago

s3 is a representation of 3 geographically diverse mirrored stores. In addition, versioning is available to address the issue of a rogue credentialed service deleting objects. Your point is still valid. Vendor and application level redundancy is still desirable if one can afford it.

lsc 16y ago

They sound like they have a pretty good system, and if you can afford five petabytes and don't do much moving in and out, they are cheap, too. On the other hand, their system is non-standard. It's not used by or tested by anyone else. And it sounds like a fairly complex system. The fact that they haven't lost a whole lot of data yet means that they must be pretty good... but the more complex (and unusual) a system, the more I would fear a failure caused by a software bug.

cperciva 16y ago
[S3] sounds like a fairly complex system That depends on your definition of "complex", of course. In terms of the number of lines of code, I'd guess that S3 is significantly simpler than a typical filesystem -- key-blob CRUD is a much simpler thing to implement than directory trees, file metadata, memory-mapped files, and all the other horrible messes filesystems need to handle. As far as data loss specifically goes: S3 is in a good place. Most of the complexity of a system like S3 is in finding data -- that is, routing requests to the right node -- not in merely not losing data. I don't think it's a coincidence at all that despite several outages over the past few years which have made S3 unavailable, they've always been able to bring the service back online with no loss of data.
- lsc 16y ago
  You'd know much more about these sorts of things than I would, but most filesystems largely ignore disk corruption. If amazon did that, we'd be seeing a lot more trouble than we have. This is where I'm seeing the complexity; detecting and recovering from subtle errors. I've gone a short way down that path and from where I stand it looks quite hairy. Besides, I've seen data loss caused by ext3 on my own systems that was not caused by disk corruption, so really, at s3's scale, it's got to function (and seems to have been functioning) much better than a regular filesystem.
  - cperciva 16y ago
    S3 has an advantage over regular filesystems there too. S3 doesn't need to provide single-node durability as long as data loss events on different nodes are uncorrelated; so they can protect data with cryptographic checksums and throw it away (from an individual node) at the first sign of corruption, knowing that they'll still have it on all the other nodes.