5 ms·
That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is store
by Johnny555 8y ago
That could be a power surge taking out a whole rack
This failure mode, at least, is already accounted for by sharding data across cabinets:
Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet.
https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...
However, they don't seem to offer multi-datacenter (or multi-region) redundancy so are still susceptible to a datacenter fire/failure.
In comparison, AWS S3 distributes data across 3 AZ's (datacenters), and you can further replicate across regions if you choose. Though you pay for that added redundancy in 3 - 4X higher cost.
- gm-conspiracy 8y agoI thought AZs were in the same physical location, just separate networks, no?
- coderholic 8y agoNo, they're different physical locations. From https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce...: > Each AWS Region has multiple, isolated locations known as Availability Zones. And from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions-availability-zones https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-re...: > Each region is completely independent. Each Availability Zone is isolated, but the Availability Zones in a region are connected through low-latency links.
- jsjohnst 8y agoNope. They aren’t even in the same city. For example, each of the AZs in us-west-2 are separated by about 50-60 miles. This doesn’t list all the locations, but is a good map to get an idea: https://www.google.com/maps/d/u/0/viewer?ll=50.95842700000002%2C4.130858999999987&spn=26.684562%2C56.25&t=m&msa=0&z=4&source=embed&ie=UTF8&mid=1m6v8XPxwp0Dx4THiakRKFkZwIGE https://www.google.com/maps/d/u/0/viewer?ll=50.9584270000000...
- fs111 8y agoThat map is incorrect: us-west-2a, 2b and 2c are not static names for the AZs. Every user gets their own mapping of which physical location is a, which one is b and which one is c. My us-west-2a may be your us-west-2c. They are not the same.
- gregsadetsky 8y agoExtremely interesting, thanks for this clarification. For those curious, it's documented here[0] -- search for "To ensure that resources are distributed across the Availability Zones for a region"... [0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions-availability-zones https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-re...
- jsjohnst 8y agoYes, I know they pseudo randomize the allocations. That is irrelevant to my core point, that each AZ is not just separate networks in the same building or even adjacent buildings, but rather they are truly isolated by a non-insignificant distance of somewhere around 50mi on average.
- Johnny555 8y agoIt's a cool map, but I can't find any reference for the source of the data center locations. AWS purposely doesn't publish that information, and while I can believe it's possible to crowdsource the data by doing a little sleuthing (or working for certain vendors), it's hard to trust the map without knowing the sources.
- jsjohnst 8y agoThere’s a The Atlantic article[0] that explains how to source the data pretty easily if you are willing to put in the effort. https://www.theatlantic.com/technology/archive/2016/01/amazon-web-services-data-center/423147/ https://www.theatlantic.com/technology/archive/2016/01/amazo...
- Johnny555 8y agoThey are separate datacenters, I don't think they make any promises about how far they are apart from each other, but at least in some regions they are 10+ miles apart. The AWS Cloud infrastructure is built around AWS Regions and Availability Zones. An AWS Region is a physical location in the world where we have multiple Availability Zones. Availability Zones consist of one or more discrete data centers, each with redundant power, networking, and connectivity, housed in separate facilities https://docs.aws.amazon.com/aws-technical-content/latest/aws-overview/global-infrastructure.html https://docs.aws.amazon.com/aws-technical-content/latest/aws...
- gm-conspiracy 8y agoSo, a single AZ could span across multiple physical locations, as well? Am I reading that correctly?
- strong_silent_t 8y agoOthers have answered, but I think the general principle for a Region is that between the AZs you have less than 1ms latency, physical separation between the data centers, but they may still be in the same floodplain, be able to be hit by the same hurricane, etc. More info here if you want to see how regions are structured at a high level of detail: https://youtu.be/AyOAjFNPAbA?t=15m21s https://youtu.be/AyOAjFNPAbA?t=15m21s
- catwell 8y agoMulti-region redundancy was on the roadmap for their B2 offering 10 months ago, not sure if it has shipped yet. https://news.ycombinator.com/item?id=15125643 https://news.ycombinator.com/item?id=15125643
- ChuckMcM 8y agoA better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that bought a filer with several hundred drives started seeing large numbers fail suddenly. Up to the point they failed, not any indication. But it was run time hours related and thus correlated to all drives that started running at the same time. If you had put those drives in different data centers on different racks on different servers it wouldn't have mattered if you started 20 at the same time to be your 'tome', when they started failing it was possible to lose them all in a week.
- tlb 8y agoThere was also a failure mode in Seagate drives where the bearing increased in stiction. As long as it was spinning, there was no problem. But if you spun it down, it might not spin up again. If you had a group of disks powered up for a long time, many could fail together at the next power cycle. A chaos monkey that randomly powers down disks one at a time can prevent this.
- deleted 8y ago[deleted]
- Dylan16807 8y agoThough that sounds like a fixable failure mode, so it's more annoyance than tragedy.
- tinus_hn 8y agoYou can tell from the reports they put out that Backblaze doesn’t put all their eggs in one basket, they use different brands and drive types.