4 ms·
Really a CP system but with the Availability being five 9s or better (less than one failure in 10^6) How: 1)Hardware - Gobs and Gobs of Hardware and SRE experi
by suprgeek 10y ago
Really a CP system but with the Availability being five 9s or better (less than one failure in 10^6)
How:
1)Hardware - Gobs and Gobs of Hardware and SRE experience
"Spanner is not running over the public Internet — in fact, every Spanner packet flows only over Google-controlled routers and links (excluding any edge links to remote clients). Furthermore, each data center typically has at least three independent fibers connecting it to the private global network, thus ensuring path diversity for every pair of data centers. Similarly, there is redundancy of equipment and paths within a datacenter. Thus normally catastrophic events, such as cut fiber lines, do not lead to partitions or to outages."
2) Ninja 2PC
"Spanner uses two-phase commit (2PC) and strict two-phase locking to ensure isolation and strong consistency. 2PC has been called the “anti-availability” protocol [Hel16] because all members must be up for it to work. Spanner mitigates this by having each member be a Paxos group, thus ensuring each 2PC “member” is highly available even if some of its Paxos participants are down."
- jpmattia 10y ago> with the Availability being five 9s or better (less than one failure in 10^6) Anyone know how exactly this is defined for them? (Time? Queries? Results?)
- bpicolo 10y agoLooks like they define it as "downtime where downtime instance > 30s"?
- btian 10y agoIn general availability is defined in terms of time. https://en.wikipedia.org/wiki/High_availability https://en.wikipedia.org/wiki/High_availability Five-9s means 5 minutes of downtime per year.
- the_cap_theorem 10y agoWell, this is what the spanner website says if you read the fine print: >> This feature is not covered by any SLA So I would guess that you don't get _any_ guarantess. Not five nines and not even one nine.
- ecnahc515 10y agoCurrently yes, but that's because it's in Beta.
- krenoten 10y agoMTBF of 2PC-strapped-to-quorum is no different from MTBF of a 2PC-strapped-to-spof replicas. MTTR is bounded by reelection latency, rather than replica recovery, although you still may eat a write amplification cost for rereplication. write amplification is 3-5x of non-quorum-backed 2PC system, depending on replication ensemble size. google further multiplies write amplification with geo-redundancy, so bump that WA by another 3x+. it's an insanely high cost to pay for availability, but for an advertising company it's important to count the beans accurately.
- jasonwatkinspdx 10y agoYou're making some assumptions. For example if you use epaxos for the quorums there is no unavailability due to a leader failure and re-election. Even then, re-election is likely going to be a lot faster than any sort of fault tolerant 2PC coordinator recovery protocol. You're exaggerating the write amplification. The above offering is single region. Google says they do 5 way geo in their various papers/etc, and there's no way they're paying some cumulative 15 copy write amplification as you imply.
- krenoten 10y agoepaxos is not general purpose replication. It leaves the problem of reconciling multiple replication streams to another layer. I've heard that they do 20+WA (distributed, before you consider on-disk WA) for much of it. shrug
- findateamfirst 10y agoI've worked on a storage system that used 9. 15 doesn't seem impossible to me.
- ec109685 10y ago9x write amplification isn't that high. If you are setup with a master slave configuration, that creates write amplification and for reliability, you probably have some sort of raid, that increase the number of places writes go to as well.