4 ms·

Just to clarify, Backblaze states eight 9s of durability, not reliability. Durability refers to the idea that your data will still be retrievable (ie no corrup

Just to clarify, Backblaze states eight 9s of durability, not reliability. Durability refers to the idea that your data will still be retrievable (ie no corruption) similar to S3's claim of eleven 9s of durability. Reliability however would be say; the actual availability of the Web UI or API server that you download your data through. If it were down, that wouldn't impact the actual integrity of the data itself. For anyone interested, any more than about five 9s of reliability is basically impossible anyway when it comes to human intervention. As an example, 6 nines of reliability would allow you 3 seconds of unavailability a year so 250 milliseconds of unavailability a month. From a users point of view, being "unavailable" includes everything from going through a tunnel and having your mobile connection drop out to a shark biting one of the undersea cables in the middle of the ocean. As you might imagine, with a human involved, they couldn't even get acknowledge an alert fast enough to meet that deadline let alone actually going about doing any repairs and diagnosis :) It could be spread over multiple instances and redundant hardware as well but as with any system being touched by humans, it's near guaranteed that something will go wrong eventually.

andrewaylett 6y ago
At that scale, a complete outage is unlikely. I have services which haven't gone down _at all_ for longer than a year. But we lose requests every now and then -- during a deploy, or due to a bug. So we've moved from a time-based view of outage to a request-based view. This helps, too, as it lets us build out services to be more reliable in combination, rather than less reliable. With retries and fail-over, an outage in an entire region may not necessarily result in any user requests failing. For scale, pre-pandemic our published figures claimed >100M MAU. I find https://andrewaylett.github.io/multi-burn-rate-calculator/ https://andrewaylett.github.io/multi-burn-rate-calculator/ helpful for visualising error rates -- largely cribbed from the project it's forked from :) but with the tweakables switched around and the time between alert and error budget exhaustion in the tooltip. It's worth noting that we only evaluate our alerts at most once a minute.
- chrisandchris 6y ago
  Said OVH. Then their datacenter burned to the grounds. Said Oracle. Then their DNS was misconfigured and their whole cloud went offline for 2 hours [1]. Shit happens, always, at all scales. [1] https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj Edit: typo
  - andrewaylett 6y ago
    OVH didn't suffer a complete outage. If you were relying on that single DC, then you're probably not sufficiently large for this to apply to you. But perhaps my point wasn't clearly enough made: a claim of "100% uptime" on a service level isn't particularly _useful_ when our users still only see a 99.9% success rate.
- chaz6 6y ago
  I think the weak point is their domain name. I think cloud providers should have a second domain, with a different registrar and managed sompletely independently, so that if one is subject to a problem (hijacking, dns outage, etc) clients can fallback to the alternative domain.
  - sp8 6y ago
    That literally happened, they blogged about it recently. https://www.backblaze.com/blog/recent-outages-why-we-accelerated-registry-changes/ https://www.backblaze.com/blog/recent-outages-why-we-acceler...
myself248 6y ago
You say that, but telephony used to be obsessed with uptime. These are the people who invented live patching of a running system, after all. Case in point, the #1 ESS telephone switch was designed for less than 2 hours of cumulative downtime over a 40-year service life. In practice, most achieved less than 1 hour. (How many nines is that?) Which is to say, individual subscriber lines can have all sorts of faults, but they only affect that subscriber. Trunks between offices can fail, but they only affect a certain number of circuits. The central call processing ability, of the switch to react appropriately to lines changing state and numbers being dialed, for calls to be completed if both stations are available, is what had to meet that target. I was skeptical when I first heard this, so next time I was in an office that still had a #1 (actually a 1A, it had been upgraded in the late 80s), I talked to the switchman for a bit, and he showed me the downtime counter. It's a mechanical thing like an odometer, and once a second when the switch is executing its main loop, it touches a register that keeps the counter from incrementing. If the processor halts or isn't processing calls for some reason, the counter starts counting. The switch was installed in the mid-70s, and from the moment it took over for its crossbar predecessor, including an in-place processor upgrade, it had logged less than an hour. Most of that came a few seconds at a time when swapping between active processors during software upgrades, he said. At the time (this was around 2002 I think) it was slated to be replaced with a DMS100, but the replacement activity hadn't commenced yet. I don't know what sort of reliability numbers the DMS machines achieve, but they'd do well to match their predecessors.
- cperciva 6y ago
  To answer your question: 1 hour of downtime after 40 years is 5.5 "nines": 99.9997% availability.
  - irjustin 6y ago
    So to the grandparent comment, 6x 9's of reliability is unrealistic to guarantee as an SLA. If you get better, awesome, but SLA? Too unrealistic.
    - d1sxeyes 6y ago
gumby 6y ago
> For anyone interested, any more than about five 9s of reliability is basically impossible anyway when it comes to human intervention. As an example, 6 nines of reliability would allow you 3 seconds of unavailability a year so 250 milliseconds of unavailability a month. Not sure what the qualifier “when it comes to human intervention” means, but if I ignore it then five nines is quite standard in certain sectors. For example phone switch SLA (back when I was in the biz) was measured in minutes per decade (as in “cumulative unavailability under 1/2 minutes per decade”). Large baseline power plants can and must run uninterrupted for decades. Of course it’s a systems issue not a point solution.
- spondyl 6y ago
  Ah, apologies. I realise in hindsight that I was rambling from the point of view of offering five nines for software that is inherently flaky/unreliable. Companies where developers cycle through and knowledge is lost, technical debt accumulates and systems are used almost counter to their intended purpose (ie Redis as a database) In that sense, it's like constantly massaging applications to stay alive or at a reasonable level of service and so hence the assumption that someone will be paged and respond in order to preempt a failure or restore service during an outage.

andrewaylett 6y ago

At that scale, a complete outage is unlikely. I have services which haven't gone down _at all_ for longer than a year. But we lose requests every now and then -- during a deploy, or due to a bug. So we've moved from a time-based view of outage to a request-based view. This helps, too, as it lets us build out services to be more reliable in combination, rather than less reliable. With retries and fail-over, an outage in an entire region may not necessarily result in any user requests failing. For scale, pre-pandemic our published figures claimed >100M MAU. I find https://andrewaylett.github.io/multi-burn-rate-calculator/ https://andrewaylett.github.io/multi-burn-rate-calculator/ helpful for visualising error rates -- largely cribbed from the project it's forked from :) but with the tweakables switched around and the time between alert and error budget exhaustion in the tooltip. It's worth noting that we only evaluate our alerts at most once a minute.

chrisandchris 6y ago
Said OVH. Then their datacenter burned to the grounds. Said Oracle. Then their DNS was misconfigured and their whole cloud went offline for 2 hours [1]. Shit happens, always, at all scales. [1] https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj Edit: typo
- andrewaylett 6y ago
  OVH didn't suffer a complete outage. If you were relying on that single DC, then you're probably not sufficiently large for this to apply to you. But perhaps my point wasn't clearly enough made: a claim of "100% uptime" on a service level isn't particularly _useful_ when our users still only see a 99.9% success rate.
chaz6 6y ago
I think the weak point is their domain name. I think cloud providers should have a second domain, with a different registrar and managed sompletely independently, so that if one is subject to a problem (hijacking, dns outage, etc) clients can fallback to the alternative domain.
- sp8 6y ago
  That literally happened, they blogged about it recently. https://www.backblaze.com/blog/recent-outages-why-we-accelerated-registry-changes/ https://www.backblaze.com/blog/recent-outages-why-we-acceler...

myself248 6y ago

You say that, but telephony used to be obsessed with uptime. These are the people who invented live patching of a running system, after all. Case in point, the #1 ESS telephone switch was designed for less than 2 hours of cumulative downtime over a 40-year service life. In practice, most achieved less than 1 hour. (How many nines is that?) Which is to say, individual subscriber lines can have all sorts of faults, but they only affect that subscriber. Trunks between offices can fail, but they only affect a certain number of circuits. The central call processing ability, of the switch to react appropriately to lines changing state and numbers being dialed, for calls to be completed if both stations are available, is what had to meet that target. I was skeptical when I first heard this, so next time I was in an office that still had a #1 (actually a 1A, it had been upgraded in the late 80s), I talked to the switchman for a bit, and he showed me the downtime counter. It's a mechanical thing like an odometer, and once a second when the switch is executing its main loop, it touches a register that keeps the counter from incrementing. If the processor halts or isn't processing calls for some reason, the counter starts counting. The switch was installed in the mid-70s, and from the moment it took over for its crossbar predecessor, including an in-place processor upgrade, it had logged less than an hour. Most of that came a few seconds at a time when swapping between active processors during software upgrades, he said. At the time (this was around 2002 I think) it was slated to be replaced with a DMS100, but the replacement activity hadn't commenced yet. I don't know what sort of reliability numbers the DMS machines achieve, but they'd do well to match their predecessors.

cperciva 6y ago
To answer your question: 1 hour of downtime after 40 years is 5.5 "nines": 99.9997% availability.
- irjustin 6y ago
  So to the grandparent comment, 6x 9's of reliability is unrealistic to guarantee as an SLA. If you get better, awesome, but SLA? Too unrealistic.
  - d1sxeyes 6y ago

gumby 6y ago

> For anyone interested, any more than about five 9s of reliability is basically impossible anyway when it comes to human intervention. As an example, 6 nines of reliability would allow you 3 seconds of unavailability a year so 250 milliseconds of unavailability a month. Not sure what the qualifier “when it comes to human intervention” means, but if I ignore it then five nines is quite standard in certain sectors. For example phone switch SLA (back when I was in the biz) was measured in minutes per decade (as in “cumulative unavailability under 1/2 minutes per decade”). Large baseline power plants can and must run uninterrupted for decades. Of course it’s a systems issue not a point solution.

spondyl 6y ago
Ah, apologies. I realise in hindsight that I was rambling from the point of view of offering five nines for software that is inherently flaky/unreliable. Companies where developers cycle through and knowledge is lost, technical debt accumulates and systems are used almost counter to their intended purpose (ie Redis as a database) In that sense, it's like constantly massaging applications to stay alive or at a reasonable level of service and so hence the assumption that someone will be paged and respond in order to preempt a failure or restore service during an outage.