5 ms·

Also keep in mind all the data is in a single datacenter, so if something like OVH happens to backblaze, you'll probably lose the data I mainly only use them f

Also keep in mind all the data is in a single datacenter, so if something like OVH happens to backblaze, you'll probably lose the data I mainly only use them for backups instead of production data EDIT: As of 2019 there are multiple datacenters, but it doesn't seem like the data is stored redundantly across them

jiofih 6y ago
They have four datacenters, three in the US and one in the EU. Details are not given regarding how the 20 shards that comprise your data are distributed geographically, but they state eight 9s of reliability.
- qw3rty01 6y ago
  Oh how new is that? Last time I was looking through they just said their datacenter was in a bunker so nothing outside of a major natural disaster would affect it.
  - jiofih 6y ago
    Since 2019 it seems: https://help.backblaze.com/hc/en-us/articles/217667468-B2-Security-and-Redundancy?mobile_site=true https://help.backblaze.com/hc/en-us/articles/217667468-B2-Se...
- spondyl 6y ago
  Just to clarify, Backblaze states eight 9s of durability, not reliability. Durability refers to the idea that your data will still be retrievable (ie no corruption) similar to S3's claim of eleven 9s of durability. Reliability however would be say; the actual availability of the Web UI or API server that you download your data through. If it were down, that wouldn't impact the actual integrity of the data itself. For anyone interested, any more than about five 9s of reliability is basically impossible anyway when it comes to human intervention. As an example, 6 nines of reliability would allow you 3 seconds of unavailability a year so 250 milliseconds of unavailability a month. From a users point of view, being "unavailable" includes everything from going through a tunnel and having your mobile connection drop out to a shark biting one of the undersea cables in the middle of the ocean. As you might imagine, with a human involved, they couldn't even get acknowledge an alert fast enough to meet that deadline let alone actually going about doing any repairs and diagnosis :) It could be spread over multiple instances and redundant hardware as well but as with any system being touched by humans, it's near guaranteed that something will go wrong eventually.
  - andrewaylett 6y ago
    At that scale, a complete outage is unlikely. I have services which haven't gone down _at all_ for longer than a year. But we lose requests every now and then -- during a deploy, or due to a bug. So we've moved from a time-based view of outage to a request-based view. This helps, too, as it lets us build out services to be more reliable in combination, rather than less reliable. With retries and fail-over, an outage in an entire region may not necessarily result in any user requests failing. For scale, pre-pandemic our published figures claimed >100M MAU. I find https://andrewaylett.github.io/multi-burn-rate-calculator/ https://andrewaylett.github.io/multi-burn-rate-calculator/ helpful for visualising error rates -- largely cribbed from the project it's forked from :) but with the tweakables switched around and the time between alert and error budget exhaustion in the tooltip. It's worth noting that we only evaluate our alerts at most once a minute.
    - chrisandchris 6y ago
      Said OVH. Then their datacenter burned to the grounds. Said Oracle. Then their DNS was misconfigured and their whole cloud went offline for 2 hours [1]. Shit happens, always, at all scales. [1] https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj Edit: typo
      - andrewaylett 6y ago
        OVH didn't suffer a complete outage. If you were relying on that single DC, then you're probably not sufficiently large for this to apply to you. But perhaps my point wasn't clearly enough made: a claim of "100% uptime" on a service level isn't particularly _useful_ when our users still only see a 99.9% success rate.
    - chaz6 6y ago
      I think the weak point is their domain name. I think cloud providers should have a second domain, with a different registrar and managed sompletely independently, so that if one is subject to a problem (hijacking, dns outage, etc) clients can fallback to the alternative domain.
- Dylan16807 6y ago
  Given that the parity can only lose 3 shards it's easy to show that a datacenter loss will always result in data loss, so there's no reason to distribute and we should assume that all 20 shards are always going to be in the same place.
- dx034 6y ago
  A fire like that in SBG a few days ago is extremely rare. The data lost there compared to all data stored in data centers probably justifies eight 9s. The problem with these numbers is that it's extremely unlikely to lose data but if you do, the event is so severe that everything might be gone. That's why I feel more comfortable storing data across two "unreliable" providers with a lot of physical space in between them rather than one super reliable provider. You also have to consider that data loss can result of simple things such as an account that gets blocked for stupid reasons. If you want to be safe you always need to have your data with at least 2 providers.
dx034 6y ago
The EU and US data centers are completely separated. You can't even use both from the same account. To change, you need to open a completely new account selecting EU at the start. So it's not really easy to use both at the same time and there's definitely no redundancy across data centers if you're a customer in the EU.

jiofih 6y ago

They have four datacenters, three in the US and one in the EU. Details are not given regarding how the 20 shards that comprise your data are distributed geographically, but they state eight 9s of reliability.

qw3rty01 6y ago
Oh how new is that? Last time I was looking through they just said their datacenter was in a bunker so nothing outside of a major natural disaster would affect it.
- jiofih 6y ago
  Since 2019 it seems: https://help.backblaze.com/hc/en-us/articles/217667468-B2-Security-and-Redundancy?mobile_site=true https://help.backblaze.com/hc/en-us/articles/217667468-B2-Se...
spondyl 6y ago
Just to clarify, Backblaze states eight 9s of durability, not reliability. Durability refers to the idea that your data will still be retrievable (ie no corruption) similar to S3's claim of eleven 9s of durability. Reliability however would be say; the actual availability of the Web UI or API server that you download your data through. If it were down, that wouldn't impact the actual integrity of the data itself. For anyone interested, any more than about five 9s of reliability is basically impossible anyway when it comes to human intervention. As an example, 6 nines of reliability would allow you 3 seconds of unavailability a year so 250 milliseconds of unavailability a month. From a users point of view, being "unavailable" includes everything from going through a tunnel and having your mobile connection drop out to a shark biting one of the undersea cables in the middle of the ocean. As you might imagine, with a human involved, they couldn't even get acknowledge an alert fast enough to meet that deadline let alone actually going about doing any repairs and diagnosis :) It could be spread over multiple instances and redundant hardware as well but as with any system being touched by humans, it's near guaranteed that something will go wrong eventually.
- andrewaylett 6y ago
  At that scale, a complete outage is unlikely. I have services which haven't gone down _at all_ for longer than a year. But we lose requests every now and then -- during a deploy, or due to a bug. So we've moved from a time-based view of outage to a request-based view. This helps, too, as it lets us build out services to be more reliable in combination, rather than less reliable. With retries and fail-over, an outage in an entire region may not necessarily result in any user requests failing. For scale, pre-pandemic our published figures claimed >100M MAU. I find https://andrewaylett.github.io/multi-burn-rate-calculator/ https://andrewaylett.github.io/multi-burn-rate-calculator/ helpful for visualising error rates -- largely cribbed from the project it's forked from :) but with the tweakables switched around and the time between alert and error budget exhaustion in the tooltip. It's worth noting that we only evaluate our alerts at most once a minute.
  - chrisandchris 6y ago
    Said OVH. Then their datacenter burned to the grounds. Said Oracle. Then their DNS was misconfigured and their whole cloud went offline for 2 hours [1]. Shit happens, always, at all scales. [1] https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj Edit: typo
    - andrewaylett 6y ago
      OVH didn't suffer a complete outage. If you were relying on that single DC, then you're probably not sufficiently large for this to apply to you. But perhaps my point wasn't clearly enough made: a claim of "100% uptime" on a service level isn't particularly _useful_ when our users still only see a 99.9% success rate.
  - chaz6 6y ago
    I think the weak point is their domain name. I think cloud providers should have a second domain, with a different registrar and managed sompletely independently, so that if one is subject to a problem (hijacking, dns outage, etc) clients can fallback to the alternative domain.
Dylan16807 6y ago
Given that the parity can only lose 3 shards it's easy to show that a datacenter loss will always result in data loss, so there's no reason to distribute and we should assume that all 20 shards are always going to be in the same place.
dx034 6y ago
A fire like that in SBG a few days ago is extremely rare. The data lost there compared to all data stored in data centers probably justifies eight 9s. The problem with these numbers is that it's extremely unlikely to lose data but if you do, the event is so severe that everything might be gone. That's why I feel more comfortable storing data across two "unreliable" providers with a lot of physical space in between them rather than one super reliable provider. You also have to consider that data loss can result of simple things such as an account that gets blocked for stupid reasons. If you want to be safe you always need to have your data with at least 2 providers.

dx034 6y ago

The EU and US data centers are completely separated. You can't even use both from the same account. To change, you need to open a completely new account selecting EU at the start. So it's not really easy to use both at the same time and there's definitely no redundancy across data centers if you're a customer in the EU.