10 ms·

> 99.99% availability That seems... really bad, for the Core Core Service that Everything Depends On.

by dymk 3y ago

abeppu 3y ago
S3 standard storage class also says it's 99.99% availability so ... I wonder how hard it is to get beyond that? https://aws.amazon.com/s3/faqs/ https://aws.amazon.com/s3/faqs/
- sosodev 3y ago
  It's extremely hard to get availability higher than that. Possible but not something companies are willing to promise in an SLA.
  - awill 3y ago
    I'd say it's not so much that it's really hard, it's that the compromises aren't worth it. note, I worked on S3 2015-2017
    - abeppu 3y ago
      So ... e.g. for 5 nines, the erasure code configuration would demand that do more writes in more locations and the cost impact is too high?
      - jl6 3y ago
        I imagine that as you add more nines, you start to hit more problems that are more out of your control. Like, how many nines do your backup diesel generators have?
      - skrtskrt 3y ago
        or replication if you're not erasure coding. So yeah you need more storage-optimized server racks and all the associated manpower and maintenance, you also need them to be distributed across different datacenters and zones which of course also impacts latency and your ability to provide some appearance of consistency, then you also need the same distribution for stateless services serving the data. On and on and on and you might be nearly doubling cost to get to an extra 9 that almost all of your customers won't care about.
    - sosodev 3y ago
      That's a fair point. It would be quite possible if the customers were willing to pay a lot more. I worked on DigitalOcean's object storage for about a year not long ago. Makes sense that those of us who have been in this space would be interested in this article haha.
- foobiekr 3y ago
  Absent here is any statement about how available is defined in terms of being _meaningfully up_ like iops or bps.
- CobrastanJorji 3y ago
  It's a cost problem. You can absolutely get higher availability, but it requires more hardware, uses more networking, and it makes maintenance and feature development more expensive. Blob storage customers are very price sensitive, but they also want lots of availability. That's why you'll see various "tiers" of storage, to try and capture all the combinations of availability and storage cost that various kinds of customers might want. Pretty much nobody wants to pay twice the cost to turn an exabyte of 99.99% available data to 99.995% available data, so it's not a product.
petters 3y ago
Not really bad. It’s less than an hour per year. Most users would not notice.
sosodev 3y ago
How is that bad? 99.99% means the system is down for less than an hour each year. https://uptime.is/99.99 https://uptime.is/99.99
- foobiekr 3y ago
  of course, it actually depends on how they define "up."
rbanffy 3y ago
When you build anything on it, you know it may not be available for a couple hours a year and architect around it. 99.99 isn't bad at all. Your ISP probably has worse SLAs.
gumby 3y ago
By comparison the old US POTS phone system was required to maintain downtime of less than five minutes per decade. I have no idea what rules, if any, apply to the modern wireless systems. Note: The consent decree, the actual legal obligation of the Bell system, actually specified the percentage of times someone would pick up the phone and not get a dial tone or operator. It was surprisingly high -- IIRC something like 2%, which is why you can see people toggling the hook in old movies. The five minutes per decade constraint I know because some of my old customers made digital phone switches and they had to provide that SLA to their phone company customers, or else not get the order.
- deleted 3y ago
  [deleted]
- _trampeltier 3y ago
  That was before we had to update every system every day :-)
- biofunsf 3y ago
  I'm having trouble finding any reference for the "five minutes per decade" downtime limit. That would be "6 9s" or 99.9999% uptime which is just crazy in today's world. Even today VOIP providers only claim 99.999%, though it seems like in practice many fail to even get close too that. I think I found the 756 page consent decree between Bell and the US you mentioned but I can't find a reference there either, though it is quite a massive doc with so-so OCR: https://www.google.com/books/edition/Consent_Decree_Program_of_the_Department/ovboRuQHqhAC?hl=en&bshm=ncc/1,ncc/1&gbpv=0 https://www.google.com/books/edition/Consent_Decree_Program_... Stepping back a bit, it seems like the FCC would be in charge of establishing rules like this but that consent decree I found (which might be the wrong one), is with the house Antitrust committee. (After my own Googling failed I also asked GPT-4 and it isn't aware of this either)
  - sosodev 3y ago
    I think they got 5 nines mixed up with 6 nines. The 5 nines is 5 minutes per year. I asked ChatGPT and it claims they had a target of 5 nines which makes sense.
colesantiago 3y ago
Oof. You're not wrong, this is really really bad, especially for Dropbox, storage is their business so I expected way better. These stats are no different to S3 at all. All of this engineering and moving away from AWS and for so few gain in availability. I was initially excited when they moved away from AWS and expected industry leading higher availability when they moved away, but I was wrong. This is disappointing for Dropbox which this is their main business, storing files without any minor hiccups or outages for years.
- awill 3y ago
  this is an overreaction. Dropbox is mostly storing files that are already stored on the customer's device, so customers usually won't notice an outage.
  - colesantiago 3y ago
    How is this an overreaction? I would expect Dropbox, a file storage company that proudly invests heavily in tech and infrastructure to achieve a better availability than what they already were on AWS (99.99%) In terms of availability the change is pretty much 0 and as a business / enterprise customer I might as well choose a different service with similar or higher 9s or (if my needs are complex) choose S3.
    - nmjohn 3y ago
      Can you provide an example of an alternative service which will give higher than 4 nines for availability that an enterprise customer would pick instead if that < 1hr of downtime per year was too high?
      - colesantiago 3y ago
        AWS (Architected Correctly) which I am sure Dropbox has experience in. https://aws.amazon.com/blogs/publicsector/achieving-five-nines-cloud-justice-public-safety/ https://aws.amazon.com/blogs/publicsector/achieving-five-nin... Here is a service that has managed to achieve 5x9s of availability: https://ably.com/ https://ably.com/

> 99.99% availability That seems... really bad, for the Core Core Service that Everything Depends On.

by dymk 3y ago

> 99.99% availability That seems... really bad, for the Core Core Service that Everything Depends On.

S3 standard storage class also says it's 99.99% availability so ... I wonder how hard it is to get beyond that? https://aws.amazon.com/s3/faqs/ https://aws.amazon.com/s3/faqs/

sosodev 3y ago
It's extremely hard to get availability higher than that. Possible but not something companies are willing to promise in an SLA.
- awill 3y ago
  I'd say it's not so much that it's really hard, it's that the compromises aren't worth it. note, I worked on S3 2015-2017
  - abeppu 3y ago
    So ... e.g. for 5 nines, the erasure code configuration would demand that do more writes in more locations and the cost impact is too high?
    - jl6 3y ago
      I imagine that as you add more nines, you start to hit more problems that are more out of your control. Like, how many nines do your backup diesel generators have?
    - skrtskrt 3y ago
      or replication if you're not erasure coding. So yeah you need more storage-optimized server racks and all the associated manpower and maintenance, you also need them to be distributed across different datacenters and zones which of course also impacts latency and your ability to provide some appearance of consistency, then you also need the same distribution for stateless services serving the data. On and on and on and you might be nearly doubling cost to get to an extra 9 that almost all of your customers won't care about.
  - sosodev 3y ago
    That's a fair point. It would be quite possible if the customers were willing to pay a lot more. I worked on DigitalOcean's object storage for about a year not long ago. Makes sense that those of us who have been in this space would be interested in this article haha.
foobiekr 3y ago
Absent here is any statement about how available is defined in terms of being _meaningfully up_ like iops or bps.
CobrastanJorji 3y ago
It's a cost problem. You can absolutely get higher availability, but it requires more hardware, uses more networking, and it makes maintenance and feature development more expensive. Blob storage customers are very price sensitive, but they also want lots of availability. That's why you'll see various "tiers" of storage, to try and capture all the combinations of availability and storage cost that various kinds of customers might want. Pretty much nobody wants to pay twice the cost to turn an exabyte of 99.99% available data to 99.995% available data, so it's not a product.

petters 3y ago

Not really bad. It’s less than an hour per year. Most users would not notice.

sosodev 3y ago

How is that bad? 99.99% means the system is down for less than an hour each year. https://uptime.is/99.99 https://uptime.is/99.99

foobiekr 3y ago
of course, it actually depends on how they define "up."

rbanffy 3y ago

When you build anything on it, you know it may not be available for a couple hours a year and architect around it. 99.99 isn't bad at all. Your ISP probably has worse SLAs.

gumby 3y ago

By comparison the old US POTS phone system was required to maintain downtime of less than five minutes per decade. I have no idea what rules, if any, apply to the modern wireless systems. Note: The consent decree, the actual legal obligation of the Bell system, actually specified the percentage of times someone would pick up the phone and not get a dial tone or operator. It was surprisingly high -- IIRC something like 2%, which is why you can see people toggling the hook in old movies. The five minutes per decade constraint I know because some of my old customers made digital phone switches and they had to provide that SLA to their phone company customers, or else not get the order.

deleted 3y ago
[deleted]
_trampeltier 3y ago
That was before we had to update every system every day :-)
biofunsf 3y ago
I'm having trouble finding any reference for the "five minutes per decade" downtime limit. That would be "6 9s" or 99.9999% uptime which is just crazy in today's world. Even today VOIP providers only claim 99.999%, though it seems like in practice many fail to even get close too that. I think I found the 756 page consent decree between Bell and the US you mentioned but I can't find a reference there either, though it is quite a massive doc with so-so OCR: https://www.google.com/books/edition/Consent_Decree_Program_of_the_Department/ovboRuQHqhAC?hl=en&bshm=ncc/1,ncc/1&gbpv=0 https://www.google.com/books/edition/Consent_Decree_Program_... Stepping back a bit, it seems like the FCC would be in charge of establishing rules like this but that consent decree I found (which might be the wrong one), is with the house Antitrust committee. (After my own Googling failed I also asked GPT-4 and it isn't aware of this either)
- sosodev 3y ago
  I think they got 5 nines mixed up with 6 nines. The 5 nines is 5 minutes per year. I asked ChatGPT and it claims they had a target of 5 nines which makes sense.

colesantiago 3y ago

Oof. You're not wrong, this is really really bad, especially for Dropbox, storage is their business so I expected way better. These stats are no different to S3 at all. All of this engineering and moving away from AWS and for so few gain in availability. I was initially excited when they moved away from AWS and expected industry leading higher availability when they moved away, but I was wrong. This is disappointing for Dropbox which this is their main business, storing files without any minor hiccups or outages for years.

awill 3y ago
this is an overreaction. Dropbox is mostly storing files that are already stored on the customer's device, so customers usually won't notice an outage.
- colesantiago 3y ago
  How is this an overreaction? I would expect Dropbox, a file storage company that proudly invests heavily in tech and infrastructure to achieve a better availability than what they already were on AWS (99.99%) In terms of availability the change is pretty much 0 and as a business / enterprise customer I might as well choose a different service with similar or higher 9s or (if my needs are complex) choose S3.
  - nmjohn 3y ago
    Can you provide an example of an alternative service which will give higher than 4 nines for availability that an enterprise customer would pick instead if that < 1hr of downtime per year was too high?
    - colesantiago 3y ago
      AWS (Architected Correctly) which I am sure Dropbox has experience in. https://aws.amazon.com/blogs/publicsector/achieving-five-nines-cloud-justice-public-safety/ https://aws.amazon.com/blogs/publicsector/achieving-five-nin... Here is a service that has managed to achieve 5x9s of availability: https://ably.com/ https://ably.com/