8 ms·

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more tha

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to commodity hard-drive failure rates of 4%. Hence, if you wanted a higher level of data availability you'd use more than one EBS volume in different regions. These outages are affecting North America, and not Europe and Asia Pacific. That's it. Why is this even news? Were you expecting 100% availability?

stef25 16y ago
Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.
- JonnieCache 16y ago
  0.05% of one year is 4 hours, 22 minutes and 48 seconds.
  - bad_user 16y ago
    Which means their entire quota for this year is all gone.
    - yuvadam 16y ago
      and then some...
  - stef25 16y ago
    Don't forget to take in to account ALL of Reddit's downtime; there is quite a bit of it.
  - cmurdock 16y ago
    Reddit experienced some issues with Amazon a month ago that resulted in the site being down for almost a day. I'm pretty sure they're way below that percentage.
    - deleted 16y ago
      [deleted]
    - forgotusername 16y ago
      That's conflating SLAs again, Reddit's long-running problems have been with EBS reliability.
      - cube13 16y ago
        Even so, .5% downtime per year is about 44 hours, and Reddit's definitely had more downtime in the last few months than that. Of course, that's assuming that 100% of Reddit's problems were due to EBS only and not a combination of EBS, EC2 and their own code.
  - ssmoot 16y ago
    Is this really standard practice for measuring the SLA? The contracts I've seen for a couple small businesses are generally per billable period. Which always made sense to me. I pay you for 99.5% uptime this month. If you don't achieve it, then I get a discount, as simple as that. If your availability is below that, I don't pay full price for the billable period and then reconcile at the end of the year. Any links or general advice on this topic anyone has I'd be pretty interested on finding out if there's a general consensus of it being done differently?
- asymptotic 16y ago
  If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.
  - bad_user 16y ago
    The whole point of AWS is to forget about maintaining hardware infrastructure. Amazon are the ones who should have made backups in multiple regions, and transfer the load on failure.
    - edanm 16y ago
      "[S]hould" is the wrong word here. Clearly, they don't maintain such backups. This is clear to anyone using their service. They pay for the service anyway, so apparently it's still worth it to them, even without auto-backups. Would it make sense for Amazon to maintain automatic backups (and potentially charge more for them)? I don't know. It might make business sense, it might not. But their service is apparently popular enough even without it.
      - deleted 16y ago
        [deleted]
    - jasonkester 16y ago
      Actually, the whole point of AWS is to have options for using hardware that you don't own. They don't offer any magic "all your stuff in one package, guaranteed to work all the time" service. So yes, you do still need to think about your hardware infrastructure. You just don't have to own it. And Amazon does have all their stuff available in multiple regions. It's up to you to use it though.
    - ayb 16y ago
      Not sure if I could say Amazon should be doing that - but I'd love it if other value added providers (such as Heroku) could implement this.
    - 16y ago
- tedjdziuba 16y ago
  Reddit being down is not news.
  - davidreiss666 16y ago
    No. Not really. But we kind of miss it anyway.
  - jedberg 16y ago
    That hurts. But you're right, we've had a lot of issues. I think the reason this is news is because it is a massive Amazon failure.
    - Peroni 16y ago
      Reddit gets a lot of grief for stability issues but the fact is it is an immensely popular site that a huge number of people have a close affiliation to. A massive percentage of these people spend a significant portion of their day browsing Reddit, interacting with other redditors, etc. and for the site to be down for as long as it has is news, regardless of similar issues occuring in the past. The main reason this is news is because this is an Amazon issue but also because tens of thousands of people who frequent the site regularly are now aimlessly browsing the internet in the hopes of finding alternative lulz and in my case some of us are even getting work done. shudder
    - mattdeboard 16y ago
      I can't imagine how frustrating the jobs of the Reddit admins must be.
      - jedberg 16y ago
        It's usually very rewarding. The awesome community is what keeps me doing it.
        freakball 16y ago
        Awesome, I just figured out that you kept the votes tallied during the 'downtime'. What's interesting is how clearly good and bad submissions were dichotomized when nobody had anything else to vote on.
- blhack 16y ago
  http://cache-scale.appspot.com/c/www.reddit.com/ http://cache-scale.appspot.com/c/www.reddit.com/ That's a cache of it. I really wish that the admins at reddit would implement something like this themselves, then link to it when downtime like this happens.
  - wallnutboy 16y ago
    They do have a read-only mode, don't they? I'm not sure why they don't enable read-only mode when things like this happen. It may be that Amazon's service being down forbids this. I dunno.
    - groovylick 16y ago
      They have a read-only mode for "free" which is their akamai cache that the unlogged in users see.
  - evolution 16y ago
    do you have any stats of this app in terms of total data stored, bandwidth required per day, requests per second?
    - blhack 16y ago
      I don't, unfortunately, because I didn't write it :(
deleted 16y ago
[deleted]
hoop 16y ago
Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one region is independent of the state of the other regions. P(two regions fail) = P(one region fails and another region fails) = P(region fails) * P(region fails) = 0.05% * 0.05% = 0.0025% Making your availability = 100% - 0.0025% = 99.9975% Ultimately it's more of a business decision if you want to pay for the extra 0.0475% of availability. I would think (or hope) that most engineers would want it anyway. The numbers at this size appear insignificant. How would one (say an engineer) convince "the management" that the extra 0.0475% of availability is worth the investment/expense?
- gphil 16y ago
  > The numbers at this size appear insignificant. How would one (say an engineer) convince "the management" that the extra 0.0475% of availability is worth the investment/expense? In businesses where physical goods are sold to customers, "the management" is generally very motivated to avoid stock-out situations in which sales are lost due to lack of inventory (even if it's only a very small percentage.) The reason for this is because they are concerned about the potential loss of customer goodwill. It seems that the same applies in this situation.
- bartonfink 16y ago
  Why wouldn't a simple expected value calculation work? You've shown that you can calculate the extra availability that subscribing to another region provides. Simply multiply the cost of an outage by the extra availability provided by an additional region that would have prevented that outage. If expanding to another region costs more than just taking the outage, then it's categorically not a good option. If management still says no in the face of numbers that suggest yes, then that tells you that you're missing a hidden objection, and how you proceed will depend on a lot of factors specific to your situation.
  - 16y ago
bdonlan 16y ago
Note also that 0.1-0.5% refers to irrecoverable data loss, not temporary unavailability.

stef25 16y ago

Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.

JonnieCache 16y ago
0.05% of one year is 4 hours, 22 minutes and 48 seconds.
- bad_user 16y ago
  Which means their entire quota for this year is all gone.
  - yuvadam 16y ago
    and then some...
- stef25 16y ago
  Don't forget to take in to account ALL of Reddit's downtime; there is quite a bit of it.
- cmurdock 16y ago
  Reddit experienced some issues with Amazon a month ago that resulted in the site being down for almost a day. I'm pretty sure they're way below that percentage.
  - deleted 16y ago
    [deleted]
  - forgotusername 16y ago
    That's conflating SLAs again, Reddit's long-running problems have been with EBS reliability.
    - cube13 16y ago
      Even so, .5% downtime per year is about 44 hours, and Reddit's definitely had more downtime in the last few months than that. Of course, that's assuming that 100% of Reddit's problems were due to EBS only and not a combination of EBS, EC2 and their own code.
- ssmoot 16y ago
  Is this really standard practice for measuring the SLA? The contracts I've seen for a couple small businesses are generally per billable period. Which always made sense to me. I pay you for 99.5% uptime this month. If you don't achieve it, then I get a discount, as simple as that. If your availability is below that, I don't pay full price for the billable period and then reconcile at the end of the year. Any links or general advice on this topic anyone has I'd be pretty interested on finding out if there's a general consensus of it being done differently?
asymptotic 16y ago
If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.
- bad_user 16y ago
  The whole point of AWS is to forget about maintaining hardware infrastructure. Amazon are the ones who should have made backups in multiple regions, and transfer the load on failure.
  - edanm 16y ago
    "[S]hould" is the wrong word here. Clearly, they don't maintain such backups. This is clear to anyone using their service. They pay for the service anyway, so apparently it's still worth it to them, even without auto-backups. Would it make sense for Amazon to maintain automatic backups (and potentially charge more for them)? I don't know. It might make business sense, it might not. But their service is apparently popular enough even without it.
    - deleted 16y ago
      [deleted]
  - jasonkester 16y ago
    Actually, the whole point of AWS is to have options for using hardware that you don't own. They don't offer any magic "all your stuff in one package, guaranteed to work all the time" service. So yes, you do still need to think about your hardware infrastructure. You just don't have to own it. And Amazon does have all their stuff available in multiple regions. It's up to you to use it though.
  - ayb 16y ago
    Not sure if I could say Amazon should be doing that - but I'd love it if other value added providers (such as Heroku) could implement this.
  - 16y ago
tedjdziuba 16y ago
Reddit being down is not news.
- davidreiss666 16y ago
  No. Not really. But we kind of miss it anyway.
- jedberg 16y ago
  That hurts. But you're right, we've had a lot of issues. I think the reason this is news is because it is a massive Amazon failure.
  - Peroni 16y ago
    Reddit gets a lot of grief for stability issues but the fact is it is an immensely popular site that a huge number of people have a close affiliation to. A massive percentage of these people spend a significant portion of their day browsing Reddit, interacting with other redditors, etc. and for the site to be down for as long as it has is news, regardless of similar issues occuring in the past. The main reason this is news is because this is an Amazon issue but also because tens of thousands of people who frequent the site regularly are now aimlessly browsing the internet in the hopes of finding alternative lulz and in my case some of us are even getting work done. shudder
  - mattdeboard 16y ago
    I can't imagine how frustrating the jobs of the Reddit admins must be.
    - jedberg 16y ago
      It's usually very rewarding. The awesome community is what keeps me doing it.
      - freakball 16y ago
        Awesome, I just figured out that you kept the votes tallied during the 'downtime'. What's interesting is how clearly good and bad submissions were dichotomized when nobody had anything else to vote on.
blhack 16y ago
http://cache-scale.appspot.com/c/www.reddit.com/ http://cache-scale.appspot.com/c/www.reddit.com/ That's a cache of it. I really wish that the admins at reddit would implement something like this themselves, then link to it when downtime like this happens.
- wallnutboy 16y ago
  They do have a read-only mode, don't they? I'm not sure why they don't enable read-only mode when things like this happen. It may be that Amazon's service being down forbids this. I dunno.
  - groovylick 16y ago
    They have a read-only mode for "free" which is their akamai cache that the unlogged in users see.
- evolution 16y ago
  do you have any stats of this app in terms of total data stored, bandwidth required per day, requests per second?
  - blhack 16y ago
    I don't, unfortunately, because I didn't write it :(

deleted 16y ago

[deleted]

hoop 16y ago

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one region is independent of the state of the other regions. P(two regions fail) = P(one region fails and another region fails) = P(region fails) * P(region fails) = 0.05% * 0.05% = 0.0025% Making your availability = 100% - 0.0025% = 99.9975% Ultimately it's more of a business decision if you want to pay for the extra 0.0475% of availability. I would think (or hope) that most engineers would want it anyway. The numbers at this size appear insignificant. How would one (say an engineer) convince "the management" that the extra 0.0475% of availability is worth the investment/expense?

gphil 16y ago
> The numbers at this size appear insignificant. How would one (say an engineer) convince "the management" that the extra 0.0475% of availability is worth the investment/expense? In businesses where physical goods are sold to customers, "the management" is generally very motivated to avoid stock-out situations in which sales are lost due to lack of inventory (even if it's only a very small percentage.) The reason for this is because they are concerned about the potential loss of customer goodwill. It seems that the same applies in this situation.
bartonfink 16y ago
Why wouldn't a simple expected value calculation work? You've shown that you can calculate the extra availability that subscribing to another region provides. Simply multiply the cost of an outage by the extra availability provided by an additional region that would have prevented that outage. If expanding to another region costs more than just taking the outage, then it's categorically not a good option. If management still says no in the face of numbers that suggest yes, then that tells you that you're missing a hidden objection, and how you proceed will depend on a lot of factors specific to your situation.
- 16y ago

bdonlan 16y ago

Note also that 0.1-0.5% refers to irrecoverable data loss, not temporary unavailability.