4 ms·

Five nines is impossible Really. It’s just not going to happen. Actually, it is going to happen. One individual part may fail more than 99.999% of the time, b

by davidu 17y ago

Five nines is impossible Really. It’s just not going to happen. Actually, it is going to happen. One individual part may fail more than 99.999% of the time, but overall system integrity can certainly be designed with greater than 99.999% uptime. Just get a second datacenter, get a second transit provider. Just as folks scale horizontally, you can built out reliability into the many many 9's such that when one component fails, the overall system integrity isn't impacted. It's not easy, and not always cheap, but it's quite doable.

mmt 17y ago
Perhaps "impossible" is overly dramatic, but "economically impractical" may, in this context (general purpose networked computing), be the same thing. It seems to me that "five nines" is achieved more often with careful use of definitions, rather than careful engineering. Low-hanging fruit are scheduled maintenance and degraded performance. Otherwise, wouldn't Tandem have been wildly successful?
bestes 17y ago
Complexity costs. It costs money and it costs downtime. Look at Google: they have redundant everything, super-smart people, super good operations and still have some downtime.
jacobian 17y ago
How long does it take to fail over to a second data center? Can you do it in less than 26 seconds? I can't.
- smhinsey 17y ago
  According to Amazon, EC2 can remove a server from an Elastic Load Balancer rotation at a polling interval of 5 seconds. You can pool servers in different availability zones into the same ELB. I'm not sure the question even makes sense in that context, which I sort of view as the logical extreme of what davidu wrote.
  - ensignavenger 17y ago
    Pardon my ignorance, but what happens when the datacenter hosting the load balancer goes down? Are the load balncers redundant across data centers?
    - smhinsey 17y ago
      In short, yes. There's a good description of it here. http://clouddevelopertips.blogspot.com/2009/07/elastic-in-elastic-load-balancing-elb.html http://clouddevelopertips.blogspot.com/2009/07/elastic-in-el...
- lsc 17y ago
  I've worked places that had that setup with BGP. Of course, you need to announce the same IP block out of both data centers, your application needs to be able to deal with that, and that sort of thing gets pricy, but it's doable.
InclinedPlane 17y ago
5 nines is completely unrealistic for the internet. The internet backbone doesn't even have 5 nines. Nobody has 5 nines, not even google. 5 nines is 5 minutes of downtime per year, or an hour per decade. If you have an hour of downtime, you need to have 100% uptime for a decade to achieve 5 nines. The only thing that has 5 nines uptime is the land based telephony network in the US, and only after many decades of effort struggling to get there. Think about the massive amount of redundancy, extra capacity, and backup systems the phone network has, much of it mandated by law. Not merely battery backups everywhere, but backup generators too. Think about the implications of building a system for actual 5 nines reliability. With today's technology it is very, very costly. If google built their systems the same way the telephone network does they would be operating at a huge loss instead of a hefty profit. Over-engineering is a waste of money. For the vast majority of cases, 5 nines reliability on the web is excessive and unrealistic.

by davidu 17y ago

mmt 17y ago

Perhaps "impossible" is overly dramatic, but "economically impractical" may, in this context (general purpose networked computing), be the same thing. It seems to me that "five nines" is achieved more often with careful use of definitions, rather than careful engineering. Low-hanging fruit are scheduled maintenance and degraded performance. Otherwise, wouldn't Tandem have been wildly successful?

bestes 17y ago

Complexity costs. It costs money and it costs downtime. Look at Google: they have redundant everything, super-smart people, super good operations and still have some downtime.

jacobian 17y ago

How long does it take to fail over to a second data center? Can you do it in less than 26 seconds? I can't.

smhinsey 17y ago
According to Amazon, EC2 can remove a server from an Elastic Load Balancer rotation at a polling interval of 5 seconds. You can pool servers in different availability zones into the same ELB. I'm not sure the question even makes sense in that context, which I sort of view as the logical extreme of what davidu wrote.
- ensignavenger 17y ago
  Pardon my ignorance, but what happens when the datacenter hosting the load balancer goes down? Are the load balncers redundant across data centers?
  - smhinsey 17y ago
    In short, yes. There's a good description of it here. http://clouddevelopertips.blogspot.com/2009/07/elastic-in-elastic-load-balancing-elb.html http://clouddevelopertips.blogspot.com/2009/07/elastic-in-el...
lsc 17y ago
I've worked places that had that setup with BGP. Of course, you need to announce the same IP block out of both data centers, your application needs to be able to deal with that, and that sort of thing gets pricy, but it's doable.

InclinedPlane 17y ago

5 nines is completely unrealistic for the internet. The internet backbone doesn't even have 5 nines. Nobody has 5 nines, not even google. 5 nines is 5 minutes of downtime per year, or an hour per decade. If you have an hour of downtime, you need to have 100% uptime for a decade to achieve 5 nines. The only thing that has 5 nines uptime is the land based telephony network in the US, and only after many decades of effort struggling to get there. Think about the massive amount of redundancy, extra capacity, and backup systems the phone network has, much of it mandated by law. Not merely battery backups everywhere, but backup generators too. Think about the implications of building a system for actual 5 nines reliability. With today's technology it is very, very costly. If google built their systems the same way the telephone network does they would be operating at a huge loss instead of a hefty profit. Over-engineering is a waste of money. For the vast majority of cases, 5 nines reliability on the web is excessive and unrealistic.