4 ms·

‘99.9999%’ uptime – it’s an illusion

22 points by vinnyglennon 9y ago

TaylorGood 9y ago
However, there feels like exceptions - I’m on my 10th year of using Dreamhost. Here was my tweet to them: “My 10th anniversary of using @DreamHost – hosting over 60 domains with zero downtime thus far. Can I just say it's one of the greatest tools in my life? Their site: bit.ly/2kl6Vr2“ Is it zero downtime? Probably not but my sites haven’t seen an outage that I recall. Amazing support too.
- slphil 9y ago
  I have had downtime with their email on many occasions but never their web hosting.
jakobegger 9y ago
Stories like this make me so glad that the software I sell runs locally, on the clients computer. For me there's never been a bug so bad it couldn't wait until monday morning.
krautt 9y ago
maybe if your 'app' is entropy in the universe.
SlowBro 9y ago
Three words: Disaster recovery site. That’s saved our butts so many times. We fail over then work on resolution and RCA. I am a systems admin at a large multi national company. Downtime can run in the billions per hour. I’ve only ever seen billion dollar downtime once in eight years, and it was for a few hours only. The DBAs delayed the failover decision, God knows why. They should have done that immediately. We have strict change controls and auditing so that the disaster recovery site always acts like it should, just like prod. Replication always occurs and when it fails it gets attention. Failover is regularly tested. Good governance makes all the difference. This article describes a failure of governance. It is a management issue. Edit: And governance is free. Replication is now within reach of even super tight budgets. See my reply below. So small companies can do this, which was my intended point.
- rando444 9y ago
  I'm curious what type of replication you use.
  - SlowBro 9y ago
    I’m not terribly privy to it because the DBAs manage it, but it’s some kind of Oracle replication like GoldenGate? But replication is everywhere now. MySQL, rsync, whatever. Plenty of choices.
    - rando444 9y ago
      But there being plenty of choices is exactly why I'm so curious as to what a large operation such as yours would choose :) I appreciate the feedback though!
      - SlowBro 9y ago
        I'll ask around. I had started to learn Oracle in my spare time but moved on to entrepreneurship and starting a business, electrical engineering, etc. At the office, I only know what I need to know :-/
        SlowBro 9y ago
        We use Oracle's GoldenGate[1] and Data Guard[2] technologies. [1] http://www.oracle.com/technetwork/middleware/goldengate/overview/index.html http://www.oracle.com/technetwork/middleware/goldengate/over... [2] https://docs.oracle.com/cd/B19306_01/server.102/b14239/concepts.htm#g1049956 https://docs.oracle.com/cd/B19306_01/server.102/b14239/conce...
- jpeg_hero 9y ago
  For those of us running smaller companies, it might not be a management issue, it’s a budget issue. The company probably doesn’t have the budget to support a $billion uptime organization with proper “governance” But I does sound like five 9’s is possible in your case. Edit: just saw author is claiming “six 9’s” is impossible ... maybe
airbreather 9y ago
What seems to have happened here is that the six nines up-time was calculated/predicted on the basis of no common cause failures, of which it appears this scenario depicts. Taking into account common cause failures (which are often as a result of root causes by human factors) your six nines can very easily be more like three or four nines. As a functional safety engineer that applies IEC61508/61511 for a living this is a common problem and addressed by diversity in systems, achieving the same goal in redundant systems by different hardware/software means. Airbus and Space Shuttle flight controls do a similar thing with multiple CPUs of different types with software implemented to the same spec by different teams, in some cases different languages. Not saying this is what they should be doing in their case as the mission is not so critical, but if you really want five or six nines this becomes extremely difficult with limited diversity in hardware and software.

22 points by vinnyglennon 9y ago

However, there feels like exceptions - I’m on my 10th year of using Dreamhost. Here was my tweet to them: “My 10th anniversary of using @DreamHost – hosting over 60 domains with zero downtime thus far. Can I just say it's one of the greatest tools in my life? Their site: bit.ly/2kl6Vr2“ Is it zero downtime? Probably not but my sites haven’t seen an outage that I recall. Amazing support too.

slphil 9y ago
I have had downtime with their email on many occasions but never their web hosting.

jakobegger 9y ago

Stories like this make me so glad that the software I sell runs locally, on the clients computer. For me there's never been a bug so bad it couldn't wait until monday morning.

krautt 9y ago

maybe if your 'app' is entropy in the universe.

SlowBro 9y ago

Three words: Disaster recovery site. That’s saved our butts so many times. We fail over then work on resolution and RCA. I am a systems admin at a large multi national company. Downtime can run in the billions per hour. I’ve only ever seen billion dollar downtime once in eight years, and it was for a few hours only. The DBAs delayed the failover decision, God knows why. They should have done that immediately. We have strict change controls and auditing so that the disaster recovery site always acts like it should, just like prod. Replication always occurs and when it fails it gets attention. Failover is regularly tested. Good governance makes all the difference. This article describes a failure of governance. It is a management issue. Edit: And governance is free. Replication is now within reach of even super tight budgets. See my reply below. So small companies can do this, which was my intended point.

rando444 9y ago
I'm curious what type of replication you use.
- SlowBro 9y ago
  I’m not terribly privy to it because the DBAs manage it, but it’s some kind of Oracle replication like GoldenGate? But replication is everywhere now. MySQL, rsync, whatever. Plenty of choices.
  - rando444 9y ago
    But there being plenty of choices is exactly why I'm so curious as to what a large operation such as yours would choose :) I appreciate the feedback though!
    - SlowBro 9y ago
      I'll ask around. I had started to learn Oracle in my spare time but moved on to entrepreneurship and starting a business, electrical engineering, etc. At the office, I only know what I need to know :-/
      - SlowBro 9y ago
        We use Oracle's GoldenGate[1] and Data Guard[2] technologies. [1] http://www.oracle.com/technetwork/middleware/goldengate/overview/index.html http://www.oracle.com/technetwork/middleware/goldengate/over... [2] https://docs.oracle.com/cd/B19306_01/server.102/b14239/concepts.htm#g1049956 https://docs.oracle.com/cd/B19306_01/server.102/b14239/conce...
jpeg_hero 9y ago
For those of us running smaller companies, it might not be a management issue, it’s a budget issue. The company probably doesn’t have the budget to support a $billion uptime organization with proper “governance” But I does sound like five 9’s is possible in your case. Edit: just saw author is claiming “six 9’s” is impossible ... maybe

airbreather 9y ago

What seems to have happened here is that the six nines up-time was calculated/predicted on the basis of no common cause failures, of which it appears this scenario depicts. Taking into account common cause failures (which are often as a result of root causes by human factors) your six nines can very easily be more like three or four nines. As a functional safety engineer that applies IEC61508/61511 for a living this is a common problem and addressed by diversity in systems, achieving the same goal in redundant systems by different hardware/software means. Airbus and Space Shuttle flight controls do a similar thing with multiple CPUs of different types with software implemented to the same spec by different teams, in some cases different languages. Not saying this is what they should be doing in their case as the mission is not so critical, but if you really want five or six nines this becomes extremely difficult with limited diversity in hardware and software.