4 ms·

Three words: Disaster recovery site. That’s saved our butts so many times. We fail over then work on resolution and RCA. I am a systems admin at a large multi

Three words: Disaster recovery site. That’s saved our butts so many times. We fail over then work on resolution and RCA. I am a systems admin at a large multi national company. Downtime can run in the billions per hour. I’ve only ever seen billion dollar downtime once in eight years, and it was for a few hours only. The DBAs delayed the failover decision, God knows why. They should have done that immediately. We have strict change controls and auditing so that the disaster recovery site always acts like it should, just like prod. Replication always occurs and when it fails it gets attention. Failover is regularly tested. Good governance makes all the difference. This article describes a failure of governance. It is a management issue. Edit: And governance is free. Replication is now within reach of even super tight budgets. See my reply below. So small companies can do this, which was my intended point.

rando444 9y ago
I'm curious what type of replication you use.
- SlowBro 9y ago
  I’m not terribly privy to it because the DBAs manage it, but it’s some kind of Oracle replication like GoldenGate? But replication is everywhere now. MySQL, rsync, whatever. Plenty of choices.
  - rando444 9y ago
    But there being plenty of choices is exactly why I'm so curious as to what a large operation such as yours would choose :) I appreciate the feedback though!
    - SlowBro 9y ago
      I'll ask around. I had started to learn Oracle in my spare time but moved on to entrepreneurship and starting a business, electrical engineering, etc. At the office, I only know what I need to know :-/
      - SlowBro 9y ago
        We use Oracle's GoldenGate[1] and Data Guard[2] technologies. [1] http://www.oracle.com/technetwork/middleware/goldengate/overview/index.html http://www.oracle.com/technetwork/middleware/goldengate/over... [2] https://docs.oracle.com/cd/B19306_01/server.102/b14239/concepts.htm#g1049956 https://docs.oracle.com/cd/B19306_01/server.102/b14239/conce...
jpeg_hero 9y ago
For those of us running smaller companies, it might not be a management issue, it’s a budget issue. The company probably doesn’t have the budget to support a $billion uptime organization with proper “governance” But I does sound like five 9’s is possible in your case. Edit: just saw author is claiming “six 9’s” is impossible ... maybe
- deleted 9y ago
  [deleted]
- SlowBro 9y ago
  I am also running a side business with a tight budget. Governance is free, and nowadays multiple sites are cheap. Create an identical setup, use replication, strong change controls, and test failover. Monitor replication failures. If you do it on an AWS instance it costs little until needed. I believe CPU can be scaled up or down as needed? Just pay for storage. Also use solid versioning, tested backups, have a tested backout strategy with a secondary remediation path for all changes. Paranoia is cheap :-) Someone might say, “It’s more work.” And paging engineers at 3am isn’t work? Would you rather burn out your engineers or have them relaxed and focused on responding to a replication issue at 2pm? :-) You know what they say about an ounce of prevention.
deleted 9y ago
[deleted]

rando444 9y ago

I'm curious what type of replication you use.

SlowBro 9y ago
I’m not terribly privy to it because the DBAs manage it, but it’s some kind of Oracle replication like GoldenGate? But replication is everywhere now. MySQL, rsync, whatever. Plenty of choices.
- rando444 9y ago
  But there being plenty of choices is exactly why I'm so curious as to what a large operation such as yours would choose :) I appreciate the feedback though!
  - SlowBro 9y ago
    I'll ask around. I had started to learn Oracle in my spare time but moved on to entrepreneurship and starting a business, electrical engineering, etc. At the office, I only know what I need to know :-/
    - SlowBro 9y ago
      We use Oracle's GoldenGate[1] and Data Guard[2] technologies. [1] http://www.oracle.com/technetwork/middleware/goldengate/overview/index.html http://www.oracle.com/technetwork/middleware/goldengate/over... [2] https://docs.oracle.com/cd/B19306_01/server.102/b14239/concepts.htm#g1049956 https://docs.oracle.com/cd/B19306_01/server.102/b14239/conce...

jpeg_hero 9y ago

For those of us running smaller companies, it might not be a management issue, it’s a budget issue. The company probably doesn’t have the budget to support a $billion uptime organization with proper “governance” But I does sound like five 9’s is possible in your case. Edit: just saw author is claiming “six 9’s” is impossible ... maybe

deleted 9y ago
[deleted]
SlowBro 9y ago
I am also running a side business with a tight budget. Governance is free, and nowadays multiple sites are cheap. Create an identical setup, use replication, strong change controls, and test failover. Monitor replication failures. If you do it on an AWS instance it costs little until needed. I believe CPU can be scaled up or down as needed? Just pay for storage. Also use solid versioning, tested backups, have a tested backout strategy with a secondary remediation path for all changes. Paranoia is cheap :-) Someone might say, “It’s more work.” And paging engineers at 3am isn’t work? Would you rather burn out your engineers or have them relaxed and focused on responding to a replication issue at 2pm? :-) You know what they say about an ounce of prevention.

deleted 9y ago

[deleted]