5 ms·

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," acco

Seems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly flipping between them on a regular basis, at least for a short time, so the 'passive' part of the active-passive pair gets some work, is worth considering.

dylan604 3y ago
"No recovery plan is worth anything if it is not tested regularly." "An untested recovery plan is worth less than the paper on which it is printed." I'm sure there are many other versions of the quote
- pphysch 3y ago
  But think of all the boxes you cannot check if you are honest about your recovery plan
- user3939382 3y ago
  If you haven’t tested your backup you don’t have one.
- hinkley 3y ago
  Have you seen this video of the guy who on some sort of “adventure” outing? They are way up in the air on some sort of platform where you are supposed to jump across these planks from one platform to another. Someone is filming from either a platform or plateau next to the rig. There’s a harness connected to a rail above you to catch you if you fall. In the video the man gets halfway across and the cable attached to his harness just pops off. The instructor behind him instinctively reaches out as if he can use The Force to catch him if he biffs, but he makes it to the other side unharmed, and is all smiles until the cable catches up and bops him in the back. At which point he turns around and connects the dots. It’s a bit like that. And this is the image that comes into my mind every time I discover we have untested recovery processes while we are in the middle of a recovery.
- bsder 3y ago
  "Backup--very, very boring. Restore--very, very exciting."
f001 3y ago
I’m surprised at this to be honest. At my employer (a bank) we have to do a flip a couple times a year and just continue running from the alternate DC until the next flip. Pain in the butt on the weekends of the flip but at least we know for sure that our DR plan works and is good enough to serve our regular load…
paulddraper 3y ago
Probably observer's bias, but I've never seen a recovery plan work like it's supposed to.
- BirAdam 3y ago
  I’ve worked a few places where the company did regular disaster recovery testing to ensure stuff worked how it was intended to. Most of the time, it worked. Every once in a while, it failed.
  - paulddraper 3y ago
    Yeah the "testing" worked....but did it really? Maybe I'm just too skeptical.
- tgsovlerkhgsel 3y ago
  Seeing this play out in IT makes me really worried when it comes to critical infrastructure. On paper, there are plans and redundancies upon redundancies, and everything is perfectly safe. In practice, I haven't seen them exercised, and some of these plans sound incredibly optimistic...
  - nly 3y ago
    Humans are pretty good at scrambling around to hack something together when shit really hits the fan.
    - tgsovlerkhgsel 3y ago
      That scrambling often comes too late to avert serious damage or deaths though.
HideousKojima 3y ago
Or you could do chaos engineering: https://en.m.wikipedia.org/wiki/Chaos_engineering https://en.m.wikipedia.org/wiki/Chaos_engineering
- stuff4ben 3y ago
  I would venture to say that most non-tech startup companies are not equipped with the knowledge, the hardware, or the political fortitude to do something like that. Especially when it's customer's money at stake. It's one thing if Joe Schmo can't watch the latest Great British Baking Show. It's another matter entirely if Joe's paycheck is missing.
SandraBucky 3y ago
Bureaucracy triumphs over engineering considerations most of the time. Engineering division at banks might have suggested testing backup but top execs might disagree as it is unnecessary risk on operational system. Of-course they will play the blame game afterward and some low-level contractor will receive the heat cz of inability of decision making at top. Disclaimer: I worked as ELV contractor for an asset management megacorp.
stuff4ben 3y ago
We used to do this for some large Artifactory clusters (~1bn artifacts) we ran on both the east and west coast of the US. Failing over to our DR cluster and our (internal) global DNS load balancer handled it just fine. Made upgrading those clusters seamless for the most part. Gave me confidence we could continue development in the event our main site fell into the ocean for some reason.
oooyay 3y ago
Kind of a spicy opinion as a SRE, but if your disaster recovery strategy isn't part of your operational strategy then it's not actually a strategy. It's a moonshot in the face of failure.
- kevincox 3y ago
  That's a strategy. And depending on your downtime tolerance it may be an acceptable strategy. In this case it probably wasn't an appropriate strategy.
  - hinkley 3y ago
    Sticking your face in front of a pit bull you’ve never met before is “a strategy”. If you’re going to use the word strategy this way it has no meaning, which kills the conversation and communication in general. I think you’re confusing strategy and behavior. Strategy is a plan to do better than default behavior.
    - kevincox 3y ago
      Sticking your face in front of a pit bull is a strategy for what? Writing up a plan for disaster recovery is a strategy for disaster recovery. The pros are that it is cheap and ticks the regulatory checkboxes. The downside is that it has a high probability of a high recovery time in the case of a disaster. My point is that there are different pros and cons to every strategy that you can pick. I also don't think that all strategies are better than "default behaviour" whatever that is.
      - hinkley 3y ago
        > Sticking your face in front of a pit bull is a strategy for what? For meeting dogs of course. The problem I have is that it seems like you agree with other people in this thread, but real world experience tells me that stating things this way is more fuel for the lazy and for anarchists than it is support for the SREs. Don’t let the idiots think for a moment they are entitled to use a word as heavy as strategy. It ends farcically. Writing it down doesn’t make it a strategy. Satire is also written down.
bell-cot 3y ago
> Perhaps regularly flipping between them on a regular basis [...] is worth considering. True. But getting the PHB's at a bank (and perhaps the banking regulators) to sign off on doing that...could prove to be a non-trivial task.
TheIronMark 3y ago
It's odd that regular testing wasn't part of a compliance framework for them. BC/DR testing has been part of every security/compliance framework I've worked with.
captainkrtek 3y ago
Something something, the spare tire was flat.

dylan604 3y ago

"No recovery plan is worth anything if it is not tested regularly." "An untested recovery plan is worth less than the paper on which it is printed." I'm sure there are many other versions of the quote

pphysch 3y ago
But think of all the boxes you cannot check if you are honest about your recovery plan
user3939382 3y ago
If you haven’t tested your backup you don’t have one.
hinkley 3y ago
Have you seen this video of the guy who on some sort of “adventure” outing? They are way up in the air on some sort of platform where you are supposed to jump across these planks from one platform to another. Someone is filming from either a platform or plateau next to the rig. There’s a harness connected to a rail above you to catch you if you fall. In the video the man gets halfway across and the cable attached to his harness just pops off. The instructor behind him instinctively reaches out as if he can use The Force to catch him if he biffs, but he makes it to the other side unharmed, and is all smiles until the cable catches up and bops him in the back. At which point he turns around and connects the dots. It’s a bit like that. And this is the image that comes into my mind every time I discover we have untested recovery processes while we are in the middle of a recovery.
bsder 3y ago
"Backup--very, very boring. Restore--very, very exciting."

f001 3y ago

I’m surprised at this to be honest. At my employer (a bank) we have to do a flip a couple times a year and just continue running from the alternate DC until the next flip. Pain in the butt on the weekends of the flip but at least we know for sure that our DR plan works and is good enough to serve our regular load…

paulddraper 3y ago

Probably observer's bias, but I've never seen a recovery plan work like it's supposed to.

BirAdam 3y ago
I’ve worked a few places where the company did regular disaster recovery testing to ensure stuff worked how it was intended to. Most of the time, it worked. Every once in a while, it failed.
- paulddraper 3y ago
  Yeah the "testing" worked....but did it really? Maybe I'm just too skeptical.
tgsovlerkhgsel 3y ago
Seeing this play out in IT makes me really worried when it comes to critical infrastructure. On paper, there are plans and redundancies upon redundancies, and everything is perfectly safe. In practice, I haven't seen them exercised, and some of these plans sound incredibly optimistic...
- nly 3y ago
  Humans are pretty good at scrambling around to hack something together when shit really hits the fan.
  - tgsovlerkhgsel 3y ago
    That scrambling often comes too late to avert serious damage or deaths though.

HideousKojima 3y ago

Or you could do chaos engineering: https://en.m.wikipedia.org/wiki/Chaos_engineering https://en.m.wikipedia.org/wiki/Chaos_engineering

stuff4ben 3y ago
I would venture to say that most non-tech startup companies are not equipped with the knowledge, the hardware, or the political fortitude to do something like that. Especially when it's customer's money at stake. It's one thing if Joe Schmo can't watch the latest Great British Baking Show. It's another matter entirely if Joe's paycheck is missing.

SandraBucky 3y ago

Bureaucracy triumphs over engineering considerations most of the time. Engineering division at banks might have suggested testing backup but top execs might disagree as it is unnecessary risk on operational system. Of-course they will play the blame game afterward and some low-level contractor will receive the heat cz of inability of decision making at top. Disclaimer: I worked as ELV contractor for an asset management megacorp.

stuff4ben 3y ago

We used to do this for some large Artifactory clusters (~1bn artifacts) we ran on both the east and west coast of the US. Failing over to our DR cluster and our (internal) global DNS load balancer handled it just fine. Made upgrading those clusters seamless for the most part. Gave me confidence we could continue development in the event our main site fell into the ocean for some reason.

oooyay 3y ago

Kind of a spicy opinion as a SRE, but if your disaster recovery strategy isn't part of your operational strategy then it's not actually a strategy. It's a moonshot in the face of failure.

kevincox 3y ago
That's a strategy. And depending on your downtime tolerance it may be an acceptable strategy. In this case it probably wasn't an appropriate strategy.
- hinkley 3y ago
  Sticking your face in front of a pit bull you’ve never met before is “a strategy”. If you’re going to use the word strategy this way it has no meaning, which kills the conversation and communication in general. I think you’re confusing strategy and behavior. Strategy is a plan to do better than default behavior.
  - kevincox 3y ago
    Sticking your face in front of a pit bull is a strategy for what? Writing up a plan for disaster recovery is a strategy for disaster recovery. The pros are that it is cheap and ticks the regulatory checkboxes. The downside is that it has a high probability of a high recovery time in the case of a disaster. My point is that there are different pros and cons to every strategy that you can pick. I also don't think that all strategies are better than "default behaviour" whatever that is.
    - hinkley 3y ago
      > Sticking your face in front of a pit bull is a strategy for what? For meeting dogs of course. The problem I have is that it seems like you agree with other people in this thread, but real world experience tells me that stating things this way is more fuel for the lazy and for anarchists than it is support for the SREs. Don’t let the idiots think for a moment they are entitled to use a word as heavy as strategy. It ends farcically. Writing it down doesn’t make it a strategy. Satire is also written down.

bell-cot 3y ago

> Perhaps regularly flipping between them on a regular basis [...] is worth considering. True. But getting the PHB's at a bank (and perhaps the banking regulators) to sign off on doing that...could prove to be a non-trivial task.

TheIronMark 3y ago

It's odd that regular testing wasn't part of a compliance framework for them. BC/DR testing has been part of every security/compliance framework I've worked with.

captainkrtek 3y ago

Something something, the spare tire was flat.