3 ms·

For the record, I don’t work for cloudflare. I was merely pointing out that failsafe and bulletproof tools for every situation is a myth in my humble opinion. I

by harikb 6y ago

For the record, I don’t work for cloudflare. I was merely pointing out that failsafe and bulletproof tools for every situation is a myth in my humble opinion. If you have seen and work for organizations that know how to do this better, good for you.

txcwpalpha 6y ago
>If you have seen and work for organizations that know how to do this better, good for you. Let's be clear: I'm not talking about some small subset of fantasy companies. It is a strict compliance requirement for many of the world's largest industries that you must have things like change management processes, incident response playbooks, and testing in place. A "myth" it certainly is not. Again, it's a fundamental thing no different than the most basic security requirements that we also expect (which is no coincidence, as one of the components of many security frameworks is proper change management controls). I'm not saying it has to be bulletproof and perfect (in fact, it never will be), but you still have to have something. Put another, possible more easily understood way: "don't test in production" is a requirement (some times even a legal requirement) for most companies once they reach a certain size. It's not just a funny saying that devs joke about.
- closeparen 6y ago
  Compliance can make you do something that looks like code review, something that looks like testing, something that looks like runbooks, etc. That can't will into existence engineers who are actually thoughtful, creative, and skilled at those things. They can't make the test suite actually anticipate the bugs that will be written. They can't make the runbook have correct steps with no unwanted side effects for every failure mode. They can't make the reviewer see what the author doesn't. Two, maybe three people in my 100-person org are legitimately good at this stuff. No amount of grandstanding about what a serious business we are changes that. In fact the people most enthusiastic about compliance and process usually end up undermining the real, substantive technical quality that the compliance process was aiming at.
  - txcwpalpha 6y ago
    Absolutely. I'll be the first to say that compliance doesn't mean you're actually good at those things (and I actually have said as much here on HN). However, the point remains that things like change management aren't some nebulous, pie-in-the-sky concept. That they are in compliance frameworks speaks to the fact that they are very fundamental and basic (compliance frameworks are typically the bare minimum of things you should be doing). A half-assed CM process won't catch all errors by any means. But it will catch the most basic ones. And according to CF's post-mortem, the initial issue that kicked off the chain of events was "backbone congestion", which has got to be one of the most basic things that would be included in an IR playbook (Prince even referred to it as "routine maintenance" in an earlier message). And then the config change that was put in place also seems like a fairly basic change that probably would have been caught with with some basic testing. That's what catches my eye the most. This incident wasn't some wild, niche, couldn't-have-been-predicted event. It was "backbone congestion", followed by an attempted change of routes on the backbone to alleviate the congestion. For an internet services company to not have a standard, pre-approved and pre-tested solution for resolving something as predictable as backbone congestion is shocking to me.
    - manquer 6y ago
      You are right if the facts are exactly as presented and there is nothing more to know. I would say we should ready between the lines here. Most postmortems are marketing artefacts to some sense designed to reassure customers so taking them at face value is not really good idea, I have written and read enough to know that it is rarely ever the full story for variety of reasons. It is quite possible like you say they are playing fast and loose with config and they have systemic process risks as you point out. It is also possible that the issue is not just routine congestion i.e. it is symptom and not the root cause, and they are not talking about it in detail either to make the communication simple enough for non networking experts to understand, but has enough depth for technical folks to kind of follow along and reassure their managements that cloudflare has handle on it . It could also be revealing more may give their competitors some crucial information about their IP/architecture and they are calling it routine congestion to not reveal that. Lastly, all of the above are not mutually exclusive, it could be all of them in parts too.

by harikb 6y ago

txcwpalpha 6y ago

>If you have seen and work for organizations that know how to do this better, good for you. Let's be clear: I'm not talking about some small subset of fantasy companies. It is a strict compliance requirement for many of the world's largest industries that you must have things like change management processes, incident response playbooks, and testing in place. A "myth" it certainly is not. Again, it's a fundamental thing no different than the most basic security requirements that we also expect (which is no coincidence, as one of the components of many security frameworks is proper change management controls). I'm not saying it has to be bulletproof and perfect (in fact, it never will be), but you still have to have something. Put another, possible more easily understood way: "don't test in production" is a requirement (some times even a legal requirement) for most companies once they reach a certain size. It's not just a funny saying that devs joke about.

closeparen 6y ago
Compliance can make you do something that looks like code review, something that looks like testing, something that looks like runbooks, etc. That can't will into existence engineers who are actually thoughtful, creative, and skilled at those things. They can't make the test suite actually anticipate the bugs that will be written. They can't make the runbook have correct steps with no unwanted side effects for every failure mode. They can't make the reviewer see what the author doesn't. Two, maybe three people in my 100-person org are legitimately good at this stuff. No amount of grandstanding about what a serious business we are changes that. In fact the people most enthusiastic about compliance and process usually end up undermining the real, substantive technical quality that the compliance process was aiming at.
- txcwpalpha 6y ago
  Absolutely. I'll be the first to say that compliance doesn't mean you're actually good at those things (and I actually have said as much here on HN). However, the point remains that things like change management aren't some nebulous, pie-in-the-sky concept. That they are in compliance frameworks speaks to the fact that they are very fundamental and basic (compliance frameworks are typically the bare minimum of things you should be doing). A half-assed CM process won't catch all errors by any means. But it will catch the most basic ones. And according to CF's post-mortem, the initial issue that kicked off the chain of events was "backbone congestion", which has got to be one of the most basic things that would be included in an IR playbook (Prince even referred to it as "routine maintenance" in an earlier message). And then the config change that was put in place also seems like a fairly basic change that probably would have been caught with with some basic testing. That's what catches my eye the most. This incident wasn't some wild, niche, couldn't-have-been-predicted event. It was "backbone congestion", followed by an attempted change of routes on the backbone to alleviate the congestion. For an internet services company to not have a standard, pre-approved and pre-tested solution for resolving something as predictable as backbone congestion is shocking to me.
  - manquer 6y ago
    You are right if the facts are exactly as presented and there is nothing more to know. I would say we should ready between the lines here. Most postmortems are marketing artefacts to some sense designed to reassure customers so taking them at face value is not really good idea, I have written and read enough to know that it is rarely ever the full story for variety of reasons. It is quite possible like you say they are playing fast and loose with config and they have systemic process risks as you point out. It is also possible that the issue is not just routine congestion i.e. it is symptom and not the root cause, and they are not talking about it in detail either to make the communication simple enough for non networking experts to understand, but has enough depth for technical folks to kind of follow along and reassure their managements that cloudflare has handle on it . It could also be revealing more may give their competitors some crucial information about their IP/architecture and they are calling it routine congestion to not reveal that. Lastly, all of the above are not mutually exclusive, it could be all of them in parts too.