3 ms·

I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue th

by harikb 6y ago

I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue that originally started because of an external issue like they mention here. In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company

txcwpalpha 6y ago
>I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue that originally started because of an external issue like they mention here. Even in an "emergency" situation like described in the post-mortem (which I'm actually confused about, because Prince's tweet said it was actually "routine maintenance", which makes it even less justifiable), there should be a standard playbook with pre-tested and pre-approved failover options to enact. There definitely should not be someone just manually fingering in a config file on the fly. >In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing This is fine for some startup still hacking their way through an MVP, but for a global, multi-billion dollar company that prides themselves on supporting a significant part of the global internet, "the operator better know what they are doing" is unacceptable and not good enough. There should be multiple layers of protection against this stuff, so that even if an operator is having a nutty, they don't break half the internet. >By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company I can't agree at all. We're talking about some basic change management and testing stuff here. I've been at plenty of organizations (even ones much smaller and with less resources than CF) that have gotten this right, and they certainly are not dinosaurs nor have they been "disrupted". Testing and change management is as fundamental as encryption or password hashing. If you don't have it, it's not because it's hard, it's just because you haven't bothered to implement it.
- harikb 6y ago
  For the record, I don’t work for cloudflare. I was merely pointing out that failsafe and bulletproof tools for every situation is a myth in my humble opinion. If you have seen and work for organizations that know how to do this better, good for you.
  - txcwpalpha 6y ago
    >If you have seen and work for organizations that know how to do this better, good for you. Let's be clear: I'm not talking about some small subset of fantasy companies. It is a strict compliance requirement for many of the world's largest industries that you must have things like change management processes, incident response playbooks, and testing in place. A "myth" it certainly is not. Again, it's a fundamental thing no different than the most basic security requirements that we also expect (which is no coincidence, as one of the components of many security frameworks is proper change management controls). I'm not saying it has to be bulletproof and perfect (in fact, it never will be), but you still have to have something. Put another, possible more easily understood way: "don't test in production" is a requirement (some times even a legal requirement) for most companies once they reach a certain size. It's not just a funny saying that devs joke about.
    - closeparen 6y ago
      Compliance can make you do something that looks like code review, something that looks like testing, something that looks like runbooks, etc. That can't will into existence engineers who are actually thoughtful, creative, and skilled at those things. They can't make the test suite actually anticipate the bugs that will be written. They can't make the runbook have correct steps with no unwanted side effects for every failure mode. They can't make the reviewer see what the author doesn't. Two, maybe three people in my 100-person org are legitimately good at this stuff. No amount of grandstanding about what a serious business we are changes that. In fact the people most enthusiastic about compliance and process usually end up undermining the real, substantive technical quality that the compliance process was aiming at.
ATsch 6y ago
Indeed, mitugating DDOS effectively requires frequently making drastic changes to your network on short order.
- jonny_eh 6y ago
  And have both the primary and secondary DNS go down?

by harikb 6y ago

txcwpalpha 6y ago

>I would assume CF operations folks would be constantly making changes that affect “production” many times a day. Some of it will be to be counteract an issue that originally started because of an external issue like they mention here. Even in an "emergency" situation like described in the post-mortem (which I'm actually confused about, because Prince's tweet said it was actually "routine maintenance", which makes it even less justifiable), there should be a standard playbook with pre-tested and pre-approved failover options to enact. There definitely should not be someone just manually fingering in a config file on the fly. >In an ideal world, yes there would be higher level tools to do everything and checks and balances for everything, but I assume half the tools are like our standard Unix cli tools - all too powerful and the operator better know what they are doing This is fine for some startup still hacking their way through an MVP, but for a global, multi-billion dollar company that prides themselves on supporting a significant part of the global internet, "the operator better know what they are doing" is unacceptable and not good enough. There should be multiple layers of protection against this stuff, so that even if an operator is having a nutty, they don't break half the internet. >By the time they have all the right tools for everything, the company would have become a dinosaur disrupted by some other technology or company I can't agree at all. We're talking about some basic change management and testing stuff here. I've been at plenty of organizations (even ones much smaller and with less resources than CF) that have gotten this right, and they certainly are not dinosaurs nor have they been "disrupted". Testing and change management is as fundamental as encryption or password hashing. If you don't have it, it's not because it's hard, it's just because you haven't bothered to implement it.

harikb 6y ago
For the record, I don’t work for cloudflare. I was merely pointing out that failsafe and bulletproof tools for every situation is a myth in my humble opinion. If you have seen and work for organizations that know how to do this better, good for you.
- txcwpalpha 6y ago
  >If you have seen and work for organizations that know how to do this better, good for you. Let's be clear: I'm not talking about some small subset of fantasy companies. It is a strict compliance requirement for many of the world's largest industries that you must have things like change management processes, incident response playbooks, and testing in place. A "myth" it certainly is not. Again, it's a fundamental thing no different than the most basic security requirements that we also expect (which is no coincidence, as one of the components of many security frameworks is proper change management controls). I'm not saying it has to be bulletproof and perfect (in fact, it never will be), but you still have to have something. Put another, possible more easily understood way: "don't test in production" is a requirement (some times even a legal requirement) for most companies once they reach a certain size. It's not just a funny saying that devs joke about.
  - closeparen 6y ago
    Compliance can make you do something that looks like code review, something that looks like testing, something that looks like runbooks, etc. That can't will into existence engineers who are actually thoughtful, creative, and skilled at those things. They can't make the test suite actually anticipate the bugs that will be written. They can't make the runbook have correct steps with no unwanted side effects for every failure mode. They can't make the reviewer see what the author doesn't. Two, maybe three people in my 100-person org are legitimately good at this stuff. No amount of grandstanding about what a serious business we are changes that. In fact the people most enthusiastic about compliance and process usually end up undermining the real, substantive technical quality that the compliance process was aiming at.

ATsch 6y ago

Indeed, mitugating DDOS effectively requires frequently making drastic changes to your network on short order.

jonny_eh 6y ago
And have both the primary and secondary DNS go down?