11 ms·

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF mana

In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out. I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!

thejosh 4y ago
Yep, look at heroku and their big incident, and the amount of downtime they've had lately.
viraptor 4y ago
I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture info that people will reference in discussions later. At a certain size it feels stupid not to post them publicly. I wonder how much those posts are calculated and how much organic/culture related.
- hunter2_ 4y ago
  > I wonder how much those posts are calculated and how much organic/culture related. Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
  - viraptor 4y ago
    No they don't have such duty. In practice very little decision making is based on hard data in my experience. Real world being fuzzy and risk being hard to quantify do not help the situation.
  - rrss 4y ago
    No. https://scholarship.law.cornell.edu/cgi/viewcontent.cgi?httpsredir=1&article=2311&context=facpub https://scholarship.law.cornell.edu/cgi/viewcontent.cgi?http...
- smugma 4y ago
  I agree that this is a free ad/recruitment. However, it’s easy to see how more conservative businesses see this as a risk. They are highlighting their deficiencies, letting their big important clients know that human error can bring their network down. Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warning sign to customers. It’s a strategy other companies could adopt, but to do it effectively requires changes all across the organization.
  - saghm 4y ago
    OTOH, I think most actual engineers would know that everywhere has deficiencies and can be brought down by human error, and I'd personally rather use a product where the people running it admit this rather than just claim that their genius engineers made it 100% foolproof and nothing could ever possibly go wrong
    - ethbr0 4y ago
      Absolutely. The first step of good SRE is admitting (publicly and within the organization) that you have a problem.
    - smugma 4y ago
      100% agree. But… 1. On the buy side, crappy big companies with procurement etc. may not have “actual engineers” deciding things. Maybe for something like Cloudflare that’s likely to sit within an actual technical team’s mandate 2. Not 100% foolproof but if a startup is selling its tool and has an uptime page and details of all its downtime and as a prospective customer you go and see that they have 3-4 hour downtimes once or twice a month, it should raise alarm bells.
- Uehreka 4y ago
  Ehhhh… I think it’s good (for us) that they do this, but I don’t think it’s a free ad (contrary to popular belief, not all news is good news, and this is bad news) and any sort of conversion rate on recruitment is probably vanishingly small (which would normally be fine, but incidents like these may turn off some actual customers, which is where actual revenue comes from). I think their calculation (to the extent you can call it that) is that in the interest of PR and damage control, it’s better to get a thorough postmortem out quickly to stem the bleeding and keep people like us from going “I can’t wait to hear what happened at Cloudflare” for a week. Now we know, the customers have an explanation, and this bad news cycle has a higher chance of ending quickly.
  - viraptor 4y ago
    > but incidents like these may turn off some actual customers Incidents - yes. But why would a post-mortem turn someone off? The incident happened regardless. Do you think anyone would be more likely turned off by reading how they solved it / plan to prevent it on the future than by silence?
- zaidf 4y ago
  >I feel like others lose opportunities by not doing the same IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.
  - ethbr0 4y ago
    I believe Marcus Aurelius had something similar to say on the matter. :-)
    - jjtheblunt 4y ago
      quodcumque erat ?
mattferderer 4y ago
To further add to your point, the CTO is the one who shared it here & the CEO is incredibly active on forums & social media everywhere with customers. Communication has always been one of their strengths.
- bluehatbrit 4y ago
  I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.
  - bombcar 4y ago
    It could be good or bad; I suspect they've thought about it and have worked on succession (I hope!) and have like-minded people in the wings. But once it happens things will change and, to be honest, likely for the worse. edit> fix typo
    - unityByFreedom 4y ago
      > secession Succession?
      - bombcar 4y ago
        Eep yes, auto spell check on macOS is usually good but sometimes it causes a civil war.
  - unityByFreedom 4y ago
    Good communicators generally foster that environment. And their customers appreciate it, so there is an external expectation now too. Everything ends some day, but I think this will be regarded as a valuable attribute for awhile.
    - jgrahamc 4y ago
kevin_nisbet 4y ago
I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.
- jgrahamc 4y ago
  What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.
- cowsandmilk 4y ago
  I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.
  - bombcar 4y ago
    It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.
  - 4y ago
tyingq 4y ago
>take weeks for other companies to publish a postmortem And with nowhere near the detail level of what was presented here. Typically lots of sweeping generalizations that don't tell you much about what happened, or give you any confidence they really know what happened or have the right fix in place.
Melatonic 4y ago
To be fair though they sort of MUST do things like this to have our confidence - their whole business is about being FAST and AVAILABLE. Were not talking about Oracle here :-D
agilob 4y ago
I'd love to see the postmortem from Facebook :(
kortilla 4y ago
Well cloudflare’s entire value is in uptime and preventing outages. Showing they have a rapid response and strong fundamental technical understanding is much more critical in the “prevent downtime” business.
solardev 4y ago
No provider is perfect, but it's because of stuff like this that I trust Cloudflare waaaaaaaaaaay more than the likes of Amazon. Transparency engenders trust, and eventually, love! Thank you, Cloudflare. The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your postmortem is excellent.) I couldn't even begin to think about designing or implementing something of this complexity, much less being able to explain it to a layperson after a failure. It is really impressive, and I hope you will continue to do so into the future! Most of the companies I've worked for unfortunately don't use your services, but I've always been a staunch advocate and converted a few. Maybe the higher-ups only see downtime and name recognition (i.e. you're not Amazon), but for what it's worth, us devs down the ladder definitely notice your transparency and communications, and it means the world. I've learned to structure my own postmortems after yours, and it's really aided in internal communications. Thank you again. I can't wait for the day I get to work in a fully-Cloudflare stack :)
- nijave 4y ago
  AWS is pretty decent if you're in an NDA contract (you have paid support). You can request RCAs for any incident you were impacted and they'll usually get them within a day. Not as transparent as "post it on the internet" but at least better than the usual hand wavey bullshit

thejosh 4y ago

Yep, look at heroku and their big incident, and the amount of downtime they've had lately.

viraptor 4y ago

I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture info that people will reference in discussions later. At a certain size it feels stupid not to post them publicly. I wonder how much those posts are calculated and how much organic/culture related.

hunter2_ 4y ago
> I wonder how much those posts are calculated and how much organic/culture related. Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
- viraptor 4y ago
  No they don't have such duty. In practice very little decision making is based on hard data in my experience. Real world being fuzzy and risk being hard to quantify do not help the situation.
- rrss 4y ago
  No. https://scholarship.law.cornell.edu/cgi/viewcontent.cgi?httpsredir=1&article=2311&context=facpub https://scholarship.law.cornell.edu/cgi/viewcontent.cgi?http...
smugma 4y ago
I agree that this is a free ad/recruitment. However, it’s easy to see how more conservative businesses see this as a risk. They are highlighting their deficiencies, letting their big important clients know that human error can bring their network down. Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warning sign to customers. It’s a strategy other companies could adopt, but to do it effectively requires changes all across the organization.
- saghm 4y ago
  OTOH, I think most actual engineers would know that everywhere has deficiencies and can be brought down by human error, and I'd personally rather use a product where the people running it admit this rather than just claim that their genius engineers made it 100% foolproof and nothing could ever possibly go wrong
  - ethbr0 4y ago
    Absolutely. The first step of good SRE is admitting (publicly and within the organization) that you have a problem.
  - smugma 4y ago
    100% agree. But… 1. On the buy side, crappy big companies with procurement etc. may not have “actual engineers” deciding things. Maybe for something like Cloudflare that’s likely to sit within an actual technical team’s mandate 2. Not 100% foolproof but if a startup is selling its tool and has an uptime page and details of all its downtime and as a prospective customer you go and see that they have 3-4 hour downtimes once or twice a month, it should raise alarm bells.
Uehreka 4y ago
Ehhhh… I think it’s good (for us) that they do this, but I don’t think it’s a free ad (contrary to popular belief, not all news is good news, and this is bad news) and any sort of conversion rate on recruitment is probably vanishingly small (which would normally be fine, but incidents like these may turn off some actual customers, which is where actual revenue comes from). I think their calculation (to the extent you can call it that) is that in the interest of PR and damage control, it’s better to get a thorough postmortem out quickly to stem the bleeding and keep people like us from going “I can’t wait to hear what happened at Cloudflare” for a week. Now we know, the customers have an explanation, and this bad news cycle has a higher chance of ending quickly.
- viraptor 4y ago
  > but incidents like these may turn off some actual customers Incidents - yes. But why would a post-mortem turn someone off? The incident happened regardless. Do you think anyone would be more likely turned off by reading how they solved it / plan to prevent it on the future than by silence?
zaidf 4y ago
>I feel like others lose opportunities by not doing the same IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.
- ethbr0 4y ago
  I believe Marcus Aurelius had something similar to say on the matter. :-)
  - jjtheblunt 4y ago
    quodcumque erat ?

mattferderer 4y ago

To further add to your point, the CTO is the one who shared it here & the CEO is incredibly active on forums & social media everywhere with customers. Communication has always been one of their strengths.

bluehatbrit 4y ago
I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.
- bombcar 4y ago
  It could be good or bad; I suspect they've thought about it and have worked on succession (I hope!) and have like-minded people in the wings. But once it happens things will change and, to be honest, likely for the worse. edit> fix typo
  - unityByFreedom 4y ago
    > secession Succession?
    - bombcar 4y ago
      Eep yes, auto spell check on macOS is usually good but sometimes it causes a civil war.
- unityByFreedom 4y ago
  Good communicators generally foster that environment. And their customers appreciate it, so there is an external expectation now too. Everything ends some day, but I think this will be regarded as a valuable attribute for awhile.
  - jgrahamc 4y ago

kevin_nisbet 4y ago

I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.

jgrahamc 4y ago
What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done. There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.
cowsandmilk 4y ago
I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post: > This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically. They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.
- bombcar 4y ago
  It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.
- 4y ago

tyingq 4y ago

>take weeks for other companies to publish a postmortem And with nowhere near the detail level of what was presented here. Typically lots of sweeping generalizations that don't tell you much about what happened, or give you any confidence they really know what happened or have the right fix in place.

Melatonic 4y ago

To be fair though they sort of MUST do things like this to have our confidence - their whole business is about being FAST and AVAILABLE. Were not talking about Oracle here :-D

agilob 4y ago

I'd love to see the postmortem from Facebook :(

kortilla 4y ago

Well cloudflare’s entire value is in uptime and preventing outages. Showing they have a rapid response and strong fundamental technical understanding is much more critical in the “prevent downtime” business.

solardev 4y ago

No provider is perfect, but it's because of stuff like this that I trust Cloudflare waaaaaaaaaaay more than the likes of Amazon. Transparency engenders trust, and eventually, love! Thank you, Cloudflare. The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your postmortem is excellent.) I couldn't even begin to think about designing or implementing something of this complexity, much less being able to explain it to a layperson after a failure. It is really impressive, and I hope you will continue to do so into the future! Most of the companies I've worked for unfortunately don't use your services, but I've always been a staunch advocate and converted a few. Maybe the higher-ups only see downtime and name recognition (i.e. you're not Amazon), but for what it's worth, us devs down the ladder definitely notice your transparency and communications, and it means the world. I've learned to structure my own postmortems after yours, and it's really aided in internal communications. Thank you again. I can't wait for the day I get to work in a fully-Cloudflare stack :)

nijave 4y ago
AWS is pretty decent if you're in an NDA contract (you have paid support). You can request RCAs for any incident you were impacted and they'll usually get them within a day. Not as transparent as "post it on the internet" but at least better than the usual hand wavey bullshit