4 ms·
That's one of the main reasons I'm leary about them. Such a big f-up is difficult to forget. It shows that they have a move fast and break things culture which
by redman25 2y ago
That's one of the main reasons I'm leary about them. Such a big f-up is difficult to forget. It shows that they have a move fast and break things culture which for a company that is responsible for critical infrastructure feels wrong.
- trane_project 2y agoI interviewed there once and they asked me what I would do if a service broke after a deployment. I said the first step was to revert to the last known good version and then investigate. Color me surprised when that was not the answer they expected.
- udev4096 2y agoThat's strange. What was the "correct" answer?
- hmottestad 2y agoIf I ever interview at Cloudflare and get this question I might answer with "call the sales team and have them fix it by selling someone an enterprise subscription paid upfront by the decade" just to see if the interviewers read Hacker News :P
- trane_project 2y agoThey wanted me to roll out a fix first. Apparently rolling back first was not moving fast enough for the interviewer’s liking.
- ExoticPearTree 2y agoDepending on the service’s criticality, the cost of rolling back versus pushing a fix, service dependencies in their environment… having to push a fix might have been the better approach. Without more details about the environment, it is a 50/50 call.
- definitelyauser 2y agoWas the correct answer related to cache invalidation?
- pornel 2y agoCloudflare's internal release tool suggests revert when monitoring detects failures during deployment, so this question doesn't describe Cloudflare's practices. There must have been something more to it, or it was a misunderstanding.
- pornel 2y agoIn response to this incident Cloudflare has made big engineering changes, including huge work to move away from C as much as possible. The offending parsers were rewritten in Rust (https://github.com/cloudflare/lol-html https://github.com/cloudflare/lol-html), as well as WAF, image optimization, and a few others. Nginx is being replaced with a custom cache server. New implementations are using either the Workers platform, or are written in Rust or Golang.