6 ms·
Questions for Cloudflare
- mnholt 1y agoThis website could benefit from a CDN…
- majke 1y agoQuestions for "questions for cloudflare" owner
- jf 1y agohttps://web.archive.org/web/20251119165814/https://entropicthoughts.com/questions-for-cloudflare https://web.archive.org/web/20251119165814/https://entropict...
- internetter 1y ago8.5s... yikes... although notably they aren't adopting an anti-CDN or even really anti-cloudlare perspective, just grievances with software architecture. So the slowness of their site isn't really detrimental to their argument
- Sesse__ 1y agoI loaded it and got an LCP of ~350 ms, which is better than the ~550 ms I got from this very comment page.
- tptacek 1y agoIt's a detailed postmortem published within a couple hours of the incident and this blog post is disappointed that it didn't provide a comprehensive assessment of all the procedural changes inside the engineering organization that came as a consequence. At the point in time when this blog post was written, it would not have been possible for them to answer these questions.
- otterley 1y ago"But I need attention now!"
- kqr 1y agoPart of my argument in the article is that it does't take long to come to that realisation if using the right methods. It would absolutely have been possible to identify the problem of missing feedback by that time.
- tptacek 1y agoIt absolutely does take long with the right methods; in fact, the righter the methods, the longer it takes. You're talking about a postmortem that was up within single digit hours of the initial incident resolution. A lot of orgs would wait on the postmortem just to be sure the system is settling back into a steady state! You were way off here.
- kqr 1y agoTo be clear, I'm not expecting a full analysis within hours. I'm hoping for a method of analysis by which the major deficiencies come up at a high level, and then as more effort is spent on it, more details around those deficiencies are revealed. What otherwise tends to happen, in my experience, is the initial effort brings up some deficiencies which are only partially the major ones, and subsequent effort is spent looking mainly in that same area, never uncovering those major deficiencies which were not initially discovered.
- RationPhantoms 1y ago> I wish technical organisations would be more thorough in investigating accidents. Cloudflare is probably one of the best "voices" in the industry when it comes to post-mortems and root cause analysis.
- timenotwasted 1y ago"I don’t know. I wish technical organisations would be more thorough in investigating accidents." - This is just armchair quarterbacking at this point given that they were forthcoming during the incident and had a detailed post-mortem shortly after. The issue is that by not being a fly on the wall in the war room the OP is making massive assumptions about the level of discussions that take place about these types of incidents long after it has left the collective conscience of the mainstream.
- cogman10 1y agoPeople outside of tech (and some inside) can be really bad at understanding how something like this could slip through the cracks. Reading cloudflare's description of the problem, this is something that I could easily see my own company missing. It's the case that a file got too big which tanked performance enough to bring everything down. That's a VERY hard thing to test for. Especially since this appears to have been a configuration file and a regular update. The reason it's so hard to test for is because all tests would show that there's no problem. This isn't a code update, it was a config update. Without really extensive performance tests (which, when done well, take a long time!) there really wasn't a way to know that a change that appeared safe wasn't. I personally give Cloudflare a huge pass for this. I don't think this happened due to any sloppiness on their part. Now, if you want to see a sloppy outage you look at the Crowdstrike outage from a few years back that bricked basically everything. That is what sheer incompetence looks like.
- jsnell 1y agoI don't believe that is an accurate description of the issue. It wasn't that the system got too slow due to a big file, it's that the file getting too big was treated as a fatal error rather than causing requests to fail open.
- s1mplicissimus 1y ago> This isn't a code update, it was a config update Oh okay, well I guess the outage wasn't a real issue then
- 1y ago
- colesantiago 1y agoMaybe instead of asking "questions" to a corporation which their only interest is profit, is now beholden Wall Street and wouldn't care what we think, we should look for answers and alternatives like BunnyCDN [0], Anubis [1], etc. [0] https://bunny.net/ https://bunny.net/ [1] https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
- arbll 1y agoAh yes because both of those alternatives are non-profits right ?
- colesantiago 1y agoYou can sponsor Anubis right now and start supporting alternatives.
- arbll 1y agoI think you are misunderstanding what cloudflare provides if you think Anubis is an alternative. Even if we only consider bot protection they are completely different solutions.
- colesantiago 1y ago> I think you are misunderstanding what cloudflare provides if you think Anubis is an alternative... Do you have an open source alternative especially if you are donating to the project? Because I would to see Anubis and other alternatives thrive much more than closed ones like Cloudflare.
- arbll 1y agoCDN are by essence proprietary because they are infrastructure vendors. They can be built with open source software but what they are selling isn't software, it's physical servers. The alternative is going on-prem which is impossible for CDN if you aren't google or meta.
- blixt 1y agoIt's a bit odd to come from the outside to judge the internal process of an organization with many very complex moving parts, only a fraction of which we have been given context for, especially so soon after the incident and the post-mortem explaining it. I think the ultimate judgement must come from whether we will stay with Cloudflare now that we have seen how bad it can get. One could also say that this level of outage hasn't happened in many years, and they are now freshly frightened by it happening again so expect things to get tightened up (probably using different questions than this blog post proposes). As for what this blog post could have been: maybe a page out of how these ideas were actively used by the author at e.g. Tradera or Loop54.
- deleted 1y ago[deleted]
- kqr 1y ago> how these ideas were actively used by the author at e.g. Tradera or Loop54. This would be preferable, of course. Unfortunately both organisations were rather secretive about their technical and social deficiencies and I don't want to be the one to air them out like that.
- otterley 1y agoThe post is describing a full port-mortem process including a Five Whys (https://en.wikipedia.org/wiki/Five_whys https://en.wikipedia.org/wiki/Five_whys) inquiry. In a mature organization that follows best SRE practices, this will be performed by the relevant service teams, recorded in the port-mortem document, and used for creating follow-up actions. It's almost always an internal process and isn't shared with the public--and often not even with customers under NDA. We mustn't assume that CloudFlare isn't undertaking this process because we're not an audience to it.
- tptacek 1y agoIt also couldn't have happened by the time the postmortem was produced. The author of this blog post appears not to have noticed that the postmortem was up within a couple hours of resolving the incident.
- otterley 1y agoExactly. These deeper investigations can sometimes take weeks to complete.
- dkyc 1y agoThese engineering insights were not worth the 16 seconds load time this website took. It's extremely easy, and correspondingly valueless, to ask all kinds of "hard questions" about a system 24h after it had a huge incident. The hard part is doing this appropriately for every part of the system before something happens, while maintaining the other equally rightful goals of the organizations (such as cost-efficiency, product experience, performance, etc.). There's little evidence that suggests Cloudflare isn't doing that, and their track record is definitely good for their scale.
- raincole 1y agoEvery engineer has this phase when you're capable enough to do something at small scale, so you look at the incumbents, who are doing the similar thing but at 1000x scale, and wonder how they are so bad at it. Some never get out of this phase though.
- Nextgrid 1y agoIt is unfair to blame Cloudflare (or AWS, or Azure, or GitHub) for what’s happening, and I say that as one of the biggest “yellers at the cloud” on here. Ultimately end-users don’t have a relationship with any of those companies. They have relationships with businesses that chose to rely on them. Cloudflare, etc publish SLAs and compensation schedules in case those SLAs are missed. Businesses chose to accept those SLAs and take on that risk. If Cloudflare/etc signed a contract promising a certain SLA (with penalties) and then chose to not pay out those penalties then there would be reasons to ask questions, but nothing suggests they’re not holding up their side of the deal - you will absolutely get compensated (in the form of a refund on your bill) in case of an outage. The issue is that businesses accept this deal and then scream when it goes wrong, yet are unwilling to pay for a solution that does not fail in this way. Those solutions exist - you absolutely can build systems that are reliable and/or fail in a predictable and testable manner; it’s simply more expensive and requires more skill than just slapping a few SaaSes and CNCF projects together. But it is possible - look at the uptime of card networks, stock exchanges, or airplane avionics. It’s just more expensive and the truth is that businesses don’t want to pay for it (and neither are their end-customers - they will bitch about outages, but will immediately run the other way if you ask them to pony up for a more reliable system - and the ones that don’t, already run such a system and were unaffected by the recent outages).
- psim1 1y ago> It is unfair to blame Cloudflare (or AWS, or Azure, or GitHub) for what’s happening > Ultimately end-users don’t have a relationship with any of those companies. They have relationships with businesses that chose to rely on them Could you not say this about any supplier relationship? No, in this case, we all know the root of the outage is CloudFlare, so it absolutely makes sense to blame CloudFlare, and not their customers.
- Nextgrid 1y agoDevil’s advocate: I operate the equivalent of an online lemonade stand, some shitty service at a cheap price offered with little guarantees (“if I fuck up I’ll refund you the price of your ‘lemonade’”) for hobbyists to use to host their blog and Visa decides to use it in their critical path. Then this “lemonade stand” goes down. Do you think it’s fair to blame me? I never chose to be part of Visa’s authorization loop, and after all is done I did indeed refund them the price of their “lemonade”. It’s Visa’s fault they introduced a single point of failure with inadequate compensation schedules in their critical path.
- waiwai933 1y ago> Maybe some of these questions are obviously answered in a Cloudflare control panel or help document. I’m not in the market right now so I won’t do that research. I don't love piling on, but it still shocks me that people write without first reading.
- jcmfernandes 1y agoThe tone is off. Cloudflare shared a post-mortem on the same day as the incident. It's unreasonable to throw a "I wish technical organisations would be more thorough in investigating accidents". With that said, I would also like to know how it took them ~2 hours to see the error. That's a long, long time.
- vlovich123 1y agoA lot of these questions bely a misunderstanding of how it works - bot management is evaluated inline within the proxy as a feature on the site (similar to other features like image optimization). So during ingress there’s not an async call to the bot management service which intercepts the request before it’s outbound to origin - it’s literally a Lua script (or rust module in fl2) that runs on ingress inline as part of handling the request. Thus there’s no timeout or other concerns with the management service failing to assign a bot score. There are better questions but to me the ones posed don’t seem particularly interesting.
- kqr 1y agoMaybe I'm misunderstanding something but it being a blocking call does not make timeouts less important -- if anything they become more important!
- tptacek 1y agoI don't understand how it is you're doing distributed systems design on a system you don't even have access to. Maybe the issue is timeouts, maybe the issue is some other technical change, maybe the issue is human/procedural. How could you possibly know? The owners of the system probably don't have a full answer within just a few hours of handling the incident!
- kqr 1y agoI would be worried if they had all the answers within a few hours! I was just caught off guard by the focus on technical control measures when there seems to have been fairly obvious problems with information channels. For example, "more global kill switches for features" is good, but would "only" have shaved 30 % off the time of recovery (if reading the timeline charitably). Being able to identify the broken component faster would have shaved 30–70 % off the time of recovery depending on how fast identification could happen – even with no improvements to the kill switch situation.
- vlovich123 1y agoIt’s literally in the same process as the router. Performance issues like the layer taking too long are handled differently than just timing out afaik. It’s possible it should time out but safe cancellation in Rust can be tricky depending on how this is architected. Timeouts are irrelevant to this outage and I wouldn’t classify it as a huge issue unless there’s a possibility that a layer has an indefinite blocking call as a misbehavior and forever hangs all requests. My hunch is that this is generally already handled - FL2 is not documented enough to speculate.
- spenrose 1y agoI am disappointed to see this article flagged. I thought it was excellent.
- kqr 1y agoIn defense of your taste, it was updated based on the loud feedback here, so you probably read a slightly better version than that which was flagged.
- reilly3000 1y agoOP’s STPM post linked in the article was a fantastic read. Within the framework that sets up, these kinds of questions make perfect sense. The idea of a premortem deserves its day. https://entropicthoughts.com/aws-dynamodb-outage-stpa https://entropicthoughts.com/aws-dynamodb-outage-stpa