4 ms·
Kudos to Cloudflare for clarity and diligence. When talking of their earlier Lua code: > we have never before applied a killswitch to a rule with an action of
by w10-1 11mo ago
Kudos to Cloudflare for clarity and diligence.
When talking of their earlier Lua code:
> we have never before applied a killswitch to a rule with an action of “execute”.
I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation?
It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineering. It's just that I think of CloudFlare as an infrastructure place, where that shouldn't be true.
I had a manager who came from defense electronics in the 1980's. He said in that context, the quality engineering team was always in charge, and always more skilled. For him, software is backwards.
- braiamp 11mo agoThis is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.
- mopsi 11mo ago... until the extended-range version is ordered and no one remembers to fix the leak. :]
- wizzwizz4 11mo agoThey will remember, because it'll have been measured and documented, rigorously.
- SketchySeaBeast 11mo agoI've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.
- lloeki 11mo agoBy reading the documentation thoroughly as a compulsory first step to designing the next system that depends on it. I realise this may probably boggle the mind of the modern software developer.
- lukan 11mo agoYou say this like trivial misstakes did not happen all the time in classical engineering as well. If there is a memory leak, them this is a flaw, that might not matter so much for a specific product, but I can also easily see it being forgotten, if it was maybe mentioned somewhere in the documentation, but maybe not clear enough and deadlines and stress to ship are a thing there as well.
- hinkley 11mo agoThat is not how this usually works. Most of the time QA can tell you exactly how the product works, regardless of what the documentation says. But many of us haven’t seen a QA team in five, ten years.
- switchbak 11mo agoJust try harder. And if it still breaks, clearly you weren't trying hard enough! At some point you have to admit that humans are pretty bad at some things. Keeping documentation up to date and coherent is one of those things, especially in the age of TikTok. Better to live in the world we have and do the best you can, than to endlessly argue about how things should be but never will become.
- vimwizard 11mo ago> especially in the age of TikTok Shouldn't grey beards, grizzled by years of practicing rigorous engineering, be passing this knowledge on to the next generation? How did they learn it when just starting out? They weren't born with it. Maybe engineering has actually improved so much that we only need to experience outages this frequently, and such feelings of nostalgia are born from never having to deal with systems having such high degrees of complexity and, realistically, 100% availability expectations on a global scale.
- hinkley 11mo agoWas this one measured and documented rigorously? Well obviously not, because the front fell off. That’s a dead giveaway.
- hinkley 11mo agoAriane 5 happens.
- Etheryte 11mo agoThis paraphrased urban legend has nothing to do with quality engineering though? As described, it's designed to the spec and working as intended.
- mikkupikku 11mo agoIt tracks with my experience in software quality engineering. Asked to find problems with something already working well in the field. Dutifully find bugs/etc. Get told that it's working though so nobody will change anything. In dysfunctional companies, which is probably most of them, quality engineering exists to cover asses, not to actually guide development.
- colechristensen 11mo agoIt is not dysfunctional to ignore unreachable "bugs". A memory leak on a missile which won't be reached because it will explode long before that amount of time has passed is not a bug.
- wkat4242 11mo agoIt's a debt though. Because people will forget it's there and then at some point someone changes a counter from milliseconds to microseconds and then the issue happens 1000 times sooner. It's never right to leave structural issues even if "they don't happen under normal conditions".
- Etheryte 11mo agoI don't think this argument makes sense. You wouldn't provision a 100GB server for a service where 1GB would do just in case unexpected conditions come up. If the requirements change, then the setup can change, doing it just because is wasteful. What if we forget is not a valid argument to over engineer and over provision.
- datadrivenangel 11mo ago
- sally_glance 11mo agoHaving observed an average of two mgmt rotations at most of the clients our company is working for this comes at absolutely no surprise to me. Engineering is acting perfectly reasonable, optimizing for cost and time within the constraints they were given. Constraints are updated at a (marketing or investor pleasure) whim without consulting engineering, cue disaster. Not even surprising to me anymore...
- runlaszlorun 11mo agoMy hunch is that we do the same with memory leaks or other bugs in web applications where the time of a request is short.
- zwnow 11mo ago"Kudos"? This is like the South Park episode in which the oil company guy just excuses himself while the company just continues to fuck up over and over again. There's nothing to praise, this shouldn't happen twice in a month. Its inexcusable.
- vpShane 11mo agotwice in a month _so far_
- Bengalilol 11mo agoThose AI agents are coding fast, or am I missing some obvious concept here?
- deleted 11mo ago[deleted]
- hinkley 11mo agoWe still have two holidays and associated vacations and vacation brain to go. And then the January hangover. Every company that has ignored my following advice has experienced a day for day slip in first quarter scheduling. And that advice is: not much work gets done between Dec 15 and Jan 15. You can rely on a week worth, more than that is optimistic. People are taking it easy and they need to verify things with someone who is on vacation so they are blocked. And when that person gets back, it’s two days until their vacation so it’s a crap shoot. NB: there’s work happening on Jan 10, for certain, but it’s not getting finished until the 15th. People are often still cleaning up after bad decisions they made during the holidays and the subsequent hangover.
- kordlessagain 11mo agoreaching for that _one 9 of uptime_
- deleted 11mo ago[deleted]
- ifwinterco 11mo agoIt's weird reading these reports because they don't seem to test anything at all (or at least there's very little mention of testing). Canary deployment, testing environments, unit tests, integration tests, anything really? It sounds like they test by merging directly to production but surely they don't
- Dumbledumb 11mo agoIn the post they described that they observed errors happening in their testing env, but decided to ignore because they were rolling out a security fix. I am sure there is more nuance to this, but I don’t know whether that makes it better or worse
- misswaterfairy 11mo ago> but decided to ignore because they were rolling out a security fix. A key part of secure systems is availability... It really looks like vibe-coding.
- chippiewill 11mo agoThe problem is that Cloudflare do incremental rollouts and loads of testing for _code_. But they don't do the same thing for configuration - they globally push out changes because they want rapid response. It's still a bit silly though, their claimed reasoning probably doesn't really stack up for most of their config changes - I don't see it to be that likely that a 0.1->1->10->100 rollout over the period of 10 minutes would be a catastrophically bad idea for them for _most_ changes. And to their credit, it does seem they want to change that.
- ifwinterco 11mo agoYeah to me it doesn't make any sense - configuration changes are just as likely to break stuff (as they've discovered the hard way) and both of these issues could have been found in a testing environment before being deployed to production