4 ms·
My understanding, paraphrased: "In order to gradually roll out one change, we had to globally push a different configuration change, which broke everything at o
by xnorswap 11mo ago
My understanding, paraphrased: "In order to gradually roll out one change, we had to globally push a different configuration change, which broke everything at once".
But a more important takeaway:
> This type of code error is prevented by languages with strong type systems
- skywhopper 11mo agoThis is the exact same type of error that happened in their Rust code last time. Strong type systems don’t protect you from lazy programming.
- inejge 11mo agoIt's not remotely the same type of error -- error non-handling is very visible in the Rust code, while the Lua code shows the happy path, with no indication that it could explode at runtime. Perhaps it's the similar way of not testing the possible error path, which is an organizational problem.
- debugnik 11mo agoPrevented unless they assert the wrong invariant at runtime like they did last time.
- jsnell 11mo agoThat's a bizarre takeaway for them to suggest, when they had exactly the same kind of bug with Rust like three weeks ago. (In both cases they had code implicitly expecting results to be available. When the results weren't available, they terminated processing of the request with an exception-like mechanism. And then they had the upstream services fail closed, despite the failing requests being to optional sidecars rather than on the critical query path.)
- littlestymaar 11mo agoIn fairness, the previous bug (with the Rust unwrap) should never have happened: someone explicitly called the panicking function, the review didn't catch it and the CI didn't catch it. It required a significant organizational failure to happen. These happen but they ought to be rarer than your average bug (unless your organization is fundamentally malfunctioning, that is)
- greatgib 11mo agoThe issue would also not have happened, if someone did the right code, tests, and the review or CI caught it...
- marcosdumay 11mo agoIt's different to expect somebody to write the correct program every time than to expect somebody not to call the "break_my_system" procedure that was warnings all over it telling people it's there for quick learning-to-use examples or other things you'll never run.
- pdimitar 11mo agoTo be precise, the previous problem with Rust was because somebody copped out and used a temporary escape hatch function that absolutely has no place in production code. It was mostly an amateur mistake. Not Rust's fault. Rust could never gain adoption if it didn't have a few escape hatches. "Damned if they do, damned if they don't" kind of situation. There are even lints for the usage of the `unwrap` and `expect` functions. As the other sibling comment points out, the previous Cloudflare problem was an acute and extensive organizational failure.
- zozbot234 11mo agoYou can make an argument that .unwrap() should have no place in production code, but .expect("invariant violated: etc. etc.") very much has its place. When the system is in an unpredicted and not-designed-for state it is supposed to shut down promptly, because this makes it easier to troubleshoot the root cause failure whereas not doing so may have even worse consequences.
- pdimitar 11mo agoI don't disagree but you might as well also manually send an error to f.ex. Sentry and just halt processing of the request. Though that really depends. In companies where k8s is used the app will be brought back up immediately anyway.
- Hamuko 11mo agoYeah, my first thought was that had they used Rust, maybe we would've seen them point out a rule_result.unwrap() as the issue.