5 ms·
I work on Cloud, but not this service. In general: - All the code has unit tests and integration tests - Binary and config file changes roll out slowly job by
by esprehn 1y ago
I work on Cloud, but not this service. In general:
- All the code has unit tests and integration tests
- Binary and config file changes roll out slowly job by job, region by region, typically over several days. Canary analysis verifies these slow rollouts.
- Even panic rollbacks are done relatively slowly to avoid making the situation worse. For example globally overloading databases with job restarts. A 40m outage is better than a 4 hour outage.
I have no insider knowledge of this incident, but my read of the PM is: The code was tested, but not this edge case. The quota policy config is not rolled out as a config file, but by updating a database. The database was configured for replication which meant the change appeared in all the databases globally within seconds instead of applying job by job, region by region, like a binary or config file change.
I agree on the frustration with null pointers, though if this was a situation the engineers thought was impossible it could have just as likely been an assert() in another language making all the requests fail policy checks as well.
Rewriting a critical service like this in another language seems way higher risk than making sure all policy checks are flag guarded, that all quota policy checks fail open, and that db changes roll out slowly region by region.
Disclaimer: this is all unofficial and my personal opinions.
- blibble 1y ago> The code was tested, but not this edge case. so... it wasn't tested
- dieortin 1y agoSo you need to write a test for every single possible case to consider your code tested?
- OtherShrezzing 1y agoI don’t think it’s unreasonable to hold GCP & co to a higher standard than run of the mill software. The platform generates around $100mn per day for Google with a 10% operating margin. Testing all inevitable code paths which have the capacity to bring the platform down as soon as it’s deployed doesn’t seem like an unreasonable ask for a corp operating at that scale.
- deleted 1y ago[deleted]
- madeofpalk 1y agoI mean... yes?
- jsnell 1y ago> it could have just as likely been an assert() in another language Asserts are much easier to forbid by policy.
- esprehn 1y agoThat's fair, though `if (isInvalidPolicy) reject();` causes the same outage. So the eng process policy change seems to be failing open and slow rollouts to catch that case too.
- usrnm 1y agoHow is the fact that it was a database change and not a binary or a config supposed to make it ok? A change is a change, global changes that go everywhere at once are a recipe for disaster, it doesn't matter what kind of changes we're talking about. This is a second Crowdsrike
- fidotron 1y agoThis is the core point. A canary deployment that was not preceded by deploying data that activates the region of the binary in question will prove nothing useful at all, while promoting a false sense of security.
- ethbr1 1y agoThe root problem is that a dev team didn't appropriately communicate criteria for testing their new feature. Which definitely seems like shortcut 'on to the next thing and ignore QA due diligence.'
- sagarm 1y agoGoogle does not separate dev and QA in general (I'm sure there are exceptions).
- ethbr1 1y agoThat sounds like a conflict of interest in practice. Who's ass is responsible when a dev team starts missing coverage in their tests, like happened here?
- sagarm 1y ago...the engineering team? I'm confused by this idea that engineering teams are not responsible for software quality. Who else would be? My experience with separate QA teams was not encouraging. They dramatically slowed down releases while failing to catch bugs. Their capability to automate was poor, so they were doing manual testing that provided little coverage especially compared to our comprehensive submit-blocking unit and integration tests.
- bdhess 1y ago> Rewriting a critical service like this in another language seems way higher risk than making sure all policy checks are flag guarded So like, the requirements are unknown? Or this service isn't critical enough to staff a careful migration?