4 ms·
Another reason postmortems can be vague is that, for each detail you add, you might need to add even more background details. Then it becomes a recursive proble
by puzzle 8y ago
Another reason postmortems can be vague is that, for each detail you add, you might need to add even more background details. Then it becomes a recursive problem.
In this case, the "configuration change" could be a feature flag in the L2 GFEs, something in the L1 GFEs that changed L1->L2 requests in minor ways, or maybe something else entirely (since it's partially security-related: a dynamic LOAS handshake change to use different cyphers? So many possibilities). At the end of the day, though, it's still a specific permutation of all possible features and flags that hadn't been vetted before. Given how large Google's monorepo is, it's not impossible for one of your many dependencies, even indirect ones, which might be configured by another team entirely, to have subtle time bombs that only trigger well after you have built and deployed the code.
Having been on the other side, I know that, for every detail added, a bunch more questions come up.
- linza 8y agoIs there a write up that explains what the technologies are that you mentioned in your post? LOAS, L1 GFE, L2 GFE, ... never heard of those
- breakingcups 8y agoL1 and L2 GFE are the Level 1 and Level 2 Google Front Ends which the post mortem talks about. I'm not familiar with LOAS.
- puzzle 8y agoLOAS is how Stubby (gRPC) connections on the internal network are secured, looks a bit like mTLS and is partially being open sourced as ALTS: https://cloud.google.com/security/encryption-in-transit/application-layer-transport-security/ https://cloud.google.com/security/encryption-in-transit/appl... GFE (Google Front End) is what you connect to when you access any Google services through your browser. Think nginx or even ELB. It's a load balancing reverse proxy, as well as WAF. It's mentioned here https://cloud.google.com/security/infrastructure/design/ https://cloud.google.com/security/infrastructure/design/ and probably in the SRE book. This report might be the first time Google mentions in public that there can be two levels of GFEs, but I remember at least one service using such a setup many, many years ago.