3 ms·
Insider anecdotes: security bugs were considered P0s when I was at Google, i.e. "drop everything and fix it immediately", where "everything" also included thing
by nostrademons 8y ago
Insider anecdotes: security bugs were considered P0s when I was at Google, i.e. "drop everything and fix it immediately", where "everything" also included things like dinner and sleep (if a security bug was reported in the middle of the night, the on-call SRE had authority to roll back to the last known good version and have engineering fix it in the morning and go through the normal review process). That created a pretty strong incentive not to introduce security bugs.
There was also a strong defense-in-depth culture, where we'd strive to avoid introducing security bugs in the JS, but the frameworks would also defend against them, and the GFEs (load balancers) would themselves detect intrusion attempts, as would the webservers, which would be architected so that bad data would be rejected & logged rather than triggering undefined behavior, and so on. So if a bug was discovered, it usually wouldn't be exploitable, and the whole time the hacker is trying to exploit it they have Google's security team working against them (and potentially calling up their ISP saying "The user with IP address xxx.xxx.xxx.xxx is trying to hack us, could you kick them off the Internet so we don't have to block your whole IP range?")
- lacker 8y agoFor what it's worth, and as someone who worked at both places, Facebook does these things pretty much the same way as Google does.
- quickthrower2 8y agoOut of interest why have an on call SRE when you have Google employees around the world. It's a reasonable time somewhere? The advantage is having someone who is more 'with it' when the problem arises. I can guess the answer - maybe it is a trust issue, they want to minimize the number of people with rollback privileges.
- nostrademons 8y agoPartially trust, but also because SRE is a specialized role with specialized tools and a specialized skillset. (And a slightly different mindset from regular engineers - SREs are very attuned to "what can go wrong?", while engineers are supposed to be attuned to "what can go right?".) You need to be more conversant with the high-level details of how the system operates (eg. which binaries are run, what are their resource requirements, what are common failure modes, what can cause cascading failures, when is everything deployed), and much less conversant with the details of how a particular piece of code runs, since the default action when it breaks is just "shut it off and let the engineer debug". Plus SREs are paid more (because they have on-call hours), so it wouldn't really save Google any money to have any every engineer act as a potential SRE. That said, when I was there many teams of smaller, riskier, or less critical services actually did have engineers carry the pager, sometimes with reduced SLAs (and an interlock built into the code so that an outage in their service wouldn't eg. take down websearch).