4 ms·

I'm guessing you work/have recently worked at a big tech company (FANG or one of the ~5 other companies of comparable size) and are seriously overestimating how

I'm guessing you work/have recently worked at a big tech company (FANG or one of the ~5 other companies of comparable size) and are seriously overestimating how common their best practices are. Unless by "passing grades as an undergraduate" you mean "bonuses and promotions at a majority of the companies that handle your data every day"

jeffbee 6y ago
G did not really get serious about infrastructure security until after the China hack (and more-so after NSA/Snowden) and didn't really get serious about insider risk until after "gcreep". Still, I don't understand the reluctance of the industry at large to learn the lessons of other people's failures. Why does each company need to separately discover that insider risk cannot be prevented by recruiting, it has to be prevented in code and hardware? Building a large-scale information system is like building a nuclear power station. There are a million ways to screw it up and only a few recognized right ways. If you ignore the best practices, it will eventually destroy your company and harm your users. Twitter have nuked themselves here. How can they come back from this? It sure looks like an insider risk mitigation system would have been money well spent.
- Leherenn 6y ago
  Because it is expensive? Yes, that might be a bad trade in the long run, but history has shown us times and times again that people are bad at evaluating those risks.
- thu2111 6y ago
  I think you're a bit starry eyed about Google. I had a fairly high level of Gmail and Gaia administrator access for a while when I worked there, including the post Snowden era. Resetting the password on an account would indeed trigger an audit event, and I'd be asked what was going on. I could provide any plausible sounding reason and that was sufficient, it wasn't really investigated. And that was the right level of oversight because as far as I know nobody with that kind of access ever abused it by making up a plausible sounding reason. Stopping bad insiders is really hard. Attempting to do it makes most organisations totally dysfunctional. There is one very famous kind of company that combats bad insiders regularly and with huge quantities of systems - a bank. Investment banks in particular. Whenever you read about 'rogue traders' they inevitably had to do a lot of stuff to disable all the various security systems trying to catch rogue traders. Institutionally distrusting your own employees can lead to seriously messed up IT systems. It's one of the reasons that bank employees are notoriously unable to access so many ordinary external websites, or services like Slack. It's how you can get "administrators" that can't read the logs of the service they supposedly administer. Encrypted messaging services in particular are poison to an org that's trying to stop employees exfiltrating valuable data. Google can just about do a good job of it because it has an essentially unlimited budget, which it spends on rolling its own tools for absolutely everything and integrating it all into one uber-architecture. From an economics perspective this makes no sense - comparative advantage etc - and thus basically no other company can do it that way. They have to buy or deploy open source tools that use a wide array of threat models and security systems but 95% of them will assume a trusted admin. Then try and hack things on top to restrict what rogue admins can do. It's deeply unpleasant.
  - GauntletWizard 6y ago
    Having been in several situations - As Gaia admin, working for big budget low competence IT for a "major" company, and as a shoestring SRE on a household name that's still held together by duct tape in some corners - it weird what is obvious, what is possible, and what level of escalation would be required for what kind of attack. It would have be possible and even trivial for me to impersonate a user at any of the three. At Google, I would have left indelible tracks that would have gotten me fired, see Gcreep (whom, oddly enough, I replaced - I was the next SRE hire at Google Kirkland after he'd been sacked). At the largeco, the tracks would have been indecipherable; nobody would have been able to notice. The logging wasn't there. The ability to analyze what logs they had wasn't there. As a shoestring engineer, I'm pretty sure I would have clear knowledge of who did what if something were discovered, but I would have a significant problem finding it unless something were obviously wrong. I know I can't stop a rogue admin; my team is small enough and needs to react fast enough that we can't spare time for access controls or break-glass, even if they were handed to us on a silver platter. I'm quite concerned about what that means and what this means, and I'm watching this intently. Probably for nothing; I know this is in the realm of risk we're unprepared for, and can't prepare for. Darned if I don't worry anyway.

jeffbee 6y ago

G did not really get serious about infrastructure security until after the China hack (and more-so after NSA/Snowden) and didn't really get serious about insider risk until after "gcreep". Still, I don't understand the reluctance of the industry at large to learn the lessons of other people's failures. Why does each company need to separately discover that insider risk cannot be prevented by recruiting, it has to be prevented in code and hardware? Building a large-scale information system is like building a nuclear power station. There are a million ways to screw it up and only a few recognized right ways. If you ignore the best practices, it will eventually destroy your company and harm your users. Twitter have nuked themselves here. How can they come back from this? It sure looks like an insider risk mitigation system would have been money well spent.

Leherenn 6y ago
Because it is expensive? Yes, that might be a bad trade in the long run, but history has shown us times and times again that people are bad at evaluating those risks.
thu2111 6y ago
I think you're a bit starry eyed about Google. I had a fairly high level of Gmail and Gaia administrator access for a while when I worked there, including the post Snowden era. Resetting the password on an account would indeed trigger an audit event, and I'd be asked what was going on. I could provide any plausible sounding reason and that was sufficient, it wasn't really investigated. And that was the right level of oversight because as far as I know nobody with that kind of access ever abused it by making up a plausible sounding reason. Stopping bad insiders is really hard. Attempting to do it makes most organisations totally dysfunctional. There is one very famous kind of company that combats bad insiders regularly and with huge quantities of systems - a bank. Investment banks in particular. Whenever you read about 'rogue traders' they inevitably had to do a lot of stuff to disable all the various security systems trying to catch rogue traders. Institutionally distrusting your own employees can lead to seriously messed up IT systems. It's one of the reasons that bank employees are notoriously unable to access so many ordinary external websites, or services like Slack. It's how you can get "administrators" that can't read the logs of the service they supposedly administer. Encrypted messaging services in particular are poison to an org that's trying to stop employees exfiltrating valuable data. Google can just about do a good job of it because it has an essentially unlimited budget, which it spends on rolling its own tools for absolutely everything and integrating it all into one uber-architecture. From an economics perspective this makes no sense - comparative advantage etc - and thus basically no other company can do it that way. They have to buy or deploy open source tools that use a wide array of threat models and security systems but 95% of them will assume a trusted admin. Then try and hack things on top to restrict what rogue admins can do. It's deeply unpleasant.
- GauntletWizard 6y ago
  Having been in several situations - As Gaia admin, working for big budget low competence IT for a "major" company, and as a shoestring SRE on a household name that's still held together by duct tape in some corners - it weird what is obvious, what is possible, and what level of escalation would be required for what kind of attack. It would have be possible and even trivial for me to impersonate a user at any of the three. At Google, I would have left indelible tracks that would have gotten me fired, see Gcreep (whom, oddly enough, I replaced - I was the next SRE hire at Google Kirkland after he'd been sacked). At the largeco, the tracks would have been indecipherable; nobody would have been able to notice. The logging wasn't there. The ability to analyze what logs they had wasn't there. As a shoestring engineer, I'm pretty sure I would have clear knowledge of who did what if something were discovered, but I would have a significant problem finding it unless something were obviously wrong. I know I can't stop a rogue admin; my team is small enough and needs to react fast enough that we can't spare time for access controls or break-glass, even if they were handed to us on a silver platter. I'm quite concerned about what that means and what this means, and I'm watching this intently. Probably for nothing; I know this is in the realm of risk we're unprepared for, and can't prepare for. Darned if I don't worry anyway.