5 ms·

"Software engineers on an on-call rotation...data analysts, maybe even some external contractors" These groups usually get access through a bastion that anonym

"Software engineers on an on-call rotation...data analysts, maybe even some external contractors" These groups usually get access through a bastion that anonymizes data and logs access. I remember that as a SWE at Google, I could run aggregated & anonymized statistics across query logs, but some info (eg. IPs, user logins) had been scrubbed before any of my code could get access to it, and for things that were more personal (eg. your GMail login) you could only get access to your own account. There's nothing you can do about SREs who have root access on the box or the SWEs who need to implement & maintain the bastion servers, but that's presumably a more restricted, vetted, and trusted group.

remarkEon 7y ago
Would it be smart for companies like google and twitter to publish how they vet people for these roles? That’s a seriously tremendous amount of power. I almost think it needs to be regulated like heath info. I get that that would make it more expensive to handle this data at all, and would serve as a significant barrier to entry for startups ... but data breach after data breach is pushing me in the regulation direction.
- C1sc0cat 7y ago
  The same way defences industries do I would imagine for SF86 etc
- retrovm 7y ago
  Pretty much nobody at Google has that kind of access. You can be an SRE of just about anything at Google and never access user data. The "break-glass" means of emergency access is ridiculously booby-trapped. A person wanting to do this thing has to 1) badge into a special room, at which time both production security and privacy incident teams are notified, 2) use a special hardware security device that is used for no other purpose than to activate a VPN box with a hard-line into the production network. By the way if a random Googler just rolls up to a datacenter without a reason to be there, that also triggers privacy incident response, even though physical access to production storage is virtually useless due to all the encryption. I would say it is much more likely that Google will accidentally lose the organizational ability to become root-in-prod, than it is that a person has done this thing without being noticed. In short, insider risk cannot be mitigated with hiring practices. You need robust technical measures against insider risk.
  - remarkEon 7y ago
    Thanks for proving my point, I guess? I’d love to see a formal write up from google about these procedures. It would go a long way to increasing confidence in how they handle this data.
  - mehrdadn 7y ago
    What I would love to see is somebody at Facebook comparing their barriers against accessing user data with these from Google.
    - remarkEon 7y ago
      I’m actually inclined to think they have similar procedures, if only because we haven’t seen “whistle blower” stories in the news about folks reading texts and looking at other private user data. Maybe I’ve missed them, but because bashing Facebook is kind of a trend one would think there’d be an appetite for “I read illicit group texts for a year here’s what I saw AMA” stories.
      - dannyw 7y ago
        If this happens, every incentive (of every party that is aware) is to not whistleblow, as it is a criminal offence for the snooper, and a PR disaster for the company.
  - belltaco 7y ago
    When did that start? Here's an account of someone working at Google accessing info to stalk teens. https://www.businessinsider.com/google-engineer-stalked-teens-spied-on-chats-2010-9 https://www.businessinsider.com/google-engineer-stalked-teen...
    - dredmorbius 7y ago
      That story dates from 2010. Snowden's revelations (2013) were a major watershed. There'd been several measures taken since, based on what I heard on the outside, largely through discussions, mostly public, a few direct, with Google staff via G+. Starting on, of all days, November 9th, 2016, I began regularly posting an image of Jewish shop windows shattered during Krystallnacht, asking whether Google were thinking of brownshirt-proofing their data. That generated responses including from G+'s architect (then in a role with user data safety & privicy), and the data security lead. It wasn't until some time later that I realised I'd entirely accidentally picked the anniversary of the event for the post. Though the coincidence was useful. My understanding was that numerous protections were in place by that time. I continue to have concerns.
- mcphage 7y ago
  > Would it be smart for companies like google and twitter to publish how they vet people for these roles? I think much like anti-hacking and anti-fraud efforts, publishing information about how they vet candidates would just make it easier for attackers to figure out how to game the system.
- ENOTTY 7y ago
  Ironically, one of the vetting regimes that has the most publicly available information about it is the US government's security clearance system. https://ogc.osd.mil/doha/isp.html https://ogc.osd.mil/doha/isp.html
  - goatinaboat 7y ago
    one of the vetting regimes that has the most publicly available information about it is the US government's Why wouldn’t the information be public? It’s the same concept as crypto algos being published and peer-reviewed.
pcwalton 7y ago
Even Google's policies aren't—or at least weren't—foolproof. In 2010 there was a major incident involving a Google SRE named David Barksdale: https://www.telegraph.co.uk/technology/google/8003925/Google-engineer-fired-for-privacy-breach-after-stalking-and-harrassing-teenagers.html https://www.telegraph.co.uk/technology/google/8003925/Google...
- nostrademons 7y ago
  Yeah, I was there when it happened (and it was a fairly big deal internally). He was an SRE with pretty high-level access. There's relatively little you can do for this kind of threat - the nature of their jobs usually requires that they have root on the box. (Though a sibling comment suggests that there's even more rigorous procedures now.)
  - C1sc0cat 7y ago
    Its the same for telcos, these days there is much stricter auditing and actual real security vetting - I know team leaders on some systems had to be PV (top secret in US terms) I suspect that at some stage this might come for some FANG employees
  - retrovm 7y ago
    The centralization of infrastructure has really changed the insider risk profile at that company. A decade ago it was common for a service to own their hardware and to have broad authority over those machines but not so much these days. Nobody wants to own their hardware, everything is uniform and it's easier to spot deviants that way. Of course many of the changes around that time were in response to the breach by the Chinese government, not only in response to embarrassing privacy incidents. Later improvements came about because of things Snowden published.
    - x220 7y ago
      Why would Google change because of the Snowden Leaks? They knew about it. They were handing over data to the NSA.
heartbreak 7y ago
> usually get access through a bastion that anonymizes data and logs access I think this may be something that is unique to a certain size tech company that simply isn't the case at 99.99% of companies with user data in their possession.
- CydeWeys 7y ago
  It should be the case at Twitter though.
  - twitter_anon 7y ago
    It hasn't been until recently. We are currently locking down all internal systems based on the minimum necessary accesses for various systems. We should have done this a lot sooner.
    - CydeWeys 7y ago
      Good to hear it. Better late than never.

remarkEon 7y ago

Would it be smart for companies like google and twitter to publish how they vet people for these roles? That’s a seriously tremendous amount of power. I almost think it needs to be regulated like heath info. I get that that would make it more expensive to handle this data at all, and would serve as a significant barrier to entry for startups ... but data breach after data breach is pushing me in the regulation direction.

C1sc0cat 7y ago
The same way defences industries do I would imagine for SF86 etc
retrovm 7y ago
Pretty much nobody at Google has that kind of access. You can be an SRE of just about anything at Google and never access user data. The "break-glass" means of emergency access is ridiculously booby-trapped. A person wanting to do this thing has to 1) badge into a special room, at which time both production security and privacy incident teams are notified, 2) use a special hardware security device that is used for no other purpose than to activate a VPN box with a hard-line into the production network. By the way if a random Googler just rolls up to a datacenter without a reason to be there, that also triggers privacy incident response, even though physical access to production storage is virtually useless due to all the encryption. I would say it is much more likely that Google will accidentally lose the organizational ability to become root-in-prod, than it is that a person has done this thing without being noticed. In short, insider risk cannot be mitigated with hiring practices. You need robust technical measures against insider risk.
- remarkEon 7y ago
  Thanks for proving my point, I guess? I’d love to see a formal write up from google about these procedures. It would go a long way to increasing confidence in how they handle this data.
- mehrdadn 7y ago
  What I would love to see is somebody at Facebook comparing their barriers against accessing user data with these from Google.
  - remarkEon 7y ago
    I’m actually inclined to think they have similar procedures, if only because we haven’t seen “whistle blower” stories in the news about folks reading texts and looking at other private user data. Maybe I’ve missed them, but because bashing Facebook is kind of a trend one would think there’d be an appetite for “I read illicit group texts for a year here’s what I saw AMA” stories.
    - dannyw 7y ago
      If this happens, every incentive (of every party that is aware) is to not whistleblow, as it is a criminal offence for the snooper, and a PR disaster for the company.
- belltaco 7y ago
  When did that start? Here's an account of someone working at Google accessing info to stalk teens. https://www.businessinsider.com/google-engineer-stalked-teens-spied-on-chats-2010-9 https://www.businessinsider.com/google-engineer-stalked-teen...
  - dredmorbius 7y ago
    That story dates from 2010. Snowden's revelations (2013) were a major watershed. There'd been several measures taken since, based on what I heard on the outside, largely through discussions, mostly public, a few direct, with Google staff via G+. Starting on, of all days, November 9th, 2016, I began regularly posting an image of Jewish shop windows shattered during Krystallnacht, asking whether Google were thinking of brownshirt-proofing their data. That generated responses including from G+'s architect (then in a role with user data safety & privicy), and the data security lead. It wasn't until some time later that I realised I'd entirely accidentally picked the anniversary of the event for the post. Though the coincidence was useful. My understanding was that numerous protections were in place by that time. I continue to have concerns.
mcphage 7y ago
> Would it be smart for companies like google and twitter to publish how they vet people for these roles? I think much like anti-hacking and anti-fraud efforts, publishing information about how they vet candidates would just make it easier for attackers to figure out how to game the system.
ENOTTY 7y ago
Ironically, one of the vetting regimes that has the most publicly available information about it is the US government's security clearance system. https://ogc.osd.mil/doha/isp.html https://ogc.osd.mil/doha/isp.html
- goatinaboat 7y ago
  one of the vetting regimes that has the most publicly available information about it is the US government's Why wouldn’t the information be public? It’s the same concept as crypto algos being published and peer-reviewed.

pcwalton 7y ago

Even Google's policies aren't—or at least weren't—foolproof. In 2010 there was a major incident involving a Google SRE named David Barksdale: https://www.telegraph.co.uk/technology/google/8003925/Google-engineer-fired-for-privacy-breach-after-stalking-and-harrassing-teenagers.html https://www.telegraph.co.uk/technology/google/8003925/Google...

nostrademons 7y ago
Yeah, I was there when it happened (and it was a fairly big deal internally). He was an SRE with pretty high-level access. There's relatively little you can do for this kind of threat - the nature of their jobs usually requires that they have root on the box. (Though a sibling comment suggests that there's even more rigorous procedures now.)
- C1sc0cat 7y ago
  Its the same for telcos, these days there is much stricter auditing and actual real security vetting - I know team leaders on some systems had to be PV (top secret in US terms) I suspect that at some stage this might come for some FANG employees
- retrovm 7y ago
  The centralization of infrastructure has really changed the insider risk profile at that company. A decade ago it was common for a service to own their hardware and to have broad authority over those machines but not so much these days. Nobody wants to own their hardware, everything is uniform and it's easier to spot deviants that way. Of course many of the changes around that time were in response to the breach by the Chinese government, not only in response to embarrassing privacy incidents. Later improvements came about because of things Snowden published.
  - x220 7y ago
    Why would Google change because of the Snowden Leaks? They knew about it. They were handing over data to the NSA.

heartbreak 7y ago

> usually get access through a bastion that anonymizes data and logs access I think this may be something that is unique to a certain size tech company that simply isn't the case at 99.99% of companies with user data in their possession.

CydeWeys 7y ago
It should be the case at Twitter though.
- twitter_anon 7y ago
  It hasn't been until recently. We are currently locking down all internal systems based on the minimum necessary accesses for various systems. We should have done this a lot sooner.
  - CydeWeys 7y ago
    Good to hear it. Better late than never.