6 ms·

You can and should restrict the number of people with access to the data, but in a tech company there's always going to be a significant number of people with d

You can and should restrict the number of people with access to the data, but in a tech company there's always going to be a significant number of people with direct access to the raw data. Software engineers on an on-call rotation, full-time site reliability engineers, data analysts, maybe even some external contractors... maybe this isn't the case for Twitter, but many companies also have to put complete trust in their cloud provider or datacenter, which puts even more people in the loop. Even if you follow best practices with access control, in the end you're always going to have a group of people who you need to trust with access to folks' personal data. Maybe the solution is better audit logging and even tighter access, but I'm not sure leaks of this nature are preventable.

ForHackernews 7y ago
You can significantly reduce these problems by decentralizing data and moving away from giant platforms that make such enticing targets for espionage.
CydeWeys 7y ago
"but in a tech company there's always going to be a significant number of people with direct access to the raw data" This is absolutely not true. This number can and should be reduced to an absolute minimum number of people.
- bmiller2 7y ago
  The absolute minimum number of people may still be significant
- tristanstcyr 7y ago
  Sure, but it comes at a cost. Companies rarely push the limits of these kinds of policies because customers are not willing to pay for them.
  - rapind 7y ago
    Making the penalties for breaches far more severe would be a good place to start though.
  - CydeWeys 7y ago
    Twitter is big enough to pay that cost. Easily.
- verletx64 7y ago
  Twitter has a lot of staff. Even 1% is a relatively large amount of people.
  - RhysU 7y ago
    Moreover, anyone seeking privileged access will percolate into such roles.
  - ComputerGuru 7y ago
    (Just as an FYI, the last time I checked that number was staggeringly skewed towards sales/marketing/evangelism teams rather than coders. Not that your point is diminished.)
    - jacobsenscott 7y ago
      Those people probably have more access to PII than staff engineers - that's who the data is for.
      - ComputerGuru 7y ago
        Hardly. They get the filtered data that SEs designed for them to get, probably on an account-by-account basis. It’s the raw data SEs have access to that we are mainly talking about here.
  - CydeWeys 7y ago
    It doesn't need to be 1% of staff, it just needs to be a few dozen people (distributed geographically across the globe) who have root access.
journalctl 7y ago
Maybe because tech companies tend to be careless with data. It isn’t NP-complete to track and monitor access to prod data, or prevent access to it entirely. It’s just that people don’t want to do it.
- closeparen 7y ago
  It is pretty damn hard to reliably operate a service that you can’t introspect or debug in any way. The real world is more creative than you; things will happen in production that you didn’t think of in your synthetic test fixtures.
  - journalctl 7y ago
    That’s still no excuse. Maybe Bill has to access some customer data to troubleshoot. Fine, log it. Maybe Susan has to look at sensitive logs to fix a bug. Fine, log it. These aren’t new or unsolvable problems.
    - closeparen 7y ago
      Who says there weren’t logs? They got caught.
- d1zzy 7y ago
  Please back this up and define the terms used. As in, define "tech companies" (name specific companies that do this and why you think that can be generalized to the entire industry) and define "careless" (it's a relative term so please say if it's less or more careless of other examples of organizations that manage similarly large amounts of data but that aren't "tech companies"). Because to me the opposite seems true. It's the non-tech companies (if you equate that to FAANG) that manage large amounts of data that tend to have a lot more data leaks (internal or external) than the tech companies.
nostrademons 7y ago
"Software engineers on an on-call rotation...data analysts, maybe even some external contractors" These groups usually get access through a bastion that anonymizes data and logs access. I remember that as a SWE at Google, I could run aggregated & anonymized statistics across query logs, but some info (eg. IPs, user logins) had been scrubbed before any of my code could get access to it, and for things that were more personal (eg. your GMail login) you could only get access to your own account. There's nothing you can do about SREs who have root access on the box or the SWEs who need to implement & maintain the bastion servers, but that's presumably a more restricted, vetted, and trusted group.
- remarkEon 7y ago
  Would it be smart for companies like google and twitter to publish how they vet people for these roles? That’s a seriously tremendous amount of power. I almost think it needs to be regulated like heath info. I get that that would make it more expensive to handle this data at all, and would serve as a significant barrier to entry for startups ... but data breach after data breach is pushing me in the regulation direction.
  - C1sc0cat 7y ago
    The same way defences industries do I would imagine for SF86 etc
  - retrovm 7y ago
    Pretty much nobody at Google has that kind of access. You can be an SRE of just about anything at Google and never access user data. The "break-glass" means of emergency access is ridiculously booby-trapped. A person wanting to do this thing has to 1) badge into a special room, at which time both production security and privacy incident teams are notified, 2) use a special hardware security device that is used for no other purpose than to activate a VPN box with a hard-line into the production network. By the way if a random Googler just rolls up to a datacenter without a reason to be there, that also triggers privacy incident response, even though physical access to production storage is virtually useless due to all the encryption. I would say it is much more likely that Google will accidentally lose the organizational ability to become root-in-prod, than it is that a person has done this thing without being noticed. In short, insider risk cannot be mitigated with hiring practices. You need robust technical measures against insider risk.
JohnFen 7y ago
> in a tech company there's always going to be a significant number of people with direct access to the raw data. Why? I've worked for a few large tech companies that handled very sensitive customer data, and they didn't allow unsanitized access to it by a significant number of people. Typically (on the dev side, anyway), there was a small designated team (less than 10 people) who were the only ones who had such access. Any dev work that absolutely required access to that data -- which was very rare -- was performed by that team.
- deleted 7y ago
  [deleted]
- icedchai 7y ago
  It's not like this at smaller companies. It's anything goes. I worked at one place that had the development network permanently VPN'd into prod. One day, a developer accidentally configured his local environment to connect to a production queue and database. It was like this for over a week. A previous company didn't bother with the VPN. They had an AWS environment that predated VPC, so SSH and many other service ports were open to the office IP addresses. And several people's homes, for remote work.
  - JohnFen 7y ago
    > It's not like this at smaller companies. It's anything goes. It depends on the company (as with large ones, apparently). I currently work for a small company, and it is no less diligent about this stuff than the major companies I've worked for.
- justinclift 7y ago
  Those large tech companies probably had a team or teams of people whose job is to look after the backups for production servers. Backups generally have "god mode" access (best description) as they need to backup and restore not just filesystem data, but the audit log data as well. Most (corp) places I worked, the developers and SysAdmin's working on production servers gave little thought to the backup component apart from making sure the software is installs and runs. ;)

ForHackernews 7y ago

You can significantly reduce these problems by decentralizing data and moving away from giant platforms that make such enticing targets for espionage.

CydeWeys 7y ago

"but in a tech company there's always going to be a significant number of people with direct access to the raw data" This is absolutely not true. This number can and should be reduced to an absolute minimum number of people.

bmiller2 7y ago
The absolute minimum number of people may still be significant
tristanstcyr 7y ago
Sure, but it comes at a cost. Companies rarely push the limits of these kinds of policies because customers are not willing to pay for them.
- rapind 7y ago
  Making the penalties for breaches far more severe would be a good place to start though.
- CydeWeys 7y ago
  Twitter is big enough to pay that cost. Easily.
verletx64 7y ago
Twitter has a lot of staff. Even 1% is a relatively large amount of people.
- RhysU 7y ago
  Moreover, anyone seeking privileged access will percolate into such roles.
- ComputerGuru 7y ago
  (Just as an FYI, the last time I checked that number was staggeringly skewed towards sales/marketing/evangelism teams rather than coders. Not that your point is diminished.)
  - jacobsenscott 7y ago
    Those people probably have more access to PII than staff engineers - that's who the data is for.
    - ComputerGuru 7y ago
      Hardly. They get the filtered data that SEs designed for them to get, probably on an account-by-account basis. It’s the raw data SEs have access to that we are mainly talking about here.
- CydeWeys 7y ago
  It doesn't need to be 1% of staff, it just needs to be a few dozen people (distributed geographically across the globe) who have root access.

journalctl 7y ago

Maybe because tech companies tend to be careless with data. It isn’t NP-complete to track and monitor access to prod data, or prevent access to it entirely. It’s just that people don’t want to do it.

closeparen 7y ago
It is pretty damn hard to reliably operate a service that you can’t introspect or debug in any way. The real world is more creative than you; things will happen in production that you didn’t think of in your synthetic test fixtures.
- journalctl 7y ago
  That’s still no excuse. Maybe Bill has to access some customer data to troubleshoot. Fine, log it. Maybe Susan has to look at sensitive logs to fix a bug. Fine, log it. These aren’t new or unsolvable problems.
  - closeparen 7y ago
    Who says there weren’t logs? They got caught.
d1zzy 7y ago
Please back this up and define the terms used. As in, define "tech companies" (name specific companies that do this and why you think that can be generalized to the entire industry) and define "careless" (it's a relative term so please say if it's less or more careless of other examples of organizations that manage similarly large amounts of data but that aren't "tech companies"). Because to me the opposite seems true. It's the non-tech companies (if you equate that to FAANG) that manage large amounts of data that tend to have a lot more data leaks (internal or external) than the tech companies.

nostrademons 7y ago

"Software engineers on an on-call rotation...data analysts, maybe even some external contractors" These groups usually get access through a bastion that anonymizes data and logs access. I remember that as a SWE at Google, I could run aggregated & anonymized statistics across query logs, but some info (eg. IPs, user logins) had been scrubbed before any of my code could get access to it, and for things that were more personal (eg. your GMail login) you could only get access to your own account. There's nothing you can do about SREs who have root access on the box or the SWEs who need to implement & maintain the bastion servers, but that's presumably a more restricted, vetted, and trusted group.

remarkEon 7y ago
Would it be smart for companies like google and twitter to publish how they vet people for these roles? That’s a seriously tremendous amount of power. I almost think it needs to be regulated like heath info. I get that that would make it more expensive to handle this data at all, and would serve as a significant barrier to entry for startups ... but data breach after data breach is pushing me in the regulation direction.
- C1sc0cat 7y ago
  The same way defences industries do I would imagine for SF86 etc
- retrovm 7y ago
  Pretty much nobody at Google has that kind of access. You can be an SRE of just about anything at Google and never access user data. The "break-glass" means of emergency access is ridiculously booby-trapped. A person wanting to do this thing has to 1) badge into a special room, at which time both production security and privacy incident teams are notified, 2) use a special hardware security device that is used for no other purpose than to activate a VPN box with a hard-line into the production network. By the way if a random Googler just rolls up to a datacenter without a reason to be there, that also triggers privacy incident response, even though physical access to production storage is virtually useless due to all the encryption. I would say it is much more likely that Google will accidentally lose the organizational ability to become root-in-prod, than it is that a person has done this thing without being noticed. In short, insider risk cannot be mitigated with hiring practices. You need robust technical measures against insider risk.

JohnFen 7y ago

> in a tech company there's always going to be a significant number of people with direct access to the raw data. Why? I've worked for a few large tech companies that handled very sensitive customer data, and they didn't allow unsanitized access to it by a significant number of people. Typically (on the dev side, anyway), there was a small designated team (less than 10 people) who were the only ones who had such access. Any dev work that absolutely required access to that data -- which was very rare -- was performed by that team.

deleted 7y ago
[deleted]
icedchai 7y ago
It's not like this at smaller companies. It's anything goes. I worked at one place that had the development network permanently VPN'd into prod. One day, a developer accidentally configured his local environment to connect to a production queue and database. It was like this for over a week. A previous company didn't bother with the VPN. They had an AWS environment that predated VPC, so SSH and many other service ports were open to the office IP addresses. And several people's homes, for remote work.
- JohnFen 7y ago
  > It's not like this at smaller companies. It's anything goes. It depends on the company (as with large ones, apparently). I currently work for a small company, and it is no less diligent about this stuff than the major companies I've worked for.
justinclift 7y ago
Those large tech companies probably had a team or teams of people whose job is to look after the backups for production servers. Backups generally have "god mode" access (best description) as they need to backup and restore not just filesystem data, but the audit log data as well. Most (corp) places I worked, the developers and SysAdmin's working on production servers gave little thought to the backup component apart from making sure the software is installs and runs. ;)