4 ms·

Why are you committing PII to git in the first place?

paranoidrobot 5y ago
They're not, they're 'just' letting employees use real customer PII when training. They screwed up by allowing that. The employee screwed up by committing it to git and then pushing to a public repo. The employee wouldn't have been able to do that if they'd enforced using fake customer data for testing/training.
- shadowgovt 5y ago
  The unfortunate thing about startups is that a lot of them are this fast and loose with PII. Incentives are low to do better, and the tools that make it easy to do better cost money. This isn't to excuse Adafruit; it's to remind everyone that the hot young startup you just signed up for is probably keeping your signup information in a mysql database that everyone in the company has access to right now with a plaintext password thumb-tacked to the one office wall they have.
  - l30n4da5 5y ago
    > The unfortunate thing about startups is that a lot of them are this fast and loose with PII Unless it is a HIPAA-compliant startup. Then scrubbing PII is priority number 1.
    - earleybird 5y ago
      I know of business units IN government who explicitly ignore compliance. They sign off claiming they "accept the risk". It's worth your job if you push compliance too hard with them.
      - l30n4da5 5y ago
        I thankfully have never had the displeasure of working with a business unit that explicitly ignored compliance with regulations. i know they exist. i just havent, personally, had that experience, thankfully.
  - josephg 5y ago
    Yep. And there’s a good chance the company’s production database is on employee laptops so they can test features with real data sets. When a stranger has their laptop stolen on a bus, who knows what data was on it. Fingers crossed most people have FileVault turned on these days.
- earleybird 5y ago
  When you try to tell a business unit they can't use live/prod data then whine to their director who complains to their deputy minister, at which point the hailstorm of shit turns around and starts falling down on those of us who are "blockers". Don't get me started on "the business signs off on accepting the risk".
jrochkind1 5y ago
I would say not just why are you committing it to git, but why are you using it for "employee training"? You should not be using real data with PII for training exersizes. Hopefully that's what they mean by: > We are additionally putting in place more protocols and access controls to avoid any possible future data exposure and limiting access for employee training use.
CSSer 5y ago
What mechanisms are in place to prevent an employee from doing so? We have so many people in the industry now that it's hard to keep up with this stuff. To be fair, Github has all the ingredients necessary to be a giant, gaping security liability. It's a "secure" platform built around software designed for sharing. That sentence alone has a ton of baggage in it. I'm not even trying to diss Github either.
- layer8 5y ago
  Education/training is really the only way.
JulianWasTaken 5y ago
Lots of data scientists, ex-statisticians, analysts, etc. are pressured with "you should use git". It's generally a noble thing, and the right one -- these folks often otherwise don't use git enough for their code. It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed. I'm not saying what happened here is excusable -- it's not, it's ridiculously bad. But I can see a few ways how it'd happen. To me the equal or better question is why does this person have PII in the first place? It's not necessary to do analysis. Someone should have masked or removed it before this person ever got their hands on it. There's no way a PII dataset is needed for training.
- shadowgovt 5y ago
  Correct. This is a common failure mode... The easiest way to build a training set for a system is to take some live data, salami slice it, and store it in a static location. ... that's also a short path to dropping a user's PII in the clear.
- bborud 5y ago
  > these folks often otherwise don't use git enough for their code. Or at all. This is actually a fairly common problem among certain types of research scientists who think software engineering is "peasant work". I remember a friend of mine working at a research organization where several researchers lost months of work due to an unscheduled server reboot. Turns out when you log into ephemeral containers and pretend they are VMs, things go poof.
  - mynameisvlad 5y ago
    I read a post on Reddit's legal advice subreddit of a data scientist having their apartment robbed, and part of it was saying they'd lost all their code and currently working on projects. As horrible as the robbery was, I was screaming inside "WHAT ABOUT VERSION CONTROL?!??!"
    - 999900000999 5y ago
      That’s a really sad story. If he doesn’t come from a programming background version can seem foreign. But like ya know… Dropbox. I’d be more concerned with my hard drive corrupting than anything else
dreamcompiler 5y ago
There's nothing particularly wrong with using git for PII, as long as you use it internally on private machines or a private LAN. But it's a bad idea to put PII on github which is apparently what happened.
- dbjorge 5y ago
  I think that even just on private machines, this would make some types of legal compliance needlessly difficult. If you ever need to delete that data, for example to comply with a corporate retention policy or in response to a request from an individual in a jurisdiction that requires you allow doing so, you would need not just to rewrite history but also to ensure that history is rewritten in every clone that any employee has ever made of that repository; there might not even be a record of which clones exist.
  - dreamcompiler 5y ago
    The clone issue is real, but it's no more complicated than it would be without git: Keep track of who has a copy of this data and where. Whatever system you have in place to solve that problem can also be used to manage the cloning process. As for the individual repos, here's the procedure: cd <repo> rm -rf .git/ # DON'T TRY THIS AT HOME unless you know what you're doing. git add . git commit -m "Brand new repo" Do not do the above unless you completely understand the ramifications. But those ramifications might be precisely the ones you want for legal compliance.
    - guipsp 5y ago
      Might as well not use git then
    - JulianWasTaken 5y ago
      Please never do this -- not least of which because each person who does it will end up with now completely divergent histories and repos from each other. There are various tools to properly filter history, e.g. `git-filter-repo`, but the short answer is as the grandparent commenter said, things get hairy, you need to rewrite history and coordinate. It's not a situation you should hope to get yourself into...
Fice 5y ago
Using git is no big deal, but storing the repo on Github means sharing the information with a third party (Microsoft) even if the repo is kept "private".

They're not, they're 'just' letting employees use real customer PII when training. They screwed up by allowing that. The employee screwed up by committing it to git and then pushing to a public repo. The employee wouldn't have been able to do that if they'd enforced using fake customer data for testing/training.

shadowgovt 5y ago
The unfortunate thing about startups is that a lot of them are this fast and loose with PII. Incentives are low to do better, and the tools that make it easy to do better cost money. This isn't to excuse Adafruit; it's to remind everyone that the hot young startup you just signed up for is probably keeping your signup information in a mysql database that everyone in the company has access to right now with a plaintext password thumb-tacked to the one office wall they have.
- l30n4da5 5y ago
  > The unfortunate thing about startups is that a lot of them are this fast and loose with PII Unless it is a HIPAA-compliant startup. Then scrubbing PII is priority number 1.
  - earleybird 5y ago
    I know of business units IN government who explicitly ignore compliance. They sign off claiming they "accept the risk". It's worth your job if you push compliance too hard with them.
    - l30n4da5 5y ago
      I thankfully have never had the displeasure of working with a business unit that explicitly ignored compliance with regulations. i know they exist. i just havent, personally, had that experience, thankfully.
- josephg 5y ago
  Yep. And there’s a good chance the company’s production database is on employee laptops so they can test features with real data sets. When a stranger has their laptop stolen on a bus, who knows what data was on it. Fingers crossed most people have FileVault turned on these days.
earleybird 5y ago
When you try to tell a business unit they can't use live/prod data then whine to their director who complains to their deputy minister, at which point the hailstorm of shit turns around and starts falling down on those of us who are "blockers". Don't get me started on "the business signs off on accepting the risk".

jrochkind1 5y ago

I would say not just why are you committing it to git, but why are you using it for "employee training"? You should not be using real data with PII for training exersizes. Hopefully that's what they mean by: > We are additionally putting in place more protocols and access controls to avoid any possible future data exposure and limiting access for employee training use.

CSSer 5y ago

What mechanisms are in place to prevent an employee from doing so? We have so many people in the industry now that it's hard to keep up with this stuff. To be fair, Github has all the ingredients necessary to be a giant, gaping security liability. It's a "secure" platform built around software designed for sharing. That sentence alone has a ton of baggage in it. I'm not even trying to diss Github either.

layer8 5y ago
Education/training is really the only way.

JulianWasTaken 5y ago

Lots of data scientists, ex-statisticians, analysts, etc. are pressured with "you should use git". It's generally a noble thing, and the right one -- these folks often otherwise don't use git enough for their code. It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed. I'm not saying what happened here is excusable -- it's not, it's ridiculously bad. But I can see a few ways how it'd happen. To me the equal or better question is why does this person have PII in the first place? It's not necessary to do analysis. Someone should have masked or removed it before this person ever got their hands on it. There's no way a PII dataset is needed for training.

shadowgovt 5y ago
Correct. This is a common failure mode... The easiest way to build a training set for a system is to take some live data, salami slice it, and store it in a static location. ... that's also a short path to dropping a user's PII in the clear.
bborud 5y ago
> these folks often otherwise don't use git enough for their code. Or at all. This is actually a fairly common problem among certain types of research scientists who think software engineering is "peasant work". I remember a friend of mine working at a research organization where several researchers lost months of work due to an unscheduled server reboot. Turns out when you log into ephemeral containers and pretend they are VMs, things go poof.
- mynameisvlad 5y ago
  I read a post on Reddit's legal advice subreddit of a data scientist having their apartment robbed, and part of it was saying they'd lost all their code and currently working on projects. As horrible as the robbery was, I was screaming inside "WHAT ABOUT VERSION CONTROL?!??!"
  - 999900000999 5y ago
    That’s a really sad story. If he doesn’t come from a programming background version can seem foreign. But like ya know… Dropbox. I’d be more concerned with my hard drive corrupting than anything else

dreamcompiler 5y ago

There's nothing particularly wrong with using git for PII, as long as you use it internally on private machines or a private LAN. But it's a bad idea to put PII on github which is apparently what happened.

dbjorge 5y ago
I think that even just on private machines, this would make some types of legal compliance needlessly difficult. If you ever need to delete that data, for example to comply with a corporate retention policy or in response to a request from an individual in a jurisdiction that requires you allow doing so, you would need not just to rewrite history but also to ensure that history is rewritten in every clone that any employee has ever made of that repository; there might not even be a record of which clones exist.
- dreamcompiler 5y ago
  The clone issue is real, but it's no more complicated than it would be without git: Keep track of who has a copy of this data and where. Whatever system you have in place to solve that problem can also be used to manage the cloning process. As for the individual repos, here's the procedure: cd <repo> rm -rf .git/ # DON'T TRY THIS AT HOME unless you know what you're doing. git add . git commit -m "Brand new repo" Do not do the above unless you completely understand the ramifications. But those ramifications might be precisely the ones you want for legal compliance.
  - guipsp 5y ago
    Might as well not use git then
  - JulianWasTaken 5y ago
    Please never do this -- not least of which because each person who does it will end up with now completely divergent histories and repos from each other. There are various tools to properly filter history, e.g. `git-filter-repo`, but the short answer is as the grandparent commenter said, things get hairy, you need to rewrite history and coordinate. It's not a situation you should hope to get yourself into...

Fice 5y ago

Using git is no big deal, but storing the repo on Github means sharing the information with a third party (Microsoft) even if the repo is kept "private".