10 ms·

A GitHub repository was public-viewable

179 points by zdw 5y ago

999900000999 5y ago
Why are you committing PII to git in the first place?
- paranoidrobot 5y ago
  They're not, they're 'just' letting employees use real customer PII when training. They screwed up by allowing that. The employee screwed up by committing it to git and then pushing to a public repo. The employee wouldn't have been able to do that if they'd enforced using fake customer data for testing/training.
  - shadowgovt 5y ago
    The unfortunate thing about startups is that a lot of them are this fast and loose with PII. Incentives are low to do better, and the tools that make it easy to do better cost money. This isn't to excuse Adafruit; it's to remind everyone that the hot young startup you just signed up for is probably keeping your signup information in a mysql database that everyone in the company has access to right now with a plaintext password thumb-tacked to the one office wall they have.
    - l30n4da5 5y ago
      > The unfortunate thing about startups is that a lot of them are this fast and loose with PII Unless it is a HIPAA-compliant startup. Then scrubbing PII is priority number 1.
      - earleybird 5y ago
        I know of business units IN government who explicitly ignore compliance. They sign off claiming they "accept the risk". It's worth your job if you push compliance too hard with them.
        l30n4da5 5y ago
        I thankfully have never had the displeasure of working with a business unit that explicitly ignored compliance with regulations. i know they exist. i just havent, personally, had that experience, thankfully.
    - josephg 5y ago
      Yep. And there’s a good chance the company’s production database is on employee laptops so they can test features with real data sets. When a stranger has their laptop stolen on a bus, who knows what data was on it. Fingers crossed most people have FileVault turned on these days.
  - earleybird 5y ago
    When you try to tell a business unit they can't use live/prod data then whine to their director who complains to their deputy minister, at which point the hailstorm of shit turns around and starts falling down on those of us who are "blockers". Don't get me started on "the business signs off on accepting the risk".
- jrochkind1 5y ago
  I would say not just why are you committing it to git, but why are you using it for "employee training"? You should not be using real data with PII for training exersizes. Hopefully that's what they mean by: > We are additionally putting in place more protocols and access controls to avoid any possible future data exposure and limiting access for employee training use.
- CSSer 5y ago
  What mechanisms are in place to prevent an employee from doing so? We have so many people in the industry now that it's hard to keep up with this stuff. To be fair, Github has all the ingredients necessary to be a giant, gaping security liability. It's a "secure" platform built around software designed for sharing. That sentence alone has a ton of baggage in it. I'm not even trying to diss Github either.
  - layer8 5y ago
    Education/training is really the only way.
- JulianWasTaken 5y ago
  Lots of data scientists, ex-statisticians, analysts, etc. are pressured with "you should use git". It's generally a noble thing, and the right one -- these folks often otherwise don't use git enough for their code. It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed. I'm not saying what happened here is excusable -- it's not, it's ridiculously bad. But I can see a few ways how it'd happen. To me the equal or better question is why does this person have PII in the first place? It's not necessary to do analysis. Someone should have masked or removed it before this person ever got their hands on it. There's no way a PII dataset is needed for training.
  - shadowgovt 5y ago
    Correct. This is a common failure mode... The easiest way to build a training set for a system is to take some live data, salami slice it, and store it in a static location. ... that's also a short path to dropping a user's PII in the clear.
  - bborud 5y ago
    > these folks often otherwise don't use git enough for their code. Or at all. This is actually a fairly common problem among certain types of research scientists who think software engineering is "peasant work". I remember a friend of mine working at a research organization where several researchers lost months of work due to an unscheduled server reboot. Turns out when you log into ephemeral containers and pretend they are VMs, things go poof.
    - mynameisvlad 5y ago
      I read a post on Reddit's legal advice subreddit of a data scientist having their apartment robbed, and part of it was saying they'd lost all their code and currently working on projects. As horrible as the robbery was, I was screaming inside "WHAT ABOUT VERSION CONTROL?!??!"
      - 999900000999 5y ago
        That’s a really sad story. If he doesn’t come from a programming background version can seem foreign. But like ya know… Dropbox. I’d be more concerned with my hard drive corrupting than anything else
- dreamcompiler 5y ago
  There's nothing particularly wrong with using git for PII, as long as you use it internally on private machines or a private LAN. But it's a bad idea to put PII on github which is apparently what happened.
  - dbjorge 5y ago
    I think that even just on private machines, this would make some types of legal compliance needlessly difficult. If you ever need to delete that data, for example to comply with a corporate retention policy or in response to a request from an individual in a jurisdiction that requires you allow doing so, you would need not just to rewrite history but also to ensure that history is rewritten in every clone that any employee has ever made of that repository; there might not even be a record of which clones exist.
    - dreamcompiler 5y ago
      The clone issue is real, but it's no more complicated than it would be without git: Keep track of who has a copy of this data and where. Whatever system you have in place to solve that problem can also be used to manage the cloning process. As for the individual repos, here's the procedure: cd <repo> rm -rf .git/ # DON'T TRY THIS AT HOME unless you know what you're doing. git add . git commit -m "Brand new repo" Do not do the above unless you completely understand the ramifications. But those ramifications might be precisely the ones you want for legal compliance.
      - guipsp 5y ago
        Might as well not use git then
      - JulianWasTaken 5y ago
        Please never do this -- not least of which because each person who does it will end up with now completely divergent histories and repos from each other. There are various tools to properly filter history, e.g. `git-filter-repo`, but the short answer is as the grandparent commenter said, things get hairy, you need to rewrite history and coordinate. It's not a situation you should hope to get yourself into...
- Fice 5y ago
  Using git is no big deal, but storing the repo on Github means sharing the information with a third party (Microsoft) even if the repo is kept "private".
geewee 5y ago
Oof, committing PII to Git. So much for that "right to deletion" unless you want to rewrite your entire git history.
- that_guy_iain 5y ago
  My understanding is the right to deletion doesn't so much affect things such as logs, internal training data, etc. Also, since that's a web store and the data comes from customer orders they would have no right to deletion as that information becomes protected for other legal requirements.
  - TedDoesntTalk 5y ago
    Are they based in a jurisdiction that has right to deletion laws?
  - deleted 5y ago
    [deleted]
  - peterjuras 5y ago
    From a GDPR perspective, it also covers logs, internal training data etc. If a user requires to be deleted, you have to delete everything, there can be no trace of their existence.
    - shadowgovt 5y ago
      There's a reason many of us across the ocean look at that part of the GDPR like someone decided you could put the feathers back in the pillowcase if you just made it illegal for the feathers to be outside the pillowcase.
    - that_guy_iain 5y ago
      GDPR, from what the legal trainings I've had, logs aren't covered if you apply the technical requirement and costs, etc.
      - 5y ago
xd 5y ago
PII data should never touch github, ever. Good job you're not in the EU with GDPR as this process alone implies a serious lack of security due diligence. "Adafruit team began the forensic process" - Checking audit logs is hardly a forensic process and the rest of the speel about "privacy lawyers and legal experts" feels like a poor effort to regain some trust.
- andi999 5y ago
  Why shdnt it touch github? If you think it is not secure than also your companies source code shdnt touch it, shouldnt it? Or is there a different reason?
  - sdesol 5y ago
    Trade secret and PII are two different things. Trade secret is something you can risk. PII is not something most governments will allow you to risk.
  - CSSer 5y ago
    You never know where that repo is going to go. It could be private today and public tomorrow with a totally different license. Git should be thought of as sticky in the sense that it's intentionally designed to preserve a history of everything for all time in the most efficient way possible. Anyone who's ever had to scrub an API key they accidentally pushed or optimize a monorepo can attest to this.
    - MattRix 5y ago
      If it’s your own repo presumably you know exactly where that repo is going to go. The vast majority of private repos are never going to go public. I have many repos for projects that there is no chance I would ever make public, they’re not that kind of project. And even if I really did want them to be public, I would make a new repo for them instead.
      - CSSer 5y ago
ccooffee 5y ago
> Why aren’t we sending an email to every user? > We evaluated the risk and consulted with our privacy lawyers and legal experts, and took the approach that we thought appropriately mitigated any issues while being open and transparent and did not believe emailing directly was helpful in this case. Seems like a pretty weak justification. "Our lawyers said we don't have to notify users."
- dangerface 5y ago
  Normally you would email users notifying them of a hack so they can change passwords since there was no password breach it doesn't make a lot of sense to email people, theres nothing for them to do.
  - Hamuko 5y ago
    If you are a customer who might start receiving death threats in the mail because someone leaked your email address and home address at the same time, there might be things that they need to do.
- deleted 5y ago
  [deleted]
- devwastaken 5y ago
  Yep. Lawyers will always take the road of least liability. I have no doubt the lawyer said to not even write about it publicly. I stopped trusting adafruit when they sent me broken boards and never replaced them. Their software side is very startupy, they love to break things, especially if it was working perfectly fine before. I have a USB to uart/SPI/i2c adafruit device that won't operate in any of those modes without changing windows drivers between them, and ontop of it requires local side circuit python and it used to just be C. Circuit python was a huge mistake on their part. They didn't have the people to do C correctly (look at their nrf libraries), much less C and circuit python versions. Which ends up being a compatability nightmare. So I just buy a 5 pack of esp32's or other dev boards off Amazon and be done with it. If they're broke, I get them for free. Sad to see that Amazon breaks less consumer laws than Adafruit in this case.
tmhrtly 5y ago
I wonder how many users. They don’t say, just use the word ‘some’ which makes me wonder if it’s quite a few.
dangerface 5y ago
The issue is the information was made public I don't know why people are upset about it being stored in git. If I store PII in a database and then that database gets leaked am I dumb for storing my PII in a database or for letting it get leaked? Seems kinda obvious to me.
- kahrl 5y ago
  People are upset that it is being stored in git because it shows how moronically negligent this company is about the security of its customer data. Storing any secrets in version control is one of the stupidest things you can do, and the fact that this was allowed to happen reflects poorly on the entire company.
- NovemberWhiskey 5y ago
  If you really, really want to manage PII in a private git repo, you can probably do it - in approximately the same that you can probably make API-first web services in FORTRAN if you really, really want to. The fact remains that your choice of tooling hurts you. Putting PII into Github (even if the repo is private) is catastrophic. You just made Github into a third-party data processor by accident. Good luck explaining that from a CCPA/GDPR perspective.
- selfup 5y ago
  The idea is that git is for source code/configs. Not data. Data gets stored in databases. Secrets/Tokens are stored in secret stores/vaults. Blobs like PDFs/etc should be stored in blob storage. If a diff is not useful, don't store it in git. Doing a code review on PII is not useful. You also shouldn't add packages in a git repo if they can be downloaded immutably in CI/locally. It adds a lot of space. Every commit is a snapshot + new code. Commits are not diffs.
  - iudqnolq 5y ago
    I disagree. They were using it as sample data, when they should have used synthetic sample data. It's totally sane to check-in a JSON (or similar) file with a few hundred [edit: non-confidential] samples so that you can run integration tests without needing to set anything else up. Similarly, I keep small pdf manuals in git. I add them in their own commit, which doesn't have a useful diff. In exchange, they're always there and I don't need to spin up some special one-off system.
WYepQ4dNnG 5y ago
You should at least consider encrypting the PII on git, https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt it is not rocket sience
- deleted 5y ago
  [deleted]
- kahrl 5y ago
  You should consider never ever ever ever ever ever storing secrets in version control. Period.
  - morelisp 5y ago
    Storing encrypted secrets in SCM is pretty standard practice. Usually I recommend separate repos but it's not required especially if the secret material is small and rarely changes. This data wasn't "secrets" though. PD treated this loosely would still be an issue even if it was encrypted.
    - lmeyerov 5y ago
      Standard doesn't mean good, and the sophistication to do this well is beyond most orgs and individuals . You can come up with specific scenarios, but they'll get garbled through the telephone game over time. Much better to treat data, including secrets, as radioactive, and thus follow least priv + defense in depth. If it's never there, and better, they never got it, there's nothing to worry about Ex: Before PII hits logs, encrypt it and don't give data scientists the key. Ex: orgs we work with who version notebooks won't allow notebook output to be saved, and people know (+ software) to reject baked creds. Likewise, auth isn't baked generally, instead SSO. If it's never there, for every stage, so much easier.
      - morelisp 5y ago
        No, storing encrypted secrets in repositories is an incredibly common and safe practice in DevOps/GitOps environments. Virtually every orchestration tool has some way to support it. E.g. k8s sealed secrets https://github.com/bitnami-labs/sealed-secrets https://github.com/bitnami-labs/sealed-secrets E.g. Salt's GPG filters https://docs.saltproject.io/en/latest/ref/renderers/all/salt.renderers.gpg.html https://docs.saltproject.io/en/latest/ref/renderers/all/salt... E.g. Ansible Vault https://docs.ansible.com/ansible/latest/user_guide/vault.html https://docs.ansible.com/ansible/latest/user_guide/vault.htm... E.g. Puppet hiera-yaml https://puppet.com/docs/puppet/6/securing-sensitive-data.html https://puppet.com/docs/puppet/6/securing-sensitive-data.htm... PD/PII is a completely separate issue. First because even encrypting doesn't remove legal obligations concerning processing, and second because your DS/BI teams probably need access to the unencrypted data to like, do their actual jobs. You need completely orthogonal forms of access control for that (like SSO as you alude to).
globular-toast 5y ago
Does Github delete blobs when a repository is deleted? I doubt it.
- niros_valtos 5y ago
  Your doubt is right. Here is how repos can be restored: https://docs.github.com/en/repositories/creating-and-managing-repositories/restoring-a-deleted-repository https://docs.github.com/en/repositories/creating-and-managin...
niros_valtos 5y ago
Storing PII on GitHub is not wise. It is stored with the history and can be restored even if deleted. As for opening up the repo publicly, it can be achieved with a quick win by triggering alerts on such changes.
- deleted 5y ago
  [deleted]
kaycebasques 5y ago
Classic corporate use of passive voice to deflect responsibility.
- jcrawfordor 5y ago
  Honestly, just the title of this post has caused me to lose some trust in Adafruit as a company. It's so completely passive and doesn't explain what actually happened. It feels like it was intentionally written to avoid attracting attention, but then attracts attention because of how vague and weird it is.
  - kaycebasques 5y ago
    The Streisand Effect! https://en.m.wikipedia.org/wiki/Streisand_effect https://en.m.wikipedia.org/wiki/Streisand_effect
- tomcatfish 5y ago
  What are they supposed to do, say "Jeff the intern fucked up badly, and Christine let them do it"? What is this criticism supposed to mean?
  - hnaccount141 5y ago
    It's not about singling out individuals, it's about Adafruit as a company taking responsibility for their mistakes. The phrasing in the post makes it sound like the breach was something that happened to them, vs being the direct result of the company's actions.
WhompingWindows 5y ago
I've been in data analysis for a few years -- grew from scientific research in SAS with zero VCS, into programming and data analysis in R and STATA, still no VCS for a while...eventually I was allowed to use Github after I left my restrictive hospital research space. No one ever sat me down and trained me on Git, though. I just heard repeatedly: if you're not using Git, you're missing out on a huge set of important software-creation tools, you shouldn't be coding without VCS. Why did this happen? Imagine it's this data analysts' first year out of school, something like a crappy statistics program that only teaches SAS or base-R with no VCS. This young padawan needs to practice analysis in preparation for a work project. They grab some data and stash it in a folder where Git is tracking. They do their analysis practice, it's Friday afternoon, they get lazy, they don't look at what they're committing, and they click buttons in the GUI without much thought. It's incompetence and laziness, not malice. These tools that allow us to share widely, not just GH but also social media broadly: these tools have great power that should be used with greater training, responsibility, and care.
- duped 5y ago
  I don't think it's incompetence so much as professional ignorance (which might be incompetence, I don't think it is). Source control is for your source code, not for your data. Your data belongs in a data base. Git is not a database, or at least shouldn't be treated like one. Sure it's easy to call it lazy for a data set to be in some local directory and accidentally get committed. Happens to all of us. The bigger problem is "why is that data sitting on your file system in a directory, when it should be in some data base, preferably not locally." > these tools have great power that should be used with greater training, responsibility, and care. This screams more and more that the tools are bad. Git is famously hard to use and even harder for non-plaintext data. Databases are annoying to initialize and get access to without a developer who's done it before. The tools suck, they can be better, and require less training. It's not wrong to be lazy - it's wrong to make the lazy path dangerous.
  - Karellen 5y ago
    > Git is not a database Um, yeah, it is - by most reasonable definitions of the word "database". No doubt there are a few unusually-narrow definitions of "database" out there that would exclude Git, but I'm pretty certain they're in the minority.
kahrl 5y ago
Adafruit's response to this is appalling. Storing secrets in version control is one of the stupidest things you can do, and reflects poorly on the entire company. But the fact that they are choosing to not inform their customers that their PII was leaked due to their negligence is criminal (well, in certain jurisdictions). Shame on Adafruit. Hope they get sued out of existence.
- apgwoz 5y ago
  > Shame on Adafruit. Hope they get sued out of existence. They made a mistake. The response to the mistake is NOT GOOD. However, destroying a company that also does actually good things is not the answer here, either. How should they fix this? I think they should start be writing another blog post apologizing for how bad the first one was, for starters. Emails to affected people next... But, let's be honest. How much better is _that_? Can't unleak the data. You can only be offered subscriptions to data protection services _sooo many times_. There's not really a monetary value as damages here...
dzhiurgis 5y ago
Office Depot Salesforce source code still up. Complete with some integration private keys. Bug bounty says it’s not a bug… Salesforce employee recently published source of one of their products. I’ve reported via email since I’ve been removed from their private Hackerone programme, presumably due inactivity. Sec team just said it was “test data”. Wish I’ve made a copy since it’s gone now and bullshit like this responses just wants me published everywhere.
- dzhiurgis 5y ago
  Ok, I'm giving some credit to company above. Someone contacted me hours after this post and took down repo like minutes after I've responded (tho they say they found it in parallel). Root cause - third party was doing some POC and I'm guessing misconfigured CI.
- kaetemi 5y ago
  Hackerone is a joke, anyway. Organizations will just respond with "it's a feature, not a bug" to get out of any bounty. I once reported that you could log on to certain PP accounts with just username and CC number, bypassing configured 2FA, and allowing to wipe the 2FA. Guess the response. Lo and behold, it's fixed now.
  - throwoutway 5y ago
    Hackerone should be an escrow that can arbitrate and overrule insipid behavior like that. But they won’t, will they? Maybe someone else will fill that void
brettproctor 5y ago
For posterity, they updated the post on Mar 7 and will be emailing anyone affected. "Update March 7, 2022: We appreciate the feedback from the community and our customers, and will be emailing users as part of this disclosure. We apologize for not doing that at the same time as the post/disclosure on Friday, March 4, 2022."

179 points by zdw 5y ago

999900000999 5y ago

Why are you committing PII to git in the first place?

paranoidrobot 5y ago
They're not, they're 'just' letting employees use real customer PII when training. They screwed up by allowing that. The employee screwed up by committing it to git and then pushing to a public repo. The employee wouldn't have been able to do that if they'd enforced using fake customer data for testing/training.
- shadowgovt 5y ago
  The unfortunate thing about startups is that a lot of them are this fast and loose with PII. Incentives are low to do better, and the tools that make it easy to do better cost money. This isn't to excuse Adafruit; it's to remind everyone that the hot young startup you just signed up for is probably keeping your signup information in a mysql database that everyone in the company has access to right now with a plaintext password thumb-tacked to the one office wall they have.
  - l30n4da5 5y ago
    > The unfortunate thing about startups is that a lot of them are this fast and loose with PII Unless it is a HIPAA-compliant startup. Then scrubbing PII is priority number 1.
    - earleybird 5y ago
      I know of business units IN government who explicitly ignore compliance. They sign off claiming they "accept the risk". It's worth your job if you push compliance too hard with them.
      - l30n4da5 5y ago
        I thankfully have never had the displeasure of working with a business unit that explicitly ignored compliance with regulations. i know they exist. i just havent, personally, had that experience, thankfully.
  - josephg 5y ago
    Yep. And there’s a good chance the company’s production database is on employee laptops so they can test features with real data sets. When a stranger has their laptop stolen on a bus, who knows what data was on it. Fingers crossed most people have FileVault turned on these days.
- earleybird 5y ago
  When you try to tell a business unit they can't use live/prod data then whine to their director who complains to their deputy minister, at which point the hailstorm of shit turns around and starts falling down on those of us who are "blockers". Don't get me started on "the business signs off on accepting the risk".
jrochkind1 5y ago
I would say not just why are you committing it to git, but why are you using it for "employee training"? You should not be using real data with PII for training exersizes. Hopefully that's what they mean by: > We are additionally putting in place more protocols and access controls to avoid any possible future data exposure and limiting access for employee training use.
CSSer 5y ago
What mechanisms are in place to prevent an employee from doing so? We have so many people in the industry now that it's hard to keep up with this stuff. To be fair, Github has all the ingredients necessary to be a giant, gaping security liability. It's a "secure" platform built around software designed for sharing. That sentence alone has a ton of baggage in it. I'm not even trying to diss Github either.
- layer8 5y ago
  Education/training is really the only way.
JulianWasTaken 5y ago
Lots of data scientists, ex-statisticians, analysts, etc. are pressured with "you should use git". It's generally a noble thing, and the right one -- these folks often otherwise don't use git enough for their code. It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed. I'm not saying what happened here is excusable -- it's not, it's ridiculously bad. But I can see a few ways how it'd happen. To me the equal or better question is why does this person have PII in the first place? It's not necessary to do analysis. Someone should have masked or removed it before this person ever got their hands on it. There's no way a PII dataset is needed for training.
- shadowgovt 5y ago
  Correct. This is a common failure mode... The easiest way to build a training set for a system is to take some live data, salami slice it, and store it in a static location. ... that's also a short path to dropping a user's PII in the clear.
- bborud 5y ago
  > these folks often otherwise don't use git enough for their code. Or at all. This is actually a fairly common problem among certain types of research scientists who think software engineering is "peasant work". I remember a friend of mine working at a research organization where several researchers lost months of work due to an unscheduled server reboot. Turns out when you log into ephemeral containers and pretend they are VMs, things go poof.
  - mynameisvlad 5y ago
    I read a post on Reddit's legal advice subreddit of a data scientist having their apartment robbed, and part of it was saying they'd lost all their code and currently working on projects. As horrible as the robbery was, I was screaming inside "WHAT ABOUT VERSION CONTROL?!??!"
    - 999900000999 5y ago
      That’s a really sad story. If he doesn’t come from a programming background version can seem foreign. But like ya know… Dropbox. I’d be more concerned with my hard drive corrupting than anything else
dreamcompiler 5y ago
There's nothing particularly wrong with using git for PII, as long as you use it internally on private machines or a private LAN. But it's a bad idea to put PII on github which is apparently what happened.
- dbjorge 5y ago
  I think that even just on private machines, this would make some types of legal compliance needlessly difficult. If you ever need to delete that data, for example to comply with a corporate retention policy or in response to a request from an individual in a jurisdiction that requires you allow doing so, you would need not just to rewrite history but also to ensure that history is rewritten in every clone that any employee has ever made of that repository; there might not even be a record of which clones exist.
  - dreamcompiler 5y ago
    The clone issue is real, but it's no more complicated than it would be without git: Keep track of who has a copy of this data and where. Whatever system you have in place to solve that problem can also be used to manage the cloning process. As for the individual repos, here's the procedure: cd <repo> rm -rf .git/ # DON'T TRY THIS AT HOME unless you know what you're doing. git add . git commit -m "Brand new repo" Do not do the above unless you completely understand the ramifications. But those ramifications might be precisely the ones you want for legal compliance.
    - guipsp 5y ago
      Might as well not use git then
    - JulianWasTaken 5y ago
      Please never do this -- not least of which because each person who does it will end up with now completely divergent histories and repos from each other. There are various tools to properly filter history, e.g. `git-filter-repo`, but the short answer is as the grandparent commenter said, things get hairy, you need to rewrite history and coordinate. It's not a situation you should hope to get yourself into...
Fice 5y ago
Using git is no big deal, but storing the repo on Github means sharing the information with a third party (Microsoft) even if the repo is kept "private".

geewee 5y ago

Oof, committing PII to Git. So much for that "right to deletion" unless you want to rewrite your entire git history.

that_guy_iain 5y ago
My understanding is the right to deletion doesn't so much affect things such as logs, internal training data, etc. Also, since that's a web store and the data comes from customer orders they would have no right to deletion as that information becomes protected for other legal requirements.
- TedDoesntTalk 5y ago
  Are they based in a jurisdiction that has right to deletion laws?
- deleted 5y ago
  [deleted]
- peterjuras 5y ago
  From a GDPR perspective, it also covers logs, internal training data etc. If a user requires to be deleted, you have to delete everything, there can be no trace of their existence.
  - shadowgovt 5y ago
    There's a reason many of us across the ocean look at that part of the GDPR like someone decided you could put the feathers back in the pillowcase if you just made it illegal for the feathers to be outside the pillowcase.
  - that_guy_iain 5y ago
    GDPR, from what the legal trainings I've had, logs aren't covered if you apply the technical requirement and costs, etc.
    - 5y ago

xd 5y ago

PII data should never touch github, ever. Good job you're not in the EU with GDPR as this process alone implies a serious lack of security due diligence. "Adafruit team began the forensic process" - Checking audit logs is hardly a forensic process and the rest of the speel about "privacy lawyers and legal experts" feels like a poor effort to regain some trust.

andi999 5y ago
Why shdnt it touch github? If you think it is not secure than also your companies source code shdnt touch it, shouldnt it? Or is there a different reason?
- sdesol 5y ago
  Trade secret and PII are two different things. Trade secret is something you can risk. PII is not something most governments will allow you to risk.
- CSSer 5y ago
  You never know where that repo is going to go. It could be private today and public tomorrow with a totally different license. Git should be thought of as sticky in the sense that it's intentionally designed to preserve a history of everything for all time in the most efficient way possible. Anyone who's ever had to scrub an API key they accidentally pushed or optimize a monorepo can attest to this.
  - MattRix 5y ago
    If it’s your own repo presumably you know exactly where that repo is going to go. The vast majority of private repos are never going to go public. I have many repos for projects that there is no chance I would ever make public, they’re not that kind of project. And even if I really did want them to be public, I would make a new repo for them instead.
    - CSSer 5y ago

ccooffee 5y ago

> Why aren’t we sending an email to every user? > We evaluated the risk and consulted with our privacy lawyers and legal experts, and took the approach that we thought appropriately mitigated any issues while being open and transparent and did not believe emailing directly was helpful in this case. Seems like a pretty weak justification. "Our lawyers said we don't have to notify users."

dangerface 5y ago
Normally you would email users notifying them of a hack so they can change passwords since there was no password breach it doesn't make a lot of sense to email people, theres nothing for them to do.
- Hamuko 5y ago
  If you are a customer who might start receiving death threats in the mail because someone leaked your email address and home address at the same time, there might be things that they need to do.
deleted 5y ago
[deleted]
devwastaken 5y ago
Yep. Lawyers will always take the road of least liability. I have no doubt the lawyer said to not even write about it publicly. I stopped trusting adafruit when they sent me broken boards and never replaced them. Their software side is very startupy, they love to break things, especially if it was working perfectly fine before. I have a USB to uart/SPI/i2c adafruit device that won't operate in any of those modes without changing windows drivers between them, and ontop of it requires local side circuit python and it used to just be C. Circuit python was a huge mistake on their part. They didn't have the people to do C correctly (look at their nrf libraries), much less C and circuit python versions. Which ends up being a compatability nightmare. So I just buy a 5 pack of esp32's or other dev boards off Amazon and be done with it. If they're broke, I get them for free. Sad to see that Amazon breaks less consumer laws than Adafruit in this case.

tmhrtly 5y ago

I wonder how many users. They don’t say, just use the word ‘some’ which makes me wonder if it’s quite a few.

dangerface 5y ago

The issue is the information was made public I don't know why people are upset about it being stored in git. If I store PII in a database and then that database gets leaked am I dumb for storing my PII in a database or for letting it get leaked? Seems kinda obvious to me.

kahrl 5y ago
People are upset that it is being stored in git because it shows how moronically negligent this company is about the security of its customer data. Storing any secrets in version control is one of the stupidest things you can do, and the fact that this was allowed to happen reflects poorly on the entire company.
NovemberWhiskey 5y ago
If you really, really want to manage PII in a private git repo, you can probably do it - in approximately the same that you can probably make API-first web services in FORTRAN if you really, really want to. The fact remains that your choice of tooling hurts you. Putting PII into Github (even if the repo is private) is catastrophic. You just made Github into a third-party data processor by accident. Good luck explaining that from a CCPA/GDPR perspective.
selfup 5y ago
The idea is that git is for source code/configs. Not data. Data gets stored in databases. Secrets/Tokens are stored in secret stores/vaults. Blobs like PDFs/etc should be stored in blob storage. If a diff is not useful, don't store it in git. Doing a code review on PII is not useful. You also shouldn't add packages in a git repo if they can be downloaded immutably in CI/locally. It adds a lot of space. Every commit is a snapshot + new code. Commits are not diffs.
- iudqnolq 5y ago
  I disagree. They were using it as sample data, when they should have used synthetic sample data. It's totally sane to check-in a JSON (or similar) file with a few hundred [edit: non-confidential] samples so that you can run integration tests without needing to set anything else up. Similarly, I keep small pdf manuals in git. I add them in their own commit, which doesn't have a useful diff. In exchange, they're always there and I don't need to spin up some special one-off system.

WYepQ4dNnG 5y ago

You should at least consider encrypting the PII on git, https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt it is not rocket sience

deleted 5y ago
[deleted]
kahrl 5y ago
You should consider never ever ever ever ever ever storing secrets in version control. Period.
- morelisp 5y ago
  Storing encrypted secrets in SCM is pretty standard practice. Usually I recommend separate repos but it's not required especially if the secret material is small and rarely changes. This data wasn't "secrets" though. PD treated this loosely would still be an issue even if it was encrypted.
  - lmeyerov 5y ago
    Standard doesn't mean good, and the sophistication to do this well is beyond most orgs and individuals . You can come up with specific scenarios, but they'll get garbled through the telephone game over time. Much better to treat data, including secrets, as radioactive, and thus follow least priv + defense in depth. If it's never there, and better, they never got it, there's nothing to worry about Ex: Before PII hits logs, encrypt it and don't give data scientists the key. Ex: orgs we work with who version notebooks won't allow notebook output to be saved, and people know (+ software) to reject baked creds. Likewise, auth isn't baked generally, instead SSO. If it's never there, for every stage, so much easier.
    - morelisp 5y ago
      No, storing encrypted secrets in repositories is an incredibly common and safe practice in DevOps/GitOps environments. Virtually every orchestration tool has some way to support it. E.g. k8s sealed secrets https://github.com/bitnami-labs/sealed-secrets https://github.com/bitnami-labs/sealed-secrets E.g. Salt's GPG filters https://docs.saltproject.io/en/latest/ref/renderers/all/salt.renderers.gpg.html https://docs.saltproject.io/en/latest/ref/renderers/all/salt... E.g. Ansible Vault https://docs.ansible.com/ansible/latest/user_guide/vault.html https://docs.ansible.com/ansible/latest/user_guide/vault.htm... E.g. Puppet hiera-yaml https://puppet.com/docs/puppet/6/securing-sensitive-data.html https://puppet.com/docs/puppet/6/securing-sensitive-data.htm... PD/PII is a completely separate issue. First because even encrypting doesn't remove legal obligations concerning processing, and second because your DS/BI teams probably need access to the unencrypted data to like, do their actual jobs. You need completely orthogonal forms of access control for that (like SSO as you alude to).

globular-toast 5y ago

Does Github delete blobs when a repository is deleted? I doubt it.

niros_valtos 5y ago
Your doubt is right. Here is how repos can be restored: https://docs.github.com/en/repositories/creating-and-managing-repositories/restoring-a-deleted-repository https://docs.github.com/en/repositories/creating-and-managin...

niros_valtos 5y ago

Storing PII on GitHub is not wise. It is stored with the history and can be restored even if deleted. As for opening up the repo publicly, it can be achieved with a quick win by triggering alerts on such changes.

deleted 5y ago
[deleted]

kaycebasques 5y ago

Classic corporate use of passive voice to deflect responsibility.

jcrawfordor 5y ago
Honestly, just the title of this post has caused me to lose some trust in Adafruit as a company. It's so completely passive and doesn't explain what actually happened. It feels like it was intentionally written to avoid attracting attention, but then attracts attention because of how vague and weird it is.
- kaycebasques 5y ago
  The Streisand Effect! https://en.m.wikipedia.org/wiki/Streisand_effect https://en.m.wikipedia.org/wiki/Streisand_effect
tomcatfish 5y ago
What are they supposed to do, say "Jeff the intern fucked up badly, and Christine let them do it"? What is this criticism supposed to mean?
- hnaccount141 5y ago
  It's not about singling out individuals, it's about Adafruit as a company taking responsibility for their mistakes. The phrasing in the post makes it sound like the breach was something that happened to them, vs being the direct result of the company's actions.

WhompingWindows 5y ago

I've been in data analysis for a few years -- grew from scientific research in SAS with zero VCS, into programming and data analysis in R and STATA, still no VCS for a while...eventually I was allowed to use Github after I left my restrictive hospital research space. No one ever sat me down and trained me on Git, though. I just heard repeatedly: if you're not using Git, you're missing out on a huge set of important software-creation tools, you shouldn't be coding without VCS. Why did this happen? Imagine it's this data analysts' first year out of school, something like a crappy statistics program that only teaches SAS or base-R with no VCS. This young padawan needs to practice analysis in preparation for a work project. They grab some data and stash it in a folder where Git is tracking. They do their analysis practice, it's Friday afternoon, they get lazy, they don't look at what they're committing, and they click buttons in the GUI without much thought. It's incompetence and laziness, not malice. These tools that allow us to share widely, not just GH but also social media broadly: these tools have great power that should be used with greater training, responsibility, and care.

duped 5y ago
I don't think it's incompetence so much as professional ignorance (which might be incompetence, I don't think it is). Source control is for your source code, not for your data. Your data belongs in a data base. Git is not a database, or at least shouldn't be treated like one. Sure it's easy to call it lazy for a data set to be in some local directory and accidentally get committed. Happens to all of us. The bigger problem is "why is that data sitting on your file system in a directory, when it should be in some data base, preferably not locally." > these tools have great power that should be used with greater training, responsibility, and care. This screams more and more that the tools are bad. Git is famously hard to use and even harder for non-plaintext data. Databases are annoying to initialize and get access to without a developer who's done it before. The tools suck, they can be better, and require less training. It's not wrong to be lazy - it's wrong to make the lazy path dangerous.
- Karellen 5y ago
  > Git is not a database Um, yeah, it is - by most reasonable definitions of the word "database". No doubt there are a few unusually-narrow definitions of "database" out there that would exclude Git, but I'm pretty certain they're in the minority.

kahrl 5y ago

Adafruit's response to this is appalling. Storing secrets in version control is one of the stupidest things you can do, and reflects poorly on the entire company. But the fact that they are choosing to not inform their customers that their PII was leaked due to their negligence is criminal (well, in certain jurisdictions). Shame on Adafruit. Hope they get sued out of existence.

apgwoz 5y ago
> Shame on Adafruit. Hope they get sued out of existence. They made a mistake. The response to the mistake is NOT GOOD. However, destroying a company that also does actually good things is not the answer here, either. How should they fix this? I think they should start be writing another blog post apologizing for how bad the first one was, for starters. Emails to affected people next... But, let's be honest. How much better is _that_? Can't unleak the data. You can only be offered subscriptions to data protection services _sooo many times_. There's not really a monetary value as damages here...

dzhiurgis 5y ago

Office Depot Salesforce source code still up. Complete with some integration private keys. Bug bounty says it’s not a bug… Salesforce employee recently published source of one of their products. I’ve reported via email since I’ve been removed from their private Hackerone programme, presumably due inactivity. Sec team just said it was “test data”. Wish I’ve made a copy since it’s gone now and bullshit like this responses just wants me published everywhere.

dzhiurgis 5y ago
Ok, I'm giving some credit to company above. Someone contacted me hours after this post and took down repo like minutes after I've responded (tho they say they found it in parallel). Root cause - third party was doing some POC and I'm guessing misconfigured CI.
kaetemi 5y ago
Hackerone is a joke, anyway. Organizations will just respond with "it's a feature, not a bug" to get out of any bounty. I once reported that you could log on to certain PP accounts with just username and CC number, bypassing configured 2FA, and allowing to wipe the 2FA. Guess the response. Lo and behold, it's fixed now.
- throwoutway 5y ago
  Hackerone should be an escrow that can arbitrate and overrule insipid behavior like that. But they won’t, will they? Maybe someone else will fill that void

brettproctor 5y ago

For posterity, they updated the post on Mar 7 and will be emailing anyone affected. "Update March 7, 2022: We appreciate the feedback from the community and our customers, and will be emailing users as part of this disclosure. We apologize for not doing that at the same time as the post/disclosure on Friday, March 4, 2022."