20 ms·

Toyota suffered a data breach by accidentally exposing a secret key on GitHub

589 points by whack 4y ago

bootcat 4y ago
dieyota !
ajross 4y ago
"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking mistakes that you'd think would be easy to avoid, but aren't.
- ElevenLathe 4y ago
  Add "failure to rotate TLS certs before they expire."
  - HWR_14 4y ago
    But at least that failure just makes the services fail, not opens a security hole.
    - cmeacham98 4y ago
      Except in the scenarios where the company's support starts telling users to click through the warning, which I've seen a few times.
      - HWR_14 4y ago
        Yuck. I hadn't considered that.
    - deleted 4y ago
      [deleted]
  - deathanatos 4y ago
    My company has monitoring for this, but it still seems to be a law of nature that, 1. someone adds new service/server/infra in a submarine manner 2. it goes to prod 3. the cert expires and outage begins 4. my team is asked what to do, because "we're the cert experts" 5. we add it to the monitoring So it only happens once … per service. Which isn't great. But how do you get people to slow down and do simple shit, like add a monitor to your new service? (Or promulgate a design doc…)
    - ElevenLathe 4y ago
      I think the real answer is to only issue limited-duration certs and only via automated means (ACME or similar), thus requiring automation be in place from day 1. This still doesn't protect against the vector where somebody else in the company has managed to prove themselves to be responsible parties to another CA/issuer.
      - deathanatos 4y ago
        Oh I agree! Everything would be ACME if I could. A ridiculous amount of stuff still doesn't support it, though. And, like I said, usually it's someone who doesn't grok certs doing it without asking for help in the first place, so they're not going to get why ACME. (Because I am tired of doing cert renewals. I've had enough for a lifetime…)
    - deleted 4y ago
      [deleted]
    - HL33tibCe7 4y ago
      “If you keep smelling shit, look at your own shoe”. Your processes are failing your development teams, and you need to fix them, rather than blaming your teams, which achieves nothing.
      - deathanatos 4y ago
        I agree, but that level of authority is in a different level of the foodchain. I'm on duck tales, Larry.
- coenhyde 4y ago
  Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.
  - deathanatos 4y ago
    And yet I see it get violated all the time. People should do a lot of things, but a lot of my coworkers are lazy and do not do quality work. Given that it happens, and that I can't prevent it, one must then ask how to guard against it. At my org, we even try to generate all secrets with a standardize prefix/suffix so as to make them very greppable. That doesn't stop "Architects", "Customer Solutions", "Analytics" types from … just working around the standard tooling and manually generating one by hand because … IDK, they think they know better? I really don't get it.
    - coenhyde 4y ago
      Doctors used to not wash their hands too. I get it though, and i've seen the same thing. Really it comes down to education and not granting access to secrets to people who aren't capable of handling them.
      - robswc 4y ago
        "fun" fact - there could potentially be thousands of deaths attributed to Drs simply not washing their hands. IIRC they even basically got some hospital admin fired for creating a hand washing mandate, despite it being proven to save lives. https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... (talking centuries ago, but maybe even today) Looks like its still a "recent" issue, lol https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.html https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.htm...
        4y ago
- wereHamster 4y ago
  > Production keys in source control IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.
  - JauntyHatAngle 4y ago
    Huh? No, you use an external secrets manager or leave it to run time environment vars, or leave it to your cd servers to access/supply those details. Assuming you have been given the right tooling, there is no reason for it to be in source code.
    - danenania 4y ago
      If anyone out there is using environment variables currently, and is interested a quick path to plugging the leaks in their secrets management, check out EnvKey[1] (disclaimer: I'm the founder). Because EnvKey integrates tightly with environment variables, no app code changes are needed to switch, so it only takes a minute or two to import/integrate a typical app. EnvKey is designed to help avoid incidents exactly like the one that just hit Toyota, while staying out of your way and even making your life significantly easier as a developer by simplifying many aspects of config management. Give it a look if you know you have some room for improvement in this area and are looking for an easy, secure, open source, end-to-end encrypted solution :) 1 - https://envkey.com https://envkey.com
  - moooo99 4y ago
    Even with IaC you should store secrets only in your environment variables for the CI servers
  - coenhyde 4y ago
    You can put your keys in source control if you are encrypting them with another key which is not in source control. Otherwise you're doing it wrong.
    - Too 4y ago
- bonestamp2 4y ago
  I committed my google maps api key to a public github repository recently and github immediately sent me a warning about it. The thing is, I did it intentionally. The key is used on my website and the website is served by github pages. Now, it's an embedded maps api key, there's no cost to use it, nobody can use it from a domain other than mine, and it's easily visible in the page source if someone views that, so there's really no reason not to commit it since even if I didn't commit to somewhere publically, it's still publicly available on my website and nobody else can use it anyway. Is this a reasonable exception to this rule?
  - tomschwiha 4y ago
    I just wondered what happens if you spoof locally the domain name. Could you still use the api key?
    - bonestamp2 4y ago
      Maybe. Thankfully there's no cost to use the key if someone does it.
  - prepend 4y ago
    I could run your site locally with a customized host file so the referers all come from your domain. I don’t think it’s that much of a risk but I wouldn’t want to use a key associated with something that can bill me. You could use Google actions to build your pages site injecting the api key at build time. It’s stored a repo secret rather than in code. Of course since you deploy the site publicly, the key will still be visible.
lumberjack24 4y ago
The access model on platforms like GitHub is flawed, a single account can be used for both professional and personal projects/repositories, leading to “fat finger” errors like this one here...
- tester756 4y ago
  I don't think it is flawed. You cannot access org's repos without VPN if you create a new repo by mistake outside your org, then uhh..., it's crazy? it's like sending email with credentials to people outside your org
  - alfalfasprout 4y ago
    On github.com you don't need VPN to access your org's repos. You're referring to github enterprise (the self hosted version).
    - tester756 4y ago
      I do use github.com and I cannot access some repos unless I connect to VPN
- anamexis 4y ago
  I don't see how this is related to GitHub's access model. Was the canonical Toyota repo even on GitHub?
- gw99 4y ago
  Oh yes this. It's so easy to critically fuck up an invite into an organisation. If you get typo the username you are potentially compromised. I've seen a couple of near misses on this already. Note: the invite input box actually autocompletes ALL github usernames.
  - matai_kolila 4y ago
    > Note: the invite input box actually autocompletes ALL github usernames. I'm sorry, but that's wild. That's like, not even an easy engineering problem to solve necessarily, given their size!
greyface- 4y ago
> T-Connect enables features like remote starting, in-car Wi-Fi, digital key access, Could an attacker have remote-unlocked and remote-started the entire Toyota fleet with this access?
- mcdwayne 4y ago
  Toyota is claiming no, not with this leak. It was a partial repo that was exposed. The data they accessed with the key got customer ID numbers and emails only.
danielodievich 4y ago
Many years ago I got a trial license key for something, Aspose components of some sorts I think, and without thinking of it, checked it in into public Github repo. Well, few days later Aspose's support sends me a nicely worded note saying that they noticed that it was there and invalidated it for me. Their description and instructions were very clear about why they did it and why I shouldn't have checked it in. I thought that was very proactive and excellent customer service.
- Natsukvshii 4y ago
  I actually had something similar happen to me last month. I accidentally published a discord API key to GitHub and within minutes I got a nice message from “Safety Jim” to my personal discord account letting me know they’ve found my key on a public repo and have gone ahead and revoked it. I felt like a bit of a dope but it was neat to have it happen to me. Lesson learned for sure.
  - greysteil 4y ago
    GitHub PM here. Glad that was a good experience! We work with ~50 partners (details in the link below) to notify them when tokens for their service are exposed in public repos, so that they can notify you. https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns#supported-secrets-for-partner-patterns https://docs.github.com/en/code-security/secret-scanning/sec...
    - fiddlerwoaroof 4y ago
      I wish I could set this up to block pushes proactively instead of reacting to pushed secrets.
      - kadoban 4y ago
        You could set up something like https://github.com/godaddy/tartufo https://github.com/godaddy/tartufo in a pre-commit hook. Not sure if github has a way to hook into the push hooks on server side, they might though.
chaps 4y ago
Hah. Yeah. Found a bunch of ssh keys, passwords, etc for Comcast years back which turned into a shitshow when I tried to report it. Once I found the right people to talk to things got better, but the entire experience was really reflective of how bad large orgs are with security. A friend once told me he was having a hard time getting a client to take his security concerns seriously. So I went on github and found a commit in their repo that included a production password and sent it to him. Maybe took 5-10 minutes to find? Apparently once they found out about the commit, they panicked a bit and started taking his concerns more seriously.
- mcdwayne 4y ago
  Yikes. It is sad to hear stories like that, where security is not a concern until panic sets in. :( Yet another reason we need to adopt standards like security.txt and make it easy to report these things as it is to tell robots to ignore us with robots.txt. See securitytxt.org for more on the project.
  - dinvlad 4y ago
    I think the fundamental problem is, a lot of orgs just don't care about security, as it doesn't affect their bottom-line. Even breaches are only a temporary hit on the PR. Proper way to address that might just be legislation, with heavy fines based on total revenue. That and also security is just hard to scale. That's why if it was mandated by legislation, companies would be forced to spend a comparable amount on scaling their security teams and efforts.
  - bisby 4y ago
    It's tough. I'm our public security reporting email list. We get a lot of things that boil down to "When I go to your website, I am able to see the content of your html files!" ... yes, reporter. That is what a web server does. It gives you HTML files. Congrats that you have figure out the dev console on your browser, but you're not a hacker. I'm trying to go with Hanlon's razor here and assume this is inexperienced people and not outright scams. We don't get a lot of these, but they far outweigh actual credible reports. But we try our best and take everything seriously until it can get disproven. And it's exhausting. So I get it sometimes. Sometimes having a place for responsible disclosure just opens yourself up to doing more paperwork (verifying that the fake reports are fake). That said, we still do it.
dinvlad 4y ago
I wish hosted GitHub made pre-push hooks available to the public. Would make this a much easier problem with free scanning tools like Trufflehog. Or alternatively, if GitHub Secret Scanning was available to all public repos, instead of requiring a (very) expensive GitHub Advanced Security subscription. But I understand, they need to make money somehow.
- lumberjack24 4y ago
  In the meantime, try ggshield cli https://github.com/GitGuardian/ggshield https://github.com/GitGuardian/ggshield
  - dinvlad 4y ago
    Nah thanks, I'm already running Trufflehog for free on all of our multiple orgs' thousands of repos. I think we would consider GG if its pricing was acceptable for non-profits though.
    - mcdwayne 4y ago
      FYI - GitGuardian is free for individuals and teams smaller than 25
      - dinvlad 4y ago
        As told earlier, we have thousands of repos. And our teams are thousands of users on GH.
- greysteil 4y ago
  (GitHub PM here.) The Advanced Security secret scanning experience is coming to public repos (for free, obviously)! Give us a few more months - we have a little more work to do scaling it up
  - dinvlad 4y ago
    Awesome, thanks for the heads-up! We will definitely look forward to it, either to replace or enhance our current in-house solution.
jrockway 4y ago
> Git is an awesome version control system, used by over 93% of developers, and is at the heart of modern CI/CD pipelines. One of the benefits of Git is that everyone has a complete copy of the project they are working on. I feel like this is copy-pasted from a pitch deck on why GitGuardian should be funded. Does anyone reading the article care about this anecdote? Like do people stop reading at "well I'm one of the 7% that doesn't" or think "wow a lot of people are using Git, I should buy their thing"? Sorry for the meta comment, but it just stuck out as odd to me.
- dinvlad 4y ago
  This is definitely a marketing piece. And they charge a LOT for it, so it's not a solution for the common masses.
  - zricethezav 4y ago
    If using GitHub-Actions, Gitleaks offers competitive pricing for a secret scanning solution. https://gitleaks.io/products https://gitleaks.io/products
    - dinvlad 4y ago
      Thanks, I've already figured out how to run Trufflehog for free on our thousands of repos.
      - deleted 4y ago
        [deleted]
      - freeqaz 4y ago
        There is still a lot of noise with basic tools like this (I've also used trufflehog at scale). To properly handle secret scanning requires calling live APIs to test if keys are "real". And you need to have a way to file tickets when you do have findings... if you rotate a cred from production, that's now an outage, so you need to coordinate multiple teams. It's a lot of work and free tools only solve one part of this. I can't speak to any of the vendors in this space but I can attest that it's a harder problem than it seems!
bsimpson 4y ago
The social media manager at GitGuardian is winning. They just got us each to read a 1000 word ad for GitGuardian.
- site-packages1 4y ago
  Jokes on them, I never read the articles and only ever read the comments unless the comments indicate the article was interesting enough to read.
- lordgrenville 4y ago
  By writing an well-constructed and discussion-provoking article? "Mission fucking accomplished" https://xkcd.com/810/ https://xkcd.com/810/
- TheRealDunkirk 4y ago
  This. I used to use them, because I, too, have been burned by my own mistakes before. However, I had to stop using them as a plugin for GitHub because (at least when I was looking) there doesn't seem to be a way to exclude private repos from the reporting, and there were some false positives in private repos I don't want flagged. But I think the service is a good idea in general.
robswc 4y ago
It can actually be comical just how _bad_ things can be at large orgs. Anyone have details, theories, or a book on how such inefficiencies come about? I can't speak to tech-oriented large orgs but I've worked with others and its just... I'm not shocked at all. I've seen public facing API keys in HTML, private SSH keys that do god knows what in plaintext on FTP servers... I just don't understand how they seem to care so much but in reality, care so little. Just lazy?
- nomel 4y ago
  In my experience, security is so far removed from the actual job description/day to day cares that it's perpetually "somebody else's problem", seen as an unnecessary time sink. Usually there's a couple of people that actually care, but they're ignored and lack the power to influence change. Not lazy, just overburdened by more important things.
  - robswc 4y ago
    That seems to match what I've seen. Especially the last line. It's just so weird the dynamic between "pretending to care" and "actually caring." I've worked at or consulted on small teams that were very "lax" about security but everyone seemed to take it seriously so it "worked." At larger orgs, I notice they take it "seriously" but that results in people finding creative loopholes... like pasting in all the important production keys/passwords into a google doc and sharing the google doc because they "can't send secrets over slack" ... lol
- 0xbadcafebee 4y ago
  You're the receptionist at a high-security facility. Lots of people work there. Your job is to make sure the people get in and out quickly. The building has multiple methods of access at different stages. During the course of doing your job, you don't follow the proper protocols a few times. You closed a door without re-entering a lock code. You didn't check someone's badge and ID at the third floor access gate. You forgot to make sure someone swiped out on exit. Nobody actually trained you on any of those protocols; they just made you watch a "security is everyone's responsibility" video and then expected you to know all the protocols. Maybe once or twice you were lazy, but it's equally likely you just didn't know what to do. And once you make that mistake, somebody can exploit it as long as you don't follow the protocol. In facilities where security is important, there are safeguards. Doors left open sound an alarm, automatically lock when closed, and sensors detect if more than one person walks through without badging in. These safeguards exist because people are fallible and need help to enforce security. Companies without security safeguards either don't care, or are ignorant.
zricethezav 4y ago
Good reminder to run Gitleaks or Gitleaks-Action on your repos - https://github.com/zricethezav/gitleaks https://github.com/zricethezav/gitleaks - https://gitleaks.io/products https://gitleaks.io/products
eax-eip 4y ago
Most of these (even sometimes expensive) tools only look at repos and users who are associated with the company’s GitHub org, which barely solves the problem. The much harder problem is the number of corporate secrets that are on random repositories (personal dotfiles, automations, data science scripts, etc.) across GitHub with no strong relationship to the organization. Try using GitHub Code Search to find all the Fastly API tokens that have been leaked, for example, and I bet you’d find some wild stuff.
- lumberjack24 4y ago
  GitGuardian actually does this, it monitors an extended perimeter of devs and their personal/open-source repos for corporate secrets or keywords – https://www.gitguardian.com/monitor-public-github-for-secrets https://www.gitguardian.com/monitor-public-github-for-secret...
  - sidewndr46 4y ago
    Yeah, because as an employee what I totally want is my employer to monitor my every digital move outside of work!
    - deleted 4y ago
      [deleted]
    - elguyosupremo 4y ago
      Make a private repo. I wouldn't blame a corp if they tried to scan every public github repo for their API keys, let alone an employee's public account.
userbinator 4y ago
Too bad this wasn't for their ECUs or something else that could've benefited right-to-repair.
ajsnigrutin 4y ago
Right to repair bad, independent repair centers will expose your private data! - Toyota ( reference https://www.youtube.com/watch?v=GTiLnz23TNs https://www.youtube.com/watch?v=GTiLnz23TNs )
Kiro 4y ago
Question: Let's say I want to open source my app but a long time ago I used to have credentials hard coded. What can I do to clean this up from history?
- scarmig 4y ago
  https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository https://docs.github.com/en/authentication/keeping-your-accou...
- tyler569 4y ago
  Git does have tools that allow you to rewrite history to fix situations like this, but by far the easiest solution is to invalidate those credentials so they become worthless.
- lalopalota 4y ago
  make a new repo starting with the code in its current state.
- lumberjack24 4y ago
  Here's a checklist [1] (again, from gitguardian) of steps to follow before open-sourcing projects and [2] a guide on how to remediate hardcoded/exposed secrets. [1] https://blog.gitguardian.com/safely-open-source-software-best-practices https://blog.gitguardian.com/safely-open-source-software-bes... [2] https://blog.gitguardian.com/leaking-secrets-on-github-what-to-do/ https://blog.gitguardian.com/leaking-secrets-on-github-what-...
fny 4y ago
Is there any reason why keys don't constantly update? It seems like a service could exist where every five minutes a rotation occurs across services with decaying privileges. For example, the 5 minute old key still works, but the 10 minute woman has completely expired.
- crazygringo 4y ago
  How would you update the keys every 5 min? Either you'd use an encryption algorithm that depended on a "deeper" key... or you'd fetch the new key while authenticated with, you guessed it, another "deeper" key. It's keys all the way down. Every key you use, it's your responsibility to keep it private. (Unless you want to be dealing with physical hardware dongles that generate keys, but those aren't exactly easily portable.)
  - hnick 4y ago
    Not exactly the same but I had to integrate with a payments API which required one call to an auth endpoint with user/password under HTTP basic auth to get an access token. The actual calls to calculate and execute payments used this access token instead as a bearer token. Those ones expired. I'm not sure I really see the point, but I guess you could lock down the main call on another system, store the temporary token somewhere, and have the other systems use it. As you say, you still need the top level credentials somewhere to get a new token.
    - windowsworkstoo 4y ago
      Yah, that's the OAuth Client Credentials flow but as noted, you still have a static set of creds that are required to generate the short lived access token. Besides being useful for being able to limit scope in some circumstances, the main point of the client cred flow is to appease eager sec arch's who insist on OAuth.
  - deleted 4y ago
    [deleted]
  - Too 4y ago
foxandmouse 4y ago
Was this title re-written? I feel it was phrased more vaguely when I first read it, but I could be wrong... More curious about if that's a thing hackerNews does.
- sen 4y ago
  Titles are edited to match the article usually.
- EE84M3i 4y ago
  It was "Toyota Accidently Exposed a Secret Key Publicly on GitHub for Five Years". Thank you https://hackernewstitles.netlify.app/ https://hackernewstitles.netlify.app/
beyang 4y ago
In case it's helpful, here's a bookmarklet that instantly searches any public repository for the most common secret patterns: https://about.sourcegraph.com/blog/no-more-secrets https://about.sourcegraph.com/blog/no-more-secrets. Sourcegraph built a feature called "code monitors" that basically runs a recurring search in the background to guard against anti-patterns like API keys and secrets being committed into the codebase.
gumby 4y ago
Why would Toyota need to have that info to provide service? We need to get PII as a liability on companies’ balance sheets to get them to take it seriously and to collect only the minim viable data.
- jaywalk 4y ago
  Why would they need to have > customer identification numbers and emails to provide service? Seriously?
  - gumby 4y ago
    Yes, seriously. They need to service the vehicle, OK, but communicate to and through the vehicle.
    - 8organicbits 4y ago
      What does that mean? The repair tech tells my car that it's ready for pickup and my car ignores him because it's a car not a communication channel?
      - gumby 4y ago
        I can tell my local repair guy down the road my number. No need for some megacorp to know who I am.
    - PebblesRox 4y ago
      These are the services, according to the article: “T-Connect enables features like remote starting, in-car Wi-Fi, digital key access, full control over dashboard-provided metrics, as well as a direct line to the My Toyota service app. The servers that control these options contain unique customer identification numbers and customer emails.”
      - gumby 4y ago
        I don't see why any of that should require the email address. They can communicate with the customer through the app or through the car UI. In general, apps and sites these days hoover up more info than they need simply because they can, not because it adds to the customer experience (and often doesn't help the company either). There is no incentive to be in any way judicious about what to collect and the frequent breaches show that even the companies don't value PII as something worth protecting because it's not core to their business.
rainbringer2000 4y ago
production code + keys = no bueno. I feel for them.
- deleted 4y ago
  [deleted]
juancampa 4y ago
To prevent accidental leaks perhaps Github et al could implement a mechanism analogous to the Eurion constellation [0]. E.g. A commit containing a file with a specific text pattern should always be rejected. Most fileypes accept arbitrary comments (ahem Json) so it should work for most files. 0: https://en.m.wikipedia.org/wiki/EURion_constellation https://en.m.wikipedia.org/wiki/EURion_constellation
hn_throwaway_99 4y ago
This is why you should never, never, never check production keys into git. There are tons of secret manager services out there. All the major cloud services have them and even companies like 1Password now have them.
stall84 4y ago
welp .. that makes all the times my n00b duma$$ did it not feel so bad.
GolDDranks 4y ago
There are software such as Trufflehog ( https://github.com/trufflesecurity/trufflehog https://github.com/trufflesecurity/trufflehog ), that find secrets. We are using it at an organizational level, but there's always some delay from finding something and getting it reported. I've been meaning to add it both to our CI so our team can notice right away, and even to Git push hooks, to catch these cases early.
deleted 4y ago
[deleted]
ghostbuster09 4y ago
We’ve been using Nightfall to protect against leaking secrets and other sensitive data across github and other apps https://nightfall.ai/solutions/product/github https://nightfall.ai/solutions/product/github
paradite 4y ago
I have an idea: What if companies using secret keys (not SaaS providers who generate secret keys) voluntarily send their secret keys (MD5 hashed or somehow encrypted) to GitHub and GitHub can then monitor their leakage and notify the company? Is this useful or feasible?
- devcat 4y ago
  Github already has system to invalidate tokens without storing any hashes themselves https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns https://docs.github.com/en/code-security/secret-scanning/sec...
  - paradite 4y ago
    Yes. This is dependent on some kind of agreement or understanding between SaaS provider and GitHub. I'm thinking of a generic approach which is independent of where you got the secret key and its format.
    - devcat 4y ago
      Same system can be applied for repo owner being pattern provider and getting notified with matches but if i had to guess only enterprise customers might get feature like this
AtNightWeCode 4y ago
> A credential for a DB server holding customer data was hardcoded into the repo. This is normally not a problem since databases should not be exposed to everyone one the Internet. > One of the benefits of Git is that everyone has a complete copy of the project they are working on. This is one of the problems with GIT and many programming langs. Because of this full copies of entire code bases are everywhere. GIT also only has ACL on repo level. In many langs it is recommended to not store credentials in the code but at the same time there are no guidelines for how to store credentials. If you use cloud services there are recommendations. Edit: Btw, which git service where they using? Cause I believe Github and others monitor code bases for accidental pushes of secrets.
- Gibbon1 4y ago
  I tool I would like, and likely linker magic could handle this, is a standard tool to patch elf files with things like revision info and keys[1]. [1] Notable, friend in the security biz had the advice that stored keys and passwords should never just work.
koksik202 4y ago
My access got revoked for MyT app and then few days later I was able to bind my vehicle again to profile I wonder if that was caused by this
atoav 4y ago
Github is amazing. Wanna get expensive licensed fonts for free? Just search for $FONTNAME.otf github and you will find at least a few projects using it. People don't think a lot about what they put on there it seems. Or maybe font foundaries haven't sued enough at this point.
Paul_S 4y ago
I worked for a couple corps in my life and the security theatre in here is ridiculous. At the same time things like this happen all the freaking time. And none of the company spyware can catch that. Funny how this never happened in small companies I worked for, although maybe that's just a function of number of people? Or maybe because small companies rely on fewer platforms and frameworks so fewer chances of such leaks?
- withinboredom 4y ago
  In my experience, smaller companies tend to hire the best they can attract and afford. Larger companies tend to collect talent so no one else can get it, so it usually isn't "the best."
isolli 4y ago
Reminds me of a related HN discussion a few years ago: someone searched for "remove password" in GitHub and unearthed who knows how many valid passwords in the hundreds of thousands of commits that the search returned... [0] https://news.ycombinator.com/item?id=13650818 https://news.ycombinator.com/item?id=13650818
dzhiurgis 4y ago
Apparently tons of secrets get leaked to the public Postman recipes library
bxtt 4y ago
Ah. I can’t believe this still happens in this day of age. About a decade ago, I was working for a startup and we were getting dominated in our growing space by a much larger, well funded rival. Our competitive intelligence team browsed through their git, and the rival actually exposed access to their customer, pricing and sales agent database by leaving their credentials in one of their branches. The team went to our legal department asking if they can be protected by the company, and if they can use this intel. The team then worked with the product team to integrate all their pricing engines to our POS to undercut their pricing and sent marketing blasts to their leads with targeted marketing campaigns. Long story short that company is now defunct, and it definitely undermined their growth.
- joshxyz 4y ago
  and that folks, is how you do growth hacking ;)
- empyrrhicist 4y ago
  If you're in the US, that's 100% a crime. If they were responding to an unauthenticated API or web request that's one thing, but using a leaked password on a database is not legal at all.
  - rtev 4y ago
    Yeah, 3+ years in prison for that. Corporate espionage hacking is real.
pknerd 4y ago
Surprised that they do not the likes of .env files which are usually not pushed to a repo.
vivan 4y ago
I used to report things like this that I had found, including cases where I can see people used the default "sample" config for security purposes, but I found that either people would not care at all, or massively overreact and somehow blame me. If an organisation is disorganised enough to leave critical details in public, they're probably too disorganised to handle someone reporting it.
pojzon 4y ago
Some time ago on devops reddit someone asked „What is the biggest vulnerability in your system?” Most common answer was „Developers” It do be like that. Without proper automation and processes and education of ppl involved it can get really bad.

589 points by whack 4y ago

dieyota !

"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking mistakes that you'd think would be easy to avoid, but aren't.

ElevenLathe 4y ago
Add "failure to rotate TLS certs before they expire."
- HWR_14 4y ago
  But at least that failure just makes the services fail, not opens a security hole.
  - cmeacham98 4y ago
    Except in the scenarios where the company's support starts telling users to click through the warning, which I've seen a few times.
    - HWR_14 4y ago
      Yuck. I hadn't considered that.
  - deleted 4y ago
    [deleted]
- deathanatos 4y ago
  My company has monitoring for this, but it still seems to be a law of nature that, 1. someone adds new service/server/infra in a submarine manner 2. it goes to prod 3. the cert expires and outage begins 4. my team is asked what to do, because "we're the cert experts" 5. we add it to the monitoring So it only happens once … per service. Which isn't great. But how do you get people to slow down and do simple shit, like add a monitor to your new service? (Or promulgate a design doc…)
  - ElevenLathe 4y ago
    I think the real answer is to only issue limited-duration certs and only via automated means (ACME or similar), thus requiring automation be in place from day 1. This still doesn't protect against the vector where somebody else in the company has managed to prove themselves to be responsible parties to another CA/issuer.
    - deathanatos 4y ago
      Oh I agree! Everything would be ACME if I could. A ridiculous amount of stuff still doesn't support it, though. And, like I said, usually it's someone who doesn't grok certs doing it without asking for help in the first place, so they're not going to get why ACME. (Because I am tired of doing cert renewals. I've had enough for a lifetime…)
  - deleted 4y ago
    [deleted]
  - HL33tibCe7 4y ago
    “If you keep smelling shit, look at your own shoe”. Your processes are failing your development teams, and you need to fix them, rather than blaming your teams, which achieves nothing.
    - deathanatos 4y ago
      I agree, but that level of authority is in a different level of the foodchain. I'm on duck tales, Larry.
coenhyde 4y ago
Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.
- deathanatos 4y ago
  And yet I see it get violated all the time. People should do a lot of things, but a lot of my coworkers are lazy and do not do quality work. Given that it happens, and that I can't prevent it, one must then ask how to guard against it. At my org, we even try to generate all secrets with a standardize prefix/suffix so as to make them very greppable. That doesn't stop "Architects", "Customer Solutions", "Analytics" types from … just working around the standard tooling and manually generating one by hand because … IDK, they think they know better? I really don't get it.
  - coenhyde 4y ago
    Doctors used to not wash their hands too. I get it though, and i've seen the same thing. Really it comes down to education and not granting access to secrets to people who aren't capable of handling them.
    - robswc 4y ago
      "fun" fact - there could potentially be thousands of deaths attributed to Drs simply not washing their hands. IIRC they even basically got some hospital admin fired for creating a hand washing mandate, despite it being proven to save lives. https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... (talking centuries ago, but maybe even today) Looks like its still a "recent" issue, lol https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.html https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.htm...
      - 4y ago
wereHamster 4y ago
> Production keys in source control IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.
- JauntyHatAngle 4y ago
  Huh? No, you use an external secrets manager or leave it to run time environment vars, or leave it to your cd servers to access/supply those details. Assuming you have been given the right tooling, there is no reason for it to be in source code.
  - danenania 4y ago
    If anyone out there is using environment variables currently, and is interested a quick path to plugging the leaks in their secrets management, check out EnvKey[1] (disclaimer: I'm the founder). Because EnvKey integrates tightly with environment variables, no app code changes are needed to switch, so it only takes a minute or two to import/integrate a typical app. EnvKey is designed to help avoid incidents exactly like the one that just hit Toyota, while staying out of your way and even making your life significantly easier as a developer by simplifying many aspects of config management. Give it a look if you know you have some room for improvement in this area and are looking for an easy, secure, open source, end-to-end encrypted solution :) 1 - https://envkey.com https://envkey.com
- moooo99 4y ago
  Even with IaC you should store secrets only in your environment variables for the CI servers
- coenhyde 4y ago
  You can put your keys in source control if you are encrypting them with another key which is not in source control. Otherwise you're doing it wrong.
  - Too 4y ago
bonestamp2 4y ago
I committed my google maps api key to a public github repository recently and github immediately sent me a warning about it. The thing is, I did it intentionally. The key is used on my website and the website is served by github pages. Now, it's an embedded maps api key, there's no cost to use it, nobody can use it from a domain other than mine, and it's easily visible in the page source if someone views that, so there's really no reason not to commit it since even if I didn't commit to somewhere publically, it's still publicly available on my website and nobody else can use it anyway. Is this a reasonable exception to this rule?
- tomschwiha 4y ago
  I just wondered what happens if you spoof locally the domain name. Could you still use the api key?
  - bonestamp2 4y ago
    Maybe. Thankfully there's no cost to use the key if someone does it.
- prepend 4y ago
  I could run your site locally with a customized host file so the referers all come from your domain. I don’t think it’s that much of a risk but I wouldn’t want to use a key associated with something that can bill me. You could use Google actions to build your pages site injecting the api key at build time. It’s stored a repo secret rather than in code. Of course since you deploy the site publicly, the key will still be visible.

lumberjack24 4y ago

The access model on platforms like GitHub is flawed, a single account can be used for both professional and personal projects/repositories, leading to “fat finger” errors like this one here...

tester756 4y ago
I don't think it is flawed. You cannot access org's repos without VPN if you create a new repo by mistake outside your org, then uhh..., it's crazy? it's like sending email with credentials to people outside your org
- alfalfasprout 4y ago
  On github.com you don't need VPN to access your org's repos. You're referring to github enterprise (the self hosted version).
  - tester756 4y ago
    I do use github.com and I cannot access some repos unless I connect to VPN
anamexis 4y ago
I don't see how this is related to GitHub's access model. Was the canonical Toyota repo even on GitHub?
gw99 4y ago
Oh yes this. It's so easy to critically fuck up an invite into an organisation. If you get typo the username you are potentially compromised. I've seen a couple of near misses on this already. Note: the invite input box actually autocompletes ALL github usernames.
- matai_kolila 4y ago
  > Note: the invite input box actually autocompletes ALL github usernames. I'm sorry, but that's wild. That's like, not even an easy engineering problem to solve necessarily, given their size!

greyface- 4y ago

> T-Connect enables features like remote starting, in-car Wi-Fi, digital key access, Could an attacker have remote-unlocked and remote-started the entire Toyota fleet with this access?

mcdwayne 4y ago
Toyota is claiming no, not with this leak. It was a partial repo that was exposed. The data they accessed with the key got customer ID numbers and emails only.

danielodievich 4y ago

Many years ago I got a trial license key for something, Aspose components of some sorts I think, and without thinking of it, checked it in into public Github repo. Well, few days later Aspose's support sends me a nicely worded note saying that they noticed that it was there and invalidated it for me. Their description and instructions were very clear about why they did it and why I shouldn't have checked it in. I thought that was very proactive and excellent customer service.

Natsukvshii 4y ago
I actually had something similar happen to me last month. I accidentally published a discord API key to GitHub and within minutes I got a nice message from “Safety Jim” to my personal discord account letting me know they’ve found my key on a public repo and have gone ahead and revoked it. I felt like a bit of a dope but it was neat to have it happen to me. Lesson learned for sure.
- greysteil 4y ago
  GitHub PM here. Glad that was a good experience! We work with ~50 partners (details in the link below) to notify them when tokens for their service are exposed in public repos, so that they can notify you. https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns#supported-secrets-for-partner-patterns https://docs.github.com/en/code-security/secret-scanning/sec...
  - fiddlerwoaroof 4y ago
    I wish I could set this up to block pushes proactively instead of reacting to pushed secrets.
    - kadoban 4y ago
      You could set up something like https://github.com/godaddy/tartufo https://github.com/godaddy/tartufo in a pre-commit hook. Not sure if github has a way to hook into the push hooks on server side, they might though.

chaps 4y ago

Hah. Yeah. Found a bunch of ssh keys, passwords, etc for Comcast years back which turned into a shitshow when I tried to report it. Once I found the right people to talk to things got better, but the entire experience was really reflective of how bad large orgs are with security. A friend once told me he was having a hard time getting a client to take his security concerns seriously. So I went on github and found a commit in their repo that included a production password and sent it to him. Maybe took 5-10 minutes to find? Apparently once they found out about the commit, they panicked a bit and started taking his concerns more seriously.

mcdwayne 4y ago
Yikes. It is sad to hear stories like that, where security is not a concern until panic sets in. :( Yet another reason we need to adopt standards like security.txt and make it easy to report these things as it is to tell robots to ignore us with robots.txt. See securitytxt.org for more on the project.
- dinvlad 4y ago
  I think the fundamental problem is, a lot of orgs just don't care about security, as it doesn't affect their bottom-line. Even breaches are only a temporary hit on the PR. Proper way to address that might just be legislation, with heavy fines based on total revenue. That and also security is just hard to scale. That's why if it was mandated by legislation, companies would be forced to spend a comparable amount on scaling their security teams and efforts.
- bisby 4y ago
  It's tough. I'm our public security reporting email list. We get a lot of things that boil down to "When I go to your website, I am able to see the content of your html files!" ... yes, reporter. That is what a web server does. It gives you HTML files. Congrats that you have figure out the dev console on your browser, but you're not a hacker. I'm trying to go with Hanlon's razor here and assume this is inexperienced people and not outright scams. We don't get a lot of these, but they far outweigh actual credible reports. But we try our best and take everything seriously until it can get disproven. And it's exhausting. So I get it sometimes. Sometimes having a place for responsible disclosure just opens yourself up to doing more paperwork (verifying that the fake reports are fake). That said, we still do it.

dinvlad 4y ago

I wish hosted GitHub made pre-push hooks available to the public. Would make this a much easier problem with free scanning tools like Trufflehog. Or alternatively, if GitHub Secret Scanning was available to all public repos, instead of requiring a (very) expensive GitHub Advanced Security subscription. But I understand, they need to make money somehow.

lumberjack24 4y ago
In the meantime, try ggshield cli https://github.com/GitGuardian/ggshield https://github.com/GitGuardian/ggshield
- dinvlad 4y ago
  Nah thanks, I'm already running Trufflehog for free on all of our multiple orgs' thousands of repos. I think we would consider GG if its pricing was acceptable for non-profits though.
  - mcdwayne 4y ago
    FYI - GitGuardian is free for individuals and teams smaller than 25
    - dinvlad 4y ago
      As told earlier, we have thousands of repos. And our teams are thousands of users on GH.
greysteil 4y ago
(GitHub PM here.) The Advanced Security secret scanning experience is coming to public repos (for free, obviously)! Give us a few more months - we have a little more work to do scaling it up
- dinvlad 4y ago
  Awesome, thanks for the heads-up! We will definitely look forward to it, either to replace or enhance our current in-house solution.

jrockway 4y ago

> Git is an awesome version control system, used by over 93% of developers, and is at the heart of modern CI/CD pipelines. One of the benefits of Git is that everyone has a complete copy of the project they are working on. I feel like this is copy-pasted from a pitch deck on why GitGuardian should be funded. Does anyone reading the article care about this anecdote? Like do people stop reading at "well I'm one of the 7% that doesn't" or think "wow a lot of people are using Git, I should buy their thing"? Sorry for the meta comment, but it just stuck out as odd to me.

dinvlad 4y ago
This is definitely a marketing piece. And they charge a LOT for it, so it's not a solution for the common masses.
- zricethezav 4y ago
  If using GitHub-Actions, Gitleaks offers competitive pricing for a secret scanning solution. https://gitleaks.io/products https://gitleaks.io/products
  - dinvlad 4y ago
    Thanks, I've already figured out how to run Trufflehog for free on our thousands of repos.
    - deleted 4y ago
      [deleted]
    - freeqaz 4y ago
      There is still a lot of noise with basic tools like this (I've also used trufflehog at scale). To properly handle secret scanning requires calling live APIs to test if keys are "real". And you need to have a way to file tickets when you do have findings... if you rotate a cred from production, that's now an outage, so you need to coordinate multiple teams. It's a lot of work and free tools only solve one part of this. I can't speak to any of the vendors in this space but I can attest that it's a harder problem than it seems!

bsimpson 4y ago

The social media manager at GitGuardian is winning. They just got us each to read a 1000 word ad for GitGuardian.

site-packages1 4y ago
Jokes on them, I never read the articles and only ever read the comments unless the comments indicate the article was interesting enough to read.
lordgrenville 4y ago
By writing an well-constructed and discussion-provoking article? "Mission fucking accomplished" https://xkcd.com/810/ https://xkcd.com/810/
TheRealDunkirk 4y ago
This. I used to use them, because I, too, have been burned by my own mistakes before. However, I had to stop using them as a plugin for GitHub because (at least when I was looking) there doesn't seem to be a way to exclude private repos from the reporting, and there were some false positives in private repos I don't want flagged. But I think the service is a good idea in general.

robswc 4y ago

It can actually be comical just how _bad_ things can be at large orgs. Anyone have details, theories, or a book on how such inefficiencies come about? I can't speak to tech-oriented large orgs but I've worked with others and its just... I'm not shocked at all. I've seen public facing API keys in HTML, private SSH keys that do god knows what in plaintext on FTP servers... I just don't understand how they seem to care so much but in reality, care so little. Just lazy?

nomel 4y ago
In my experience, security is so far removed from the actual job description/day to day cares that it's perpetually "somebody else's problem", seen as an unnecessary time sink. Usually there's a couple of people that actually care, but they're ignored and lack the power to influence change. Not lazy, just overburdened by more important things.
- robswc 4y ago
  That seems to match what I've seen. Especially the last line. It's just so weird the dynamic between "pretending to care" and "actually caring." I've worked at or consulted on small teams that were very "lax" about security but everyone seemed to take it seriously so it "worked." At larger orgs, I notice they take it "seriously" but that results in people finding creative loopholes... like pasting in all the important production keys/passwords into a google doc and sharing the google doc because they "can't send secrets over slack" ... lol
0xbadcafebee 4y ago
You're the receptionist at a high-security facility. Lots of people work there. Your job is to make sure the people get in and out quickly. The building has multiple methods of access at different stages. During the course of doing your job, you don't follow the proper protocols a few times. You closed a door without re-entering a lock code. You didn't check someone's badge and ID at the third floor access gate. You forgot to make sure someone swiped out on exit. Nobody actually trained you on any of those protocols; they just made you watch a "security is everyone's responsibility" video and then expected you to know all the protocols. Maybe once or twice you were lazy, but it's equally likely you just didn't know what to do. And once you make that mistake, somebody can exploit it as long as you don't follow the protocol. In facilities where security is important, there are safeguards. Doors left open sound an alarm, automatically lock when closed, and sensors detect if more than one person walks through without badging in. These safeguards exist because people are fallible and need help to enforce security. Companies without security safeguards either don't care, or are ignorant.

zricethezav 4y ago

Good reminder to run Gitleaks or Gitleaks-Action on your repos - https://github.com/zricethezav/gitleaks https://github.com/zricethezav/gitleaks - https://gitleaks.io/products https://gitleaks.io/products

eax-eip 4y ago

Most of these (even sometimes expensive) tools only look at repos and users who are associated with the company’s GitHub org, which barely solves the problem. The much harder problem is the number of corporate secrets that are on random repositories (personal dotfiles, automations, data science scripts, etc.) across GitHub with no strong relationship to the organization. Try using GitHub Code Search to find all the Fastly API tokens that have been leaked, for example, and I bet you’d find some wild stuff.

lumberjack24 4y ago
GitGuardian actually does this, it monitors an extended perimeter of devs and their personal/open-source repos for corporate secrets or keywords – https://www.gitguardian.com/monitor-public-github-for-secrets https://www.gitguardian.com/monitor-public-github-for-secret...
- sidewndr46 4y ago
  Yeah, because as an employee what I totally want is my employer to monitor my every digital move outside of work!
  - deleted 4y ago
    [deleted]
  - elguyosupremo 4y ago
    Make a private repo. I wouldn't blame a corp if they tried to scan every public github repo for their API keys, let alone an employee's public account.

userbinator 4y ago

Too bad this wasn't for their ECUs or something else that could've benefited right-to-repair.

ajsnigrutin 4y ago

Right to repair bad, independent repair centers will expose your private data! - Toyota ( reference https://www.youtube.com/watch?v=GTiLnz23TNs https://www.youtube.com/watch?v=GTiLnz23TNs )

Kiro 4y ago

Question: Let's say I want to open source my app but a long time ago I used to have credentials hard coded. What can I do to clean this up from history?

scarmig 4y ago
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository https://docs.github.com/en/authentication/keeping-your-accou...
tyler569 4y ago
Git does have tools that allow you to rewrite history to fix situations like this, but by far the easiest solution is to invalidate those credentials so they become worthless.
lalopalota 4y ago
make a new repo starting with the code in its current state.
lumberjack24 4y ago
Here's a checklist [1] (again, from gitguardian) of steps to follow before open-sourcing projects and [2] a guide on how to remediate hardcoded/exposed secrets. [1] https://blog.gitguardian.com/safely-open-source-software-best-practices https://blog.gitguardian.com/safely-open-source-software-bes... [2] https://blog.gitguardian.com/leaking-secrets-on-github-what-to-do/ https://blog.gitguardian.com/leaking-secrets-on-github-what-...

fny 4y ago

Is there any reason why keys don't constantly update? It seems like a service could exist where every five minutes a rotation occurs across services with decaying privileges. For example, the 5 minute old key still works, but the 10 minute woman has completely expired.

crazygringo 4y ago
How would you update the keys every 5 min? Either you'd use an encryption algorithm that depended on a "deeper" key... or you'd fetch the new key while authenticated with, you guessed it, another "deeper" key. It's keys all the way down. Every key you use, it's your responsibility to keep it private. (Unless you want to be dealing with physical hardware dongles that generate keys, but those aren't exactly easily portable.)
- hnick 4y ago
  Not exactly the same but I had to integrate with a payments API which required one call to an auth endpoint with user/password under HTTP basic auth to get an access token. The actual calls to calculate and execute payments used this access token instead as a bearer token. Those ones expired. I'm not sure I really see the point, but I guess you could lock down the main call on another system, store the temporary token somewhere, and have the other systems use it. As you say, you still need the top level credentials somewhere to get a new token.
  - windowsworkstoo 4y ago
    Yah, that's the OAuth Client Credentials flow but as noted, you still have a static set of creds that are required to generate the short lived access token. Besides being useful for being able to limit scope in some circumstances, the main point of the client cred flow is to appease eager sec arch's who insist on OAuth.
- deleted 4y ago
  [deleted]
- Too 4y ago

foxandmouse 4y ago

Was this title re-written? I feel it was phrased more vaguely when I first read it, but I could be wrong... More curious about if that's a thing hackerNews does.

sen 4y ago
Titles are edited to match the article usually.
EE84M3i 4y ago
It was "Toyota Accidently Exposed a Secret Key Publicly on GitHub for Five Years". Thank you https://hackernewstitles.netlify.app/ https://hackernewstitles.netlify.app/

beyang 4y ago

In case it's helpful, here's a bookmarklet that instantly searches any public repository for the most common secret patterns: https://about.sourcegraph.com/blog/no-more-secrets https://about.sourcegraph.com/blog/no-more-secrets. Sourcegraph built a feature called "code monitors" that basically runs a recurring search in the background to guard against anti-patterns like API keys and secrets being committed into the codebase.

gumby 4y ago

Why would Toyota need to have that info to provide service? We need to get PII as a liability on companies’ balance sheets to get them to take it seriously and to collect only the minim viable data.

jaywalk 4y ago
Why would they need to have > customer identification numbers and emails to provide service? Seriously?
- gumby 4y ago
  Yes, seriously. They need to service the vehicle, OK, but communicate to and through the vehicle.
  - 8organicbits 4y ago
    What does that mean? The repair tech tells my car that it's ready for pickup and my car ignores him because it's a car not a communication channel?
    - gumby 4y ago
      I can tell my local repair guy down the road my number. No need for some megacorp to know who I am.
  - PebblesRox 4y ago
    These are the services, according to the article: “T-Connect enables features like remote starting, in-car Wi-Fi, digital key access, full control over dashboard-provided metrics, as well as a direct line to the My Toyota service app. The servers that control these options contain unique customer identification numbers and customer emails.”
    - gumby 4y ago
      I don't see why any of that should require the email address. They can communicate with the customer through the app or through the car UI. In general, apps and sites these days hoover up more info than they need simply because they can, not because it adds to the customer experience (and often doesn't help the company either). There is no incentive to be in any way judicious about what to collect and the frequent breaches show that even the companies don't value PII as something worth protecting because it's not core to their business.

rainbringer2000 4y ago

production code + keys = no bueno. I feel for them.

deleted 4y ago
[deleted]

juancampa 4y ago

To prevent accidental leaks perhaps Github et al could implement a mechanism analogous to the Eurion constellation [0]. E.g. A commit containing a file with a specific text pattern should always be rejected. Most fileypes accept arbitrary comments (ahem Json) so it should work for most files. 0: https://en.m.wikipedia.org/wiki/EURion_constellation https://en.m.wikipedia.org/wiki/EURion_constellation

hn_throwaway_99 4y ago

This is why you should never, never, never check production keys into git. There are tons of secret manager services out there. All the major cloud services have them and even companies like 1Password now have them.

stall84 4y ago

welp .. that makes all the times my n00b duma$$ did it not feel so bad.

GolDDranks 4y ago

There are software such as Trufflehog ( https://github.com/trufflesecurity/trufflehog https://github.com/trufflesecurity/trufflehog ), that find secrets. We are using it at an organizational level, but there's always some delay from finding something and getting it reported. I've been meaning to add it both to our CI so our team can notice right away, and even to Git push hooks, to catch these cases early.

deleted 4y ago

[deleted]

ghostbuster09 4y ago

We’ve been using Nightfall to protect against leaking secrets and other sensitive data across github and other apps https://nightfall.ai/solutions/product/github https://nightfall.ai/solutions/product/github

paradite 4y ago

I have an idea: What if companies using secret keys (not SaaS providers who generate secret keys) voluntarily send their secret keys (MD5 hashed or somehow encrypted) to GitHub and GitHub can then monitor their leakage and notify the company? Is this useful or feasible?

devcat 4y ago
Github already has system to invalidate tokens without storing any hashes themselves https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns https://docs.github.com/en/code-security/secret-scanning/sec...
- paradite 4y ago
  Yes. This is dependent on some kind of agreement or understanding between SaaS provider and GitHub. I'm thinking of a generic approach which is independent of where you got the secret key and its format.
  - devcat 4y ago
    Same system can be applied for repo owner being pattern provider and getting notified with matches but if i had to guess only enterprise customers might get feature like this

AtNightWeCode 4y ago

> A credential for a DB server holding customer data was hardcoded into the repo. This is normally not a problem since databases should not be exposed to everyone one the Internet. > One of the benefits of Git is that everyone has a complete copy of the project they are working on. This is one of the problems with GIT and many programming langs. Because of this full copies of entire code bases are everywhere. GIT also only has ACL on repo level. In many langs it is recommended to not store credentials in the code but at the same time there are no guidelines for how to store credentials. If you use cloud services there are recommendations. Edit: Btw, which git service where they using? Cause I believe Github and others monitor code bases for accidental pushes of secrets.

Gibbon1 4y ago
I tool I would like, and likely linker magic could handle this, is a standard tool to patch elf files with things like revision info and keys[1]. [1] Notable, friend in the security biz had the advice that stored keys and passwords should never just work.

koksik202 4y ago

My access got revoked for MyT app and then few days later I was able to bind my vehicle again to profile I wonder if that was caused by this

atoav 4y ago

Github is amazing. Wanna get expensive licensed fonts for free? Just search for $FONTNAME.otf github and you will find at least a few projects using it. People don't think a lot about what they put on there it seems. Or maybe font foundaries haven't sued enough at this point.

Paul_S 4y ago

I worked for a couple corps in my life and the security theatre in here is ridiculous. At the same time things like this happen all the freaking time. And none of the company spyware can catch that. Funny how this never happened in small companies I worked for, although maybe that's just a function of number of people? Or maybe because small companies rely on fewer platforms and frameworks so fewer chances of such leaks?

withinboredom 4y ago
In my experience, smaller companies tend to hire the best they can attract and afford. Larger companies tend to collect talent so no one else can get it, so it usually isn't "the best."

isolli 4y ago

Reminds me of a related HN discussion a few years ago: someone searched for "remove password" in GitHub and unearthed who knows how many valid passwords in the hundreds of thousands of commits that the search returned... [0] https://news.ycombinator.com/item?id=13650818 https://news.ycombinator.com/item?id=13650818

dzhiurgis 4y ago

Apparently tons of secrets get leaked to the public Postman recipes library

bxtt 4y ago

Ah. I can’t believe this still happens in this day of age. About a decade ago, I was working for a startup and we were getting dominated in our growing space by a much larger, well funded rival. Our competitive intelligence team browsed through their git, and the rival actually exposed access to their customer, pricing and sales agent database by leaving their credentials in one of their branches. The team went to our legal department asking if they can be protected by the company, and if they can use this intel. The team then worked with the product team to integrate all their pricing engines to our POS to undercut their pricing and sent marketing blasts to their leads with targeted marketing campaigns. Long story short that company is now defunct, and it definitely undermined their growth.

joshxyz 4y ago
and that folks, is how you do growth hacking ;)
empyrrhicist 4y ago
If you're in the US, that's 100% a crime. If they were responding to an unauthenticated API or web request that's one thing, but using a leaked password on a database is not legal at all.
- rtev 4y ago
  Yeah, 3+ years in prison for that. Corporate espionage hacking is real.

pknerd 4y ago

Surprised that they do not the likes of .env files which are usually not pushed to a repo.

vivan 4y ago

I used to report things like this that I had found, including cases where I can see people used the default "sample" config for security purposes, but I found that either people would not care at all, or massively overreact and somehow blame me. If an organisation is disorganised enough to leave critical details in public, they're probably too disorganised to handle someone reporting it.

pojzon 4y ago

Some time ago on devops reddit someone asked „What is the biggest vulnerability in your system?” Most common answer was „Developers” It do be like that. Without proper automation and processes and education of ppl involved it can get really bad.