5 ms·

What's the best way to prevent accidental github leaks? I once accidentally leaked an API key (albeit for a relatively unimportant service, and I realized soon

What's the best way to prevent accidental github leaks? I once accidentally leaked an API key (albeit for a relatively unimportant service, and I realized soon afterwards).

solotronics 6y ago
A good key rotation scheme and separate API keys for different purposes. Precommit automated checks for secrets.
- john-tells-all 6y ago
  Absolutely! - a good, tested key rotation practice will reduce the impact of any accidental exposures. It'll also help tremendously when staff changes, and you can do things like "rotate all keys every quarter" - separate API keys for different services and environments means that even if your staging network-layer key is exposed, it can't affect your staging database (= data), nor your production database (= critical data). - I've used "search for AWS secrets" as a pre-commit and also CI hook with great luck. I've even accidentally added a secret myself :) but with the checks the error was immediately found. No problems!
ChrisSD 6y ago
I'm not sure there's a "best way" aside from being vigilant. * Don't blindly add everything to git. * Check staged files before commit. * Store secrets separately, outside of your git folder (e.g. use another folder, service or environment variables). * If it must be in your git folder, then make use of .gitignore and store keys, etc separately from code in files with a unique file name pattern.
hackerpain 6y ago
For basic live monitoring -- https://www.shhgit.com/ https://www.shhgit.com/ It's possible to monitor them while pushing code, and to have a monitoring system deployed to track them. I was thinking of open-sourcing our GitHubSniper tool, it basically has API regex from nearly a thousand SaaS and PaaS services and frequently posted sensitive strings, can be deployed via docker ;) The only thing that stops me from releasing it is I have consistently earned more in bounties than the incentives of actually releasing it publicly. But will eventually release it :)
- mellosouls 6y ago
  That looks both cool and dodgy at the same time - making potential security breaches immediately available to anybody looking at it is rather different from offering it to people as a private security check service.
- albanberg 6y ago
  sshgit: How does the notification work? Only on desktop? Nevermind...I found the docs. :-)
- deleted 6y ago
  [deleted]
beaconstudios 6y ago
Don't include any credentials in your code, pull them in through AWS secrets, environment variables, or other runtime configuration.
gostsamo 6y ago
Code reviews. If it is in the diff, you have an issue, so make someone check your work. Otherwise, secrets as environment variables, do not write them down in files, and change them regularly.
lock-free 6y ago
Assume that secrets are going to be leaked or discovered and change them regularly. Everything else is a delay tactic, people mess up (or get tricked into messing up).
- justin_oaks 6y ago
  You can change secrets regularly, but if you don't change them often enough then you can still be compromised because of it. If you change them too often then it's a maintenance problem. You might even take an outage because you have mismatched credentials (i.e. application or database had the database credentials updated but the other one didn't) Changing them regularly IS necessary. So if you're going to do it you should use a credential service like Hashicorp Vault.
pwdisswordfish0 6y ago
> What's the best way to prevent accidental github leaks? Don't use GitHub. Guaranteed to have 100% efficacy.
acdha 6y ago
Pre-commit.com has built in checks for AWS and SSH keys. I add this hook which checks for passwords, high-entropy strings, etc. and has tools for managing false positives: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
montjoy 6y ago
Run your own git server behind a VPN. It still won’t prevent people from doing deliberately stupid things but will cut down on accidents. If you are super paranoid you air-gap everything.
BeefWellington 6y ago
Detect-Secrets, TruffleHog, or GitLeaks configured as a pre-commit hook in your local git repo to block you from committing them at all. Not perfect but gets you maybe 80-90% of the way.
0xbadcafebee 6y ago
This blog post by Duo Labs explains everything you need to know. It describes the problem in detail, breaks it down, and explains mitigations. Everyone should read this. https://duo.com/labs/research/how-to-monitor-github-for-secrets https://duo.com/labs/research/how-to-monitor-github-for-secr... Here is their comparison of pre-commit hooks and other tools. It's current and comprehensive. https://github.com/duo-labs/secret-bridge/blob/master/TOOLS.md https://github.com/duo-labs/secret-bridge/blob/master/TOOLS....
tialaramex 6y ago
Don't write keys into code, even "test" code. Discipline yourself (and your team) to use dedicated key storage even on test systems, (and of course that key storage isn't "Duh, a github repo" but an appropriate technology for your platform) If you instinctively want to write shiny_api_key = 'SOME.KEY.BOSS.EMAILED.ME' Force yourself to instead take the extra few minutes to do # my-cloud-vendor key-store create shiny_api_key SOME.KEY.BOSS.EMAILED.ME and then write shiny_api_key = key_store_access('shiny_api_key') This way, when six hours later you check that code in, to GitHub, and go home for the weekend, you won't get a call from the boss ruining your Sunday lie-in, "WTF Did you upload our $5000 shiny API key to GitHub? You are fired". This habit also allows for correct separation as an organisation grows, when you're big enough you want it to be possible for somebody to write code against test systems with test API keys, and that same code to work as written in production, even though production API keys are not accessible to the coders. Using key stores to mediate makes this much more likely to actually work.
- 0xbadcafebee 6y ago
  Don't know why you were downvoted, this is 100% what everyone should be doing. Don't even write shiny_api_key = "A-CLEARLY-FAKE-KEY". Don't write ANY secret AT ALL in your code, no matter what. Even for test frameworks. Just don't do it. Build in the secret retriving function from day one. Have it grab from an environment variable and it will be 100% seamless from development to production. Add more complexity/security if and when you need it. This can later be leveraged to allow for zero-downtime credential rotations for running services, which allows for seamless use of ephemeral credentials, which is what we should all be moving towards.
- yowlingcat 6y ago
  This is the wrong answer. At a company I worked at, we had a well known saying: "Favor mechanisms over good intentions." As the saying went, if good intentions worked, we "wouldn't need mechanisms." The right answer here (as stated in sibling comments) is to install a mechanism into the build. A push that accidentally commits credentials should break the build. How it goes about doing that is up to you, but the specific way I've done that in the past is a git hook [1] specifically the pre-commit hook. It looks like there's a reasonably well-updated package by Yelp to help do this kind of thing [2]. [1] https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks [2] https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
tetha 6y ago
Hm, we have multiple layers of protection of different kinds there, since defense should always be in depth. First off, I'm pushing the principle that secrets should only be issued, created, stored and distributed by the secret management system, vault in our case. If you never see a secret, and only instruct the automation how to handle it, it is much harder to leak the secret. Ideally, in production, no human should ever need to see or access the secret. Second, you should have good rotation procedures. Something will eventually leak, or someone will leave on bad terms and you need to be able to handle that. This can be simple - an HA infra service can just reload the core credentials from vault and be done, or somewhat harder, e.g. if customers need and use those passwords. And finally, you should have entropy scanners like trufflehog or github scanners as linked in other tools - both in you precommit hooks (so you cannot commit the secret) as well as in your CI & repository host, so someone who disabled the security scanner "because it was annoying" still gets caught. And I guess finally, whiskey for that friday evening someone still manages to leak something important. Some terminations at work have not been fun.
vangelis 6y ago
Using an external config source helps. It's harder to accidentally check in 'secrets.json' than an API key in code.

What's the best way to prevent accidental github leaks? I once accidentally leaked an API key (albeit for a relatively unimportant service, and I realized soon afterwards).

solotronics 6y ago

A good key rotation scheme and separate API keys for different purposes. Precommit automated checks for secrets.

john-tells-all 6y ago
Absolutely! - a good, tested key rotation practice will reduce the impact of any accidental exposures. It'll also help tremendously when staff changes, and you can do things like "rotate all keys every quarter" - separate API keys for different services and environments means that even if your staging network-layer key is exposed, it can't affect your staging database (= data), nor your production database (= critical data). - I've used "search for AWS secrets" as a pre-commit and also CI hook with great luck. I've even accidentally added a secret myself :) but with the checks the error was immediately found. No problems!

ChrisSD 6y ago

I'm not sure there's a "best way" aside from being vigilant. * Don't blindly add everything to git. * Check staged files before commit. * Store secrets separately, outside of your git folder (e.g. use another folder, service or environment variables). * If it must be in your git folder, then make use of .gitignore and store keys, etc separately from code in files with a unique file name pattern.

hackerpain 6y ago

For basic live monitoring -- https://www.shhgit.com/ https://www.shhgit.com/ It's possible to monitor them while pushing code, and to have a monitoring system deployed to track them. I was thinking of open-sourcing our GitHubSniper tool, it basically has API regex from nearly a thousand SaaS and PaaS services and frequently posted sensitive strings, can be deployed via docker ;) The only thing that stops me from releasing it is I have consistently earned more in bounties than the incentives of actually releasing it publicly. But will eventually release it :)

mellosouls 6y ago
That looks both cool and dodgy at the same time - making potential security breaches immediately available to anybody looking at it is rather different from offering it to people as a private security check service.
albanberg 6y ago
sshgit: How does the notification work? Only on desktop? Nevermind...I found the docs. :-)
deleted 6y ago
[deleted]

beaconstudios 6y ago

Don't include any credentials in your code, pull them in through AWS secrets, environment variables, or other runtime configuration.

gostsamo 6y ago

Code reviews. If it is in the diff, you have an issue, so make someone check your work. Otherwise, secrets as environment variables, do not write them down in files, and change them regularly.

lock-free 6y ago

Assume that secrets are going to be leaked or discovered and change them regularly. Everything else is a delay tactic, people mess up (or get tricked into messing up).

justin_oaks 6y ago
You can change secrets regularly, but if you don't change them often enough then you can still be compromised because of it. If you change them too often then it's a maintenance problem. You might even take an outage because you have mismatched credentials (i.e. application or database had the database credentials updated but the other one didn't) Changing them regularly IS necessary. So if you're going to do it you should use a credential service like Hashicorp Vault.

pwdisswordfish0 6y ago

> What's the best way to prevent accidental github leaks? Don't use GitHub. Guaranteed to have 100% efficacy.

acdha 6y ago

Pre-commit.com has built in checks for AWS and SSH keys. I add this hook which checks for passwords, high-entropy strings, etc. and has tools for managing false positives: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets

montjoy 6y ago

Run your own git server behind a VPN. It still won’t prevent people from doing deliberately stupid things but will cut down on accidents. If you are super paranoid you air-gap everything.

BeefWellington 6y ago

Detect-Secrets, TruffleHog, or GitLeaks configured as a pre-commit hook in your local git repo to block you from committing them at all. Not perfect but gets you maybe 80-90% of the way.

0xbadcafebee 6y ago

This blog post by Duo Labs explains everything you need to know. It describes the problem in detail, breaks it down, and explains mitigations. Everyone should read this. https://duo.com/labs/research/how-to-monitor-github-for-secrets https://duo.com/labs/research/how-to-monitor-github-for-secr... Here is their comparison of pre-commit hooks and other tools. It's current and comprehensive. https://github.com/duo-labs/secret-bridge/blob/master/TOOLS.md https://github.com/duo-labs/secret-bridge/blob/master/TOOLS....

tialaramex 6y ago

Don't write keys into code, even "test" code. Discipline yourself (and your team) to use dedicated key storage even on test systems, (and of course that key storage isn't "Duh, a github repo" but an appropriate technology for your platform) If you instinctively want to write shiny_api_key = 'SOME.KEY.BOSS.EMAILED.ME' Force yourself to instead take the extra few minutes to do # my-cloud-vendor key-store create shiny_api_key SOME.KEY.BOSS.EMAILED.ME and then write shiny_api_key = key_store_access('shiny_api_key') This way, when six hours later you check that code in, to GitHub, and go home for the weekend, you won't get a call from the boss ruining your Sunday lie-in, "WTF Did you upload our $5000 shiny API key to GitHub? You are fired". This habit also allows for correct separation as an organisation grows, when you're big enough you want it to be possible for somebody to write code against test systems with test API keys, and that same code to work as written in production, even though production API keys are not accessible to the coders. Using key stores to mediate makes this much more likely to actually work.

0xbadcafebee 6y ago
Don't know why you were downvoted, this is 100% what everyone should be doing. Don't even write shiny_api_key = "A-CLEARLY-FAKE-KEY". Don't write ANY secret AT ALL in your code, no matter what. Even for test frameworks. Just don't do it. Build in the secret retriving function from day one. Have it grab from an environment variable and it will be 100% seamless from development to production. Add more complexity/security if and when you need it. This can later be leveraged to allow for zero-downtime credential rotations for running services, which allows for seamless use of ephemeral credentials, which is what we should all be moving towards.
yowlingcat 6y ago
This is the wrong answer. At a company I worked at, we had a well known saying: "Favor mechanisms over good intentions." As the saying went, if good intentions worked, we "wouldn't need mechanisms." The right answer here (as stated in sibling comments) is to install a mechanism into the build. A push that accidentally commits credentials should break the build. How it goes about doing that is up to you, but the specific way I've done that in the past is a git hook [1] specifically the pre-commit hook. It looks like there's a reasonably well-updated package by Yelp to help do this kind of thing [2]. [1] https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks [2] https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets

tetha 6y ago

Hm, we have multiple layers of protection of different kinds there, since defense should always be in depth. First off, I'm pushing the principle that secrets should only be issued, created, stored and distributed by the secret management system, vault in our case. If you never see a secret, and only instruct the automation how to handle it, it is much harder to leak the secret. Ideally, in production, no human should ever need to see or access the secret. Second, you should have good rotation procedures. Something will eventually leak, or someone will leave on bad terms and you need to be able to handle that. This can be simple - an HA infra service can just reload the core credentials from vault and be done, or somewhat harder, e.g. if customers need and use those passwords. And finally, you should have entropy scanners like trufflehog or github scanners as linked in other tools - both in you precommit hooks (so you cannot commit the secret) as well as in your CI & repository host, so someone who disabled the security scanner "because it was annoying" still gets caught. And I guess finally, whiskey for that friday evening someone still manages to leak something important. Some terminations at work have not been fun.

vangelis 6y ago

Using an external config source helps. It's harder to accidentally check in 'secrets.json' than an API key in code.