5 ms·

These secret scanning integrations have been very helpful. We had a client ask to take a project open source recently that had started a few years ago as closed

These secret scanning integrations have been very helpful. We had a client ask to take a project open source recently that had started a few years ago as closed source. We of course checked over the current version of the code and have had linters in place to look for secrets for a while but not in the very early days of the project. In that one codebase we had: - AWS IAM token for S3 upload access to a throwaway dev bucket. The bucket had already been deleted but still... Got an email about it informing me the IAM token had been revoked by AWS within 5 minutes - A Slack webhook notification URL/secret. Committed as a example on a working branch and then git rm'ed but still active. Got an email about it and token revoked by Slack automatically within 5 minutes. - A Mapbox API token. This one was funny. The token was indeed in there and functional but was in the docs/sample code for a dependency. Still, we got an email within the hour about it and were able to investigate. Edit: In this case we intentionally kept the commit history. A safer alternative (and one we normally practice) is to start a fresh repo for the open source variant.

ed25519FUUU 6y ago
An overlooked vector is old commits. It’s often times better to squash all commits before taking a project open source, which is a real shame for obvious reasons. Commit histories can spill a lot of secrets that are easy to overlook.
- _the_inflator 6y ago
  Absolutely this! Same problem here with inner source, that goes open source. I feel sorry for all our internal committers, however I know of "secrets", that went into the commit history. We are still considering our option, but tend to opt for deleting our commit history entirely and build a wall of fame for the former committers.
- psanford 6y ago
  There are tools available to help look for this sort of thing (for both you and any potential attackers). TruffleHog[1] is the first one that comes to mind for me. I also like shhgit[2] for looking for secrets in repositories. (I don't think shhgit will look back in the git history for you though). [1]: https://github.com/dxa4481/truffleHog https://github.com/dxa4481/truffleHog [2]: https://github.com/eth0izzle/shhgit https://github.com/eth0izzle/shhgit
  - amichal 6y ago
    Thanks! I knew they existed but hadn't investigated for one that would look over past history. Will try out truffleHog.
    - psanford 6y ago
      I've found the entropy detection in trufflehog to be pretty noisy. When I've run it I generally disable that.
  - lstamour 6y ago
    Another idea is to use a git commit hook, such as https://github.com/cloud-gov/caulking https://github.com/cloud-gov/caulking
- jgalt212 6y ago
  My current fear is versioning back up systems. KeePass files may now have secure master keys, but maybe the version saved 18 mos ago did not. 1. Get an old copy 2. run dictionary attack 3. prosper
anderskaseorg 6y ago
When I helped to take Zulip open-source in 2015, I wrote a simple script that scrubbed secrets from the commit history using git fast-export and git fast-import. We replaced all our secrets with xxxxxxx placeholders, replaced internal customer references with dummy names, deleted and renamed certain files, and even did some code replacements that caused certain commit diffs to become empty so those commits could be removed from the history. https://github.com/zulip/zulip/blob/3.3/tools/zanitizer https://github.com/zulip/zulip/blob/3.3/tools/zanitizer https://github.com/zulip/zulip/blob/3.3/tools/zanitizer_config.pm.sample https://github.com/zulip/zulip/blob/3.3/tools/zanitizer_conf... The script was really fast (all ~10000 commits in a few minutes), which allowed us to iterate quickly on its configuration as we audited using gitk and other tools for remaining items to scrub. Doing this work allowed us to release with an essentially complete history going back to the first commit in 2012, which has been a really valuable resource for understanding why various Zulip subsystems were written the way they were. Nowadays there are other tools for scrubbing history that might be more polished, like BFG: https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/
- amichal 6y ago
  Nice tooling. I've used bfg when we knew what patterns to look for. This project didn't generally access private data, had a reasonably well behaved team for most of its life (the pre-linter & code-review commits were my own damn fault). Since it was low risk, I just did a few manual `git log -S ...` and moved on. I was still very happy to have github catch my throwaway credentials and remind me in the most obvious way that these things go in `ENV` and not IN code even in examples!
danudey 6y ago
> A safer alternative (and one we normally practice) is to start a fresh repo for the open source variant. Note that it's also possible to go back and rewrite history (e.g. if you know what the tokens are and where/when they were committed), to preserve Git history while cleaning out tokens. It can be mildly slow or complicated, but there are tools to automate it, such as BFG Repo Cleaner[0] which is relatively easy to use (once you learn it). There are other awesome rewriting tools, like git filter-repo[1], but that operates solely on the structure of the repository (i.e. it can manipulate basically anything except file contents). Great for removing unwanted files or directories extremely fast, but not good for removing tokens (unless you want to remove the entire file the token was in). [0] https://rtyley.github.io/bfg-repo-cleaner/ [1] https://github.com/newren/git-filter-repo
- amichal 6y ago
  Learning so many options from this thread. I've used these tools when I knew what to look for but thats been the tricky bit. psanford also mentioned truffleHog and others, lstamour mentioned https://github.com/cloud-gov/caulking https://github.com/cloud-gov/caulking which is built on gitleaks which looks good. caulking's customized list of patterns for gitleaks is here https://github.com/cloud-gov/caulking/blob/master/local.toml https://github.com/cloud-gov/caulking/blob/master/local.toml Looks like it would have found the keys in my example case no problem.
- deleted 6y ago
  [deleted]
- nyanpasu64 6y ago
  I've found that git filter-repo actually can modify file contents, using --replace-text and a file containing replacements.

ed25519FUUU 6y ago

An overlooked vector is old commits. It’s often times better to squash all commits before taking a project open source, which is a real shame for obvious reasons. Commit histories can spill a lot of secrets that are easy to overlook.

_the_inflator 6y ago
Absolutely this! Same problem here with inner source, that goes open source. I feel sorry for all our internal committers, however I know of "secrets", that went into the commit history. We are still considering our option, but tend to opt for deleting our commit history entirely and build a wall of fame for the former committers.
psanford 6y ago
There are tools available to help look for this sort of thing (for both you and any potential attackers). TruffleHog[1] is the first one that comes to mind for me. I also like shhgit[2] for looking for secrets in repositories. (I don't think shhgit will look back in the git history for you though). [1]: https://github.com/dxa4481/truffleHog https://github.com/dxa4481/truffleHog [2]: https://github.com/eth0izzle/shhgit https://github.com/eth0izzle/shhgit
- amichal 6y ago
  Thanks! I knew they existed but hadn't investigated for one that would look over past history. Will try out truffleHog.
  - psanford 6y ago
    I've found the entropy detection in trufflehog to be pretty noisy. When I've run it I generally disable that.
- lstamour 6y ago
  Another idea is to use a git commit hook, such as https://github.com/cloud-gov/caulking https://github.com/cloud-gov/caulking
jgalt212 6y ago
My current fear is versioning back up systems. KeePass files may now have secure master keys, but maybe the version saved 18 mos ago did not. 1. Get an old copy 2. run dictionary attack 3. prosper

anderskaseorg 6y ago

When I helped to take Zulip open-source in 2015, I wrote a simple script that scrubbed secrets from the commit history using git fast-export and git fast-import. We replaced all our secrets with xxxxxxx placeholders, replaced internal customer references with dummy names, deleted and renamed certain files, and even did some code replacements that caused certain commit diffs to become empty so those commits could be removed from the history. https://github.com/zulip/zulip/blob/3.3/tools/zanitizer https://github.com/zulip/zulip/blob/3.3/tools/zanitizer https://github.com/zulip/zulip/blob/3.3/tools/zanitizer_config.pm.sample https://github.com/zulip/zulip/blob/3.3/tools/zanitizer_conf... The script was really fast (all ~10000 commits in a few minutes), which allowed us to iterate quickly on its configuration as we audited using gitk and other tools for remaining items to scrub. Doing this work allowed us to release with an essentially complete history going back to the first commit in 2012, which has been a really valuable resource for understanding why various Zulip subsystems were written the way they were. Nowadays there are other tools for scrubbing history that might be more polished, like BFG: https://rtyley.github.io/bfg-repo-cleaner/ https://rtyley.github.io/bfg-repo-cleaner/

amichal 6y ago
Nice tooling. I've used bfg when we knew what patterns to look for. This project didn't generally access private data, had a reasonably well behaved team for most of its life (the pre-linter & code-review commits were my own damn fault). Since it was low risk, I just did a few manual `git log -S ...` and moved on. I was still very happy to have github catch my throwaway credentials and remind me in the most obvious way that these things go in `ENV` and not IN code even in examples!

danudey 6y ago

> A safer alternative (and one we normally practice) is to start a fresh repo for the open source variant. Note that it's also possible to go back and rewrite history (e.g. if you know what the tokens are and where/when they were committed), to preserve Git history while cleaning out tokens. It can be mildly slow or complicated, but there are tools to automate it, such as BFG Repo Cleaner[0] which is relatively easy to use (once you learn it). There are other awesome rewriting tools, like git filter-repo[1], but that operates solely on the structure of the repository (i.e. it can manipulate basically anything except file contents). Great for removing unwanted files or directories extremely fast, but not good for removing tokens (unless you want to remove the entire file the token was in). [0] https://rtyley.github.io/bfg-repo-cleaner/ [1] https://github.com/newren/git-filter-repo

amichal 6y ago
Learning so many options from this thread. I've used these tools when I knew what to look for but thats been the tricky bit. psanford also mentioned truffleHog and others, lstamour mentioned https://github.com/cloud-gov/caulking https://github.com/cloud-gov/caulking which is built on gitleaks which looks good. caulking's customized list of patterns for gitleaks is here https://github.com/cloud-gov/caulking/blob/master/local.toml https://github.com/cloud-gov/caulking/blob/master/local.toml Looks like it would have found the keys in my example case no problem.
deleted 6y ago
[deleted]
nyanpasu64 6y ago
I've found that git filter-repo actually can modify file contents, using --replace-text and a file containing replacements.