21 ms·

Cool experiment! I PM the secret scanning team at GitHub and wanted to mention what GitHub did behind the scenes here. GitHub scans every commit to a public re

Cool experiment! I PM the secret scanning team at GitHub and wanted to mention what GitHub did behind the scenes here. GitHub scans every commit to a public repo for secrets one of our secret scanning partners may have issued. We forward those candidate secrets to the issuing partner, and they take action. In some cases they auto-revoke the secret (AWS normally does this, I believe), in some cases they notify the user, and in some cases the response is configurable. I checked that GitHub detects these tokens myself - within 1 second of the commit GitHub had notified AWS and Slack of the leak. AWS and Slack will then have taken action and informed the token owner, which in this case is Thinkst Canary, rather than Andrezj (the OP). I believe AWS normally auto-revoke, but they may have a custom setup with Thinkst Canary's tokens that allows Thinkst to continue to monitor them even once compromised. Finally, GitHub actually delays the indexing of our search by a couple of seconds to ensure that, for normal cases, our secret scanning partners have time to take action before anyone else can find the tokens. We're always looking to make secret scanning at GitHub better, so feedback as always welcome. It's also fascinating (and validating!) to see what happens to exposed tokens. * List of GitHub secret scanning partners: https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/about-secret-scanning#about-secret-scanning-for-public-repositories https://docs.github.com/en/free-pro-team@latest/github/admin... * Thnkst Canary tokens: https://www.canarytokens.org/ https://www.canarytokens.org/

0xad 6y ago
Awesome, thanks for the background information!
tobr 6y ago
Why not refuse to publish a detected secret at all until the repo owner takes an action to allow it?
- watt 6y ago
  Think about it in terms of incentives and nudges.
- greysteil 6y ago
  There are a few considerations on that one, but one very practical reason is the developer experience of dealing with false positives. False positives are one of the big problems in secret scanning. Some partners issue credentials with patterns that make them very hard to distinguish from innocuous strings. For example, a Datadog token looks identical to a commit SHA. We would never block developers from pushing commits to GitHub just because they had 40 character hexadecimal strings in them! GitHub's partnership approach works around the false positive problem by having the token issuer check whether a token is real and take action only if it is. However, this is a one-way communication from GitHub - the token issuer doesn't need to tell us whether the candidate secret we sent them was real or not, and in most cases we never know. As a result, we can't replicate the zero false positive experience in a pre-receive hook (i.e., before the commit is pushed to GitHub). There would also be performance considerations from making 30+ http requests as part of a pre-receive hook. In future, we are looking at creating a pre-receive hook solution that focuses on patterns that have a very low false positive rate. There are already some open source solutions that do this (links below) - in fact the OP linked to one from his Twitter thread. If/when GitHub offer is, it will definitely be opt-in, rather than opt-out! * https://github.com/thoughtworks/talisman/ https://github.com/thoughtworks/talisman/ * https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
  - 0xad 6y ago
    Cool! Thanks for explanation.
dorfsmay 6y ago
Do you also scan when a private repo is changed to public?
- greysteil 6y ago
  I think so, and we 100% should do, but I just did a test and the secret I committed was still working a full minute after I converted the repo. Could be that the scan was in a queue, could be that it didn't trigger. I'll dig into it and make sure this is working and is fast - it's a critical time to do a full scan of the repo's git history.
  - OJFord 6y ago
    So scanning is only done on public repos?
- mackenzie-gg 6y ago
  GitGuardian scans on every event, this includes a public event (when a Repo is made public) and will alert if secrets are found within.
the_duke 6y ago
Couldn't auto-revokation be used for a "DOS" attack of sorts by generating a lot of randomized tokens and pushing them to any repo? I realize that the search space is huge for many tokens types, but it seems viable.
- thdrdt 6y ago
  Selecting (reading) data is very fast most of the time. So if no token matches I don't think this will result in a DOS.
  - spydum 6y ago
    I think they meant it would autoinvalidate the tokens which might be valid. I think the math on an AWS secret and access key would be ridiculous to brute force.. but other types of keys might be an interesting attack vector.
    - gingerlime 6y ago
      what about the birthday paradox however? i.e. the attacker doesn’t need to brute force a specific key, but just any key... I guess for AWS the search space is still huge enough for it not to be a problem still (but didn’t do the math)
      - Hello71 6y ago
        seems like you could just log in directly at that rate
      - time0ut 6y ago
        I believe AWS secrets are 240 bits. That is a pretty massive space. I don't know how many active secrets are out there, but I think someone would need to get very lucky to collide before the attack was noticed and stopped. Other partner's secrets may be more susceptible. Edit: I did not consider the paired access key which is another 70 or so bits. I think you'd need to collide on both to make someone have a bad day.
jakub_g 6y ago
Why secret scanning is enabled only for public repos but not for private ones?
- greysteil 6y ago
  Private repos need a different approach, but committing secrets to them can still be a problem. If a secret is committed to a private repo then anyone with read access to that repo could use it. That might give those users more permissions than they're supposed to have. It's particularly a problem in large organisations, where thousands of developers may have access to a private repo, but should not necessarily have direct access to production infrastructure. That said, the risk tradeoff when a secret is found in a private repo is different to when one is found in a public repo. If it's a personal private repo that no-one else has access to, the risk may be limited. If it's a corporate repo with hundreds of contributors, someone almost certainly wants to be aware of it. Even then, each organisation will want to respond in different ways, perhaps depending on who has access to the repo, and what access the leaked secret granted. I'd be remiss not to say that GitHub has a beta offering for private repo secret scanning that we launched in May. It's a paid feature, targeted at large, security-conscious organisations, that scans your git history and each new commit for secrets and displays them in the GitHub UI.
  - jakub_g 6y ago
    Ah nice, just found it here: https://github.blog/changelog/2020-05-06-github-advanced-security-secret-scanning-for-private-repositories-now-available-in-limited-public-beta/ https://github.blog/changelog/2020-05-06-github-advanced-sec... Thanks!
- Lex-2008 6y ago
  Because it should be OK to commit secrets to private repos - that's why they're _private_, after all, right?
  - dorfsmay 6y ago
    No, that's not not why. If you have secrets, encrypt them. Private repos can be turned public, intentionally or by mistake. Repos can be exported to give software to third parties. Also, git users clone repos, which means that those secrets are copied every where. Can you make sure those stay private too? Do you make your developers encrypt their laptops or delete repos from them before they leave their house or office?
amelius 6y ago
I suppose you could still XOR your secret S with a random bitstring B, then commit both S^B and B. Am I missing something?
- deleted 6y ago
  [deleted]

0xad 6y ago

Awesome, thanks for the background information!

tobr 6y ago

Why not refuse to publish a detected secret at all until the repo owner takes an action to allow it?

watt 6y ago
Think about it in terms of incentives and nudges.
greysteil 6y ago
There are a few considerations on that one, but one very practical reason is the developer experience of dealing with false positives. False positives are one of the big problems in secret scanning. Some partners issue credentials with patterns that make them very hard to distinguish from innocuous strings. For example, a Datadog token looks identical to a commit SHA. We would never block developers from pushing commits to GitHub just because they had 40 character hexadecimal strings in them! GitHub's partnership approach works around the false positive problem by having the token issuer check whether a token is real and take action only if it is. However, this is a one-way communication from GitHub - the token issuer doesn't need to tell us whether the candidate secret we sent them was real or not, and in most cases we never know. As a result, we can't replicate the zero false positive experience in a pre-receive hook (i.e., before the commit is pushed to GitHub). There would also be performance considerations from making 30+ http requests as part of a pre-receive hook. In future, we are looking at creating a pre-receive hook solution that focuses on patterns that have a very low false positive rate. There are already some open source solutions that do this (links below) - in fact the OP linked to one from his Twitter thread. If/when GitHub offer is, it will definitely be opt-in, rather than opt-out! * https://github.com/thoughtworks/talisman/ https://github.com/thoughtworks/talisman/ * https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
- 0xad 6y ago
  Cool! Thanks for explanation.

dorfsmay 6y ago

Do you also scan when a private repo is changed to public?

greysteil 6y ago
I think so, and we 100% should do, but I just did a test and the secret I committed was still working a full minute after I converted the repo. Could be that the scan was in a queue, could be that it didn't trigger. I'll dig into it and make sure this is working and is fast - it's a critical time to do a full scan of the repo's git history.
- OJFord 6y ago
  So scanning is only done on public repos?
mackenzie-gg 6y ago
GitGuardian scans on every event, this includes a public event (when a Repo is made public) and will alert if secrets are found within.

the_duke 6y ago

Couldn't auto-revokation be used for a "DOS" attack of sorts by generating a lot of randomized tokens and pushing them to any repo? I realize that the search space is huge for many tokens types, but it seems viable.

thdrdt 6y ago
Selecting (reading) data is very fast most of the time. So if no token matches I don't think this will result in a DOS.
- spydum 6y ago
  I think they meant it would autoinvalidate the tokens which might be valid. I think the math on an AWS secret and access key would be ridiculous to brute force.. but other types of keys might be an interesting attack vector.
  - gingerlime 6y ago
    what about the birthday paradox however? i.e. the attacker doesn’t need to brute force a specific key, but just any key... I guess for AWS the search space is still huge enough for it not to be a problem still (but didn’t do the math)
    - Hello71 6y ago
      seems like you could just log in directly at that rate
    - time0ut 6y ago
      I believe AWS secrets are 240 bits. That is a pretty massive space. I don't know how many active secrets are out there, but I think someone would need to get very lucky to collide before the attack was noticed and stopped. Other partner's secrets may be more susceptible. Edit: I did not consider the paired access key which is another 70 or so bits. I think you'd need to collide on both to make someone have a bad day.

jakub_g 6y ago

Why secret scanning is enabled only for public repos but not for private ones?

greysteil 6y ago
Private repos need a different approach, but committing secrets to them can still be a problem. If a secret is committed to a private repo then anyone with read access to that repo could use it. That might give those users more permissions than they're supposed to have. It's particularly a problem in large organisations, where thousands of developers may have access to a private repo, but should not necessarily have direct access to production infrastructure. That said, the risk tradeoff when a secret is found in a private repo is different to when one is found in a public repo. If it's a personal private repo that no-one else has access to, the risk may be limited. If it's a corporate repo with hundreds of contributors, someone almost certainly wants to be aware of it. Even then, each organisation will want to respond in different ways, perhaps depending on who has access to the repo, and what access the leaked secret granted. I'd be remiss not to say that GitHub has a beta offering for private repo secret scanning that we launched in May. It's a paid feature, targeted at large, security-conscious organisations, that scans your git history and each new commit for secrets and displays them in the GitHub UI.
- jakub_g 6y ago
  Ah nice, just found it here: https://github.blog/changelog/2020-05-06-github-advanced-security-secret-scanning-for-private-repositories-now-available-in-limited-public-beta/ https://github.blog/changelog/2020-05-06-github-advanced-sec... Thanks!
Lex-2008 6y ago
Because it should be OK to commit secrets to private repos - that's why they're _private_, after all, right?
- dorfsmay 6y ago
  No, that's not not why. If you have secrets, encrypt them. Private repos can be turned public, intentionally or by mistake. Repos can be exported to give software to third parties. Also, git users clone repos, which means that those secrets are copied every where. Can you make sure those stay private too? Do you make your developers encrypt their laptops or delete repos from them before they leave their house or office?

amelius 6y ago

I suppose you could still XOR your secret S with a random bitstring B, then commit both S^B and B. Am I missing something?

deleted 6y ago
[deleted]