16 ms·

GitHub PM here. Glad that was a good experience! We work with ~50 partners (details in the link below) to notify them when tokens for their service are exposed

GitHub PM here. Glad that was a good experience! We work with ~50 partners (details in the link below) to notify them when tokens for their service are exposed in public repos, so that they can notify you. https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns#supported-secrets-for-partner-patterns https://docs.github.com/en/code-security/secret-scanning/sec...

fiddlerwoaroof 4y ago
I wish I could set this up to block pushes proactively instead of reacting to pushed secrets.
- kadoban 4y ago
  You could set up something like https://github.com/godaddy/tartufo https://github.com/godaddy/tartufo in a pre-commit hook. Not sure if github has a way to hook into the push hooks on server side, they might though.
  - fiddlerwoaroof 4y ago
    Yeah, the issue with pre-commit hooks is you have to remember to set them up client-side. I tend to push to GitHub through a gitolite mirror, though, so I could probably put this in the hooks in my gitolite middlebox.
    - mypalmike 4y ago
      What do you have to set up client side? They can be committed with the project. Or do I misunderstand?
      - kadoban 4y ago
        Pre-commit hooks can't be automatically set up on the client side. If they could, this would mean that any repo you clone could run arbitrary code on your machine. It can be as simple as a script you have to run once, but it can't be automatic. Which also means you can't really trust contributors to do it, even if they're well-meaning some will forget.
- iancarroll 4y ago
  This exists on GitHub but is not free. https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning#using-secret-scanning-as-a-push-protection-from-the-command-line https://docs.github.com/en/enterprise-cloud@latest/code-secu...
- neuronexmachina 4y ago
  Yelp has a "detect-secrets" project that can detect potential secrets and can be used as a pre-commit hook: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
- JoeMattiello 4y ago
  https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
- cwinq 4y ago
  you can try ggshield - https://github.com/GitGuardian/ggshield https://github.com/GitGuardian/ggshield -
cebert 4y ago
This is awesome!
DiggyJohnson 4y ago
Top proactive security feature of the year, for me. Nice stuff.
tough 4y ago
Is this really expensive? We're a small startup providing API keys, to our customers.
- greysteil 4y ago
  It's totally free - there are details of how to join the program at https://docs.github.com/en/developers/overview/secret-scanning-partner-program#joining-the-secret-scanning-program-on-github https://docs.github.com/en/developers/overview/secret-scanni...
  - josephg 4y ago
    Hm - this would work better if keys were easy to scan with regular expressions. Next time I implement api keys I wonder if it’s worth going out of my way to make them easy to identify. Eg, by prefixing every key with a few well known characters. Like FMLA_xxxxx for a fastmail app key.
    - crazysim 4y ago
      That's exactly what GitHub did with their own keys and their new keys fit this format. https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...
      - tough 4y ago
        I just implemented our API with a PREFIX_KEY so our self-hosted customers can change it they want to. We will be applying thanks for sharing greystell
    - nijave 4y ago
      Some services also use prefixes to provide additional context like account type and token validity length. I think Slack does this (service accounts have different prefixes than user accounts and I think temporary tokens have another prefix)
    - 4y ago
artursapek 4y ago
That's attention to detail right there. Very nice.
deleted 4y ago
[deleted]
insane_dreamer 4y ago
Awesome feature. Saved the day for us some months back when an AWS token was accidentally committed and pushed. (AWS itself also immediately notified us.)
- scarface74 4y ago
  Rant time: this isn’t directed at you. I am just replying to your comment because you said something that triggered me. Also the “you” below is the generic you - not you personally. Disclaimer: I work at AWS in Professional Services, all rants are my own. Now with that out of the way, I hate the fact that there are way too many code samples floating around on the internet that have you explicitly put your access key and secret key in the initialization code for the AWS SDK. s3 = boto3.resource(‘s3’,aws_accesskey_id=ccxx,aws_secret_access_key_id=cccc) Even if you put the access keys in a separate config file in your repo, this is wrong, unnecessary, and can easily lead to checking credentials in. When all they have to do is s3=boto3.resource(‘s3’) All of the SDKs will automatically find your credentials locally in your .config file that is in your home directory when you run “aws configure”. But really, you shouldn’t do that, you should use temporary access keys. When you do get ready to run on AWS, the SDK will automatically get the credentials from the attached role. Even when I’m integrating AWS with Azure DevOps, Microsoft provides a separate secure store that you can attach to your pipeline for your AWS credentials.
  - Mutjake 4y ago
    Hindsight is 20/20, but definitely one of those places where flat out giving the credentials should not even be an option (or it should be made artificially tedious and/or explicitly clear that it’s a bad idea by e.g. naming the param _this_is_a_bad_idea_use_credentials_file_instead_secret_key or so). Of course there are always edge cases in the vein of running notebooks in containers (probably not an optimal example, but some edge case like that) where you might need the escape hatch of embedding the credentials straight to the code. But yeah, if the wrong thing is easier or more straightforward than the right way, people tend to follow it when they have a deadline to meet. To end on a positive note, at least cli v2 makes bootstrapping the credentials to a workstation a tad easier!
comboy 4y ago
TIL: make private key for your service easy to match with regexps
- hunter2_ 4y ago
  Reminds me of how Airbnb redacts Hawaiian street addresses because they look too much like phone numbers, literally replacing them with a "phone number hidden" string in the host|guest chat. Moral of the story: make your keys regexable without likelihood of false positives!
  - Breza 4y ago
    I spend a lot of time working with physician data. In the USA, physicians have a registration system called NPI. Apparently, NPI numbers are in the same format as some passport numbers. I know this because I started getting angry warnings about PII sharing until I got our tech team to turn them off.
- echelon 4y ago
  The whole industry should adopt a convention to prefix production keys with a well known prefix, such as "prod_secret_". We should have our systems and precommit hooks then alert us when those enter places they shouldn't and help us automate rotation.
  - jve 4y ago
    Bad idea. Better do it in DEV like you would do in PROD, not to shoot yourself in the foot. If you do it right in DEV, no problem in PROD. And what if your DEV is not actually well isolated from PROD/other infra? And what if some real data sneaked into DEV? Etc.
    - mewpmewp2 4y ago
      I think prod_ might not be the important part there, so something like __secret__ should be enough.
  - 4y ago
psaux 4y ago
I had a couple questions, as this feature is awesome! How long does it take to get the response vs external bots pulling the data? What mechanisms does GitHub have in place to stop bots who monitor repo changes? I ask, as I have been there and it is super scary how fast someone/bot pulls repo data changes, as in minutes, and the repo we had back then was not popular.
- ls15 4y ago
  As long as search results can be sorted by date, anyone can see updates pretty much instantly if they monitor the search results. The repos don't have to be popular for that. Bots can just check such a feed every few seconds for example. https://github.com/search?o=desc&q=secret&s=updated&type=Repositories https://github.com/search?o=desc&q=secret&s=updated&type=Rep...
pabs3 4y ago
Would blocking commits containing such tokens/keys be a better option?
- JoeMattiello 4y ago
  https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
- hnlmorg 4y ago
  You cannot block what someone commits (they can block it themselves with tools like gitleaks invoked on a pre-commit hook) so the only thing you can do as a 3rd party is to scan and react when you do notice a secret published.
  - pabs3 4y ago
    GitHub certainly could block push requests, at least git itself can via hooks, there are a number of hooks invoked by git-receive-pack that can influence what it does.
    - hnlmorg 4y ago
      > GitHub certainly could block push requests But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature. > at least git itself can via hooks I already said that: > they can block it themselves with tools like gitleaks invoked on a pre-commit hook The problem with git hooks is that they're not cloned with the repo. So you're reliant on the user installing those git hooks locally (sure, some repos will have helper scripts to install the hooks for you. But you're still reliant on the user running that script).
      - pabs3 4y ago
        > code you cannot push to origin. Definitely not a desirable feature. If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. > The problem with git hooks I was talking about GitHub's own git hooks that run on their servers, not about any local ones. > is that they're not cloned with the repo. It would be a terrible security issue if they were automatically enabled after cloning.
kenperkins 4y ago
What are the thoughts around capabilities like this for private/enterprise customers? Is the code available in an action that could be connected to private runners perhaps?

fiddlerwoaroof 4y ago

I wish I could set this up to block pushes proactively instead of reacting to pushed secrets.

kadoban 4y ago
You could set up something like https://github.com/godaddy/tartufo https://github.com/godaddy/tartufo in a pre-commit hook. Not sure if github has a way to hook into the push hooks on server side, they might though.
- fiddlerwoaroof 4y ago
  Yeah, the issue with pre-commit hooks is you have to remember to set them up client-side. I tend to push to GitHub through a gitolite mirror, though, so I could probably put this in the hooks in my gitolite middlebox.
  - mypalmike 4y ago
    What do you have to set up client side? They can be committed with the project. Or do I misunderstand?
    - kadoban 4y ago
      Pre-commit hooks can't be automatically set up on the client side. If they could, this would mean that any repo you clone could run arbitrary code on your machine. It can be as simple as a script you have to run once, but it can't be automatic. Which also means you can't really trust contributors to do it, even if they're well-meaning some will forget.
iancarroll 4y ago
This exists on GitHub but is not free. https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning#using-secret-scanning-as-a-push-protection-from-the-command-line https://docs.github.com/en/enterprise-cloud@latest/code-secu...
neuronexmachina 4y ago
Yelp has a "detect-secrets" project that can detect potential secrets and can be used as a pre-commit hook: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
JoeMattiello 4y ago
https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
cwinq 4y ago
you can try ggshield - https://github.com/GitGuardian/ggshield https://github.com/GitGuardian/ggshield -

cebert 4y ago

This is awesome!

DiggyJohnson 4y ago

Top proactive security feature of the year, for me. Nice stuff.

tough 4y ago

Is this really expensive? We're a small startup providing API keys, to our customers.

greysteil 4y ago
It's totally free - there are details of how to join the program at https://docs.github.com/en/developers/overview/secret-scanning-partner-program#joining-the-secret-scanning-program-on-github https://docs.github.com/en/developers/overview/secret-scanni...
- josephg 4y ago
  Hm - this would work better if keys were easy to scan with regular expressions. Next time I implement api keys I wonder if it’s worth going out of my way to make them easy to identify. Eg, by prefixing every key with a few well known characters. Like FMLA_xxxxx for a fastmail app key.
  - crazysim 4y ago
    That's exactly what GitHub did with their own keys and their new keys fit this format. https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...
    - tough 4y ago
      I just implemented our API with a PREFIX_KEY so our self-hosted customers can change it they want to. We will be applying thanks for sharing greystell
  - nijave 4y ago
    Some services also use prefixes to provide additional context like account type and token validity length. I think Slack does this (service accounts have different prefixes than user accounts and I think temporary tokens have another prefix)
  - 4y ago

artursapek 4y ago

That's attention to detail right there. Very nice.

deleted 4y ago

[deleted]

insane_dreamer 4y ago

Awesome feature. Saved the day for us some months back when an AWS token was accidentally committed and pushed. (AWS itself also immediately notified us.)

scarface74 4y ago
Rant time: this isn’t directed at you. I am just replying to your comment because you said something that triggered me. Also the “you” below is the generic you - not you personally. Disclaimer: I work at AWS in Professional Services, all rants are my own. Now with that out of the way, I hate the fact that there are way too many code samples floating around on the internet that have you explicitly put your access key and secret key in the initialization code for the AWS SDK. s3 = boto3.resource(‘s3’,aws_accesskey_id=ccxx,aws_secret_access_key_id=cccc) Even if you put the access keys in a separate config file in your repo, this is wrong, unnecessary, and can easily lead to checking credentials in. When all they have to do is s3=boto3.resource(‘s3’) All of the SDKs will automatically find your credentials locally in your .config file that is in your home directory when you run “aws configure”. But really, you shouldn’t do that, you should use temporary access keys. When you do get ready to run on AWS, the SDK will automatically get the credentials from the attached role. Even when I’m integrating AWS with Azure DevOps, Microsoft provides a separate secure store that you can attach to your pipeline for your AWS credentials.
- Mutjake 4y ago
  Hindsight is 20/20, but definitely one of those places where flat out giving the credentials should not even be an option (or it should be made artificially tedious and/or explicitly clear that it’s a bad idea by e.g. naming the param _this_is_a_bad_idea_use_credentials_file_instead_secret_key or so). Of course there are always edge cases in the vein of running notebooks in containers (probably not an optimal example, but some edge case like that) where you might need the escape hatch of embedding the credentials straight to the code. But yeah, if the wrong thing is easier or more straightforward than the right way, people tend to follow it when they have a deadline to meet. To end on a positive note, at least cli v2 makes bootstrapping the credentials to a workstation a tad easier!

comboy 4y ago

TIL: make private key for your service easy to match with regexps

hunter2_ 4y ago
Reminds me of how Airbnb redacts Hawaiian street addresses because they look too much like phone numbers, literally replacing them with a "phone number hidden" string in the host|guest chat. Moral of the story: make your keys regexable without likelihood of false positives!
- Breza 4y ago
  I spend a lot of time working with physician data. In the USA, physicians have a registration system called NPI. Apparently, NPI numbers are in the same format as some passport numbers. I know this because I started getting angry warnings about PII sharing until I got our tech team to turn them off.
echelon 4y ago
The whole industry should adopt a convention to prefix production keys with a well known prefix, such as "prod_secret_". We should have our systems and precommit hooks then alert us when those enter places they shouldn't and help us automate rotation.
- jve 4y ago
  Bad idea. Better do it in DEV like you would do in PROD, not to shoot yourself in the foot. If you do it right in DEV, no problem in PROD. And what if your DEV is not actually well isolated from PROD/other infra? And what if some real data sneaked into DEV? Etc.
  - mewpmewp2 4y ago
    I think prod_ might not be the important part there, so something like __secret__ should be enough.
- 4y ago

psaux 4y ago

I had a couple questions, as this feature is awesome! How long does it take to get the response vs external bots pulling the data? What mechanisms does GitHub have in place to stop bots who monitor repo changes? I ask, as I have been there and it is super scary how fast someone/bot pulls repo data changes, as in minutes, and the repo we had back then was not popular.

ls15 4y ago
As long as search results can be sorted by date, anyone can see updates pretty much instantly if they monitor the search results. The repos don't have to be popular for that. Bots can just check such a feed every few seconds for example. https://github.com/search?o=desc&q=secret&s=updated&type=Repositories https://github.com/search?o=desc&q=secret&s=updated&type=Rep...

pabs3 4y ago

Would blocking commits containing such tokens/keys be a better option?

JoeMattiello 4y ago
https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
hnlmorg 4y ago
You cannot block what someone commits (they can block it themselves with tools like gitleaks invoked on a pre-commit hook) so the only thing you can do as a 3rd party is to scan and react when you do notice a secret published.
- pabs3 4y ago
  GitHub certainly could block push requests, at least git itself can via hooks, there are a number of hooks invoked by git-receive-pack that can influence what it does.
  - hnlmorg 4y ago
    > GitHub certainly could block push requests But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature. > at least git itself can via hooks I already said that: > they can block it themselves with tools like gitleaks invoked on a pre-commit hook The problem with git hooks is that they're not cloned with the repo. So you're reliant on the user installing those git hooks locally (sure, some repos will have helper scripts to install the hooks for you. But you're still reliant on the user running that script).
    - pabs3 4y ago
      > code you cannot push to origin. Definitely not a desirable feature. If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. > The problem with git hooks I was talking about GitHub's own git hooks that run on their servers, not about any local ones. > is that they're not cloned with the repo. It would be a terrible security issue if they were automatically enabled after cloning.

kenperkins 4y ago

What are the thoughts around capabilities like this for private/enterprise customers? Is the code available in an action that could be connected to private runners perhaps?