9 ms·

Many years ago I got a trial license key for something, Aspose components of some sorts I think, and without thinking of it, checked it in into public Github re

Many years ago I got a trial license key for something, Aspose components of some sorts I think, and without thinking of it, checked it in into public Github repo. Well, few days later Aspose's support sends me a nicely worded note saying that they noticed that it was there and invalidated it for me. Their description and instructions were very clear about why they did it and why I shouldn't have checked it in. I thought that was very proactive and excellent customer service.

Natsukvshii 4y ago
I actually had something similar happen to me last month. I accidentally published a discord API key to GitHub and within minutes I got a nice message from “Safety Jim” to my personal discord account letting me know they’ve found my key on a public repo and have gone ahead and revoked it. I felt like a bit of a dope but it was neat to have it happen to me. Lesson learned for sure.
- greysteil 4y ago
  GitHub PM here. Glad that was a good experience! We work with ~50 partners (details in the link below) to notify them when tokens for their service are exposed in public repos, so that they can notify you. https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns#supported-secrets-for-partner-patterns https://docs.github.com/en/code-security/secret-scanning/sec...
  - fiddlerwoaroof 4y ago
    I wish I could set this up to block pushes proactively instead of reacting to pushed secrets.
    - kadoban 4y ago
      You could set up something like https://github.com/godaddy/tartufo https://github.com/godaddy/tartufo in a pre-commit hook. Not sure if github has a way to hook into the push hooks on server side, they might though.
      - fiddlerwoaroof 4y ago
        Yeah, the issue with pre-commit hooks is you have to remember to set them up client-side. I tend to push to GitHub through a gitolite mirror, though, so I could probably put this in the hooks in my gitolite middlebox.
        mypalmike 4y ago
        What do you have to set up client side? They can be committed with the project. Or do I misunderstand?
        kadoban 4y ago
        Pre-commit hooks can't be automatically set up on the client side. If they could, this would mean that any repo you clone could run arbitrary code on your machine. It can be as simple as a script you have to run once, but it can't be automatic. Which also means you can't really trust contributors to do it, even if they're well-meaning some will forget.
    - iancarroll 4y ago
      This exists on GitHub but is not free. https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning#using-secret-scanning-as-a-push-protection-from-the-command-line https://docs.github.com/en/enterprise-cloud@latest/code-secu...
    - neuronexmachina 4y ago
      Yelp has a "detect-secrets" project that can detect potential secrets and can be used as a pre-commit hook: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
    - JoeMattiello 4y ago
      https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
    - cwinq 4y ago
      you can try ggshield - https://github.com/GitGuardian/ggshield https://github.com/GitGuardian/ggshield -
  - cebert 4y ago
    This is awesome!
  - DiggyJohnson 4y ago
    Top proactive security feature of the year, for me. Nice stuff.
  - tough 4y ago
    Is this really expensive? We're a small startup providing API keys, to our customers.
    - greysteil 4y ago
      It's totally free - there are details of how to join the program at https://docs.github.com/en/developers/overview/secret-scanning-partner-program#joining-the-secret-scanning-program-on-github https://docs.github.com/en/developers/overview/secret-scanni...
      - josephg 4y ago
        Hm - this would work better if keys were easy to scan with regular expressions. Next time I implement api keys I wonder if it’s worth going out of my way to make them easy to identify. Eg, by prefixing every key with a few well known characters. Like FMLA_xxxxx for a fastmail app key.
        crazysim 4y ago
        That's exactly what GitHub did with their own keys and their new keys fit this format. https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...
        tough 4y ago
        I just implemented our API with a PREFIX_KEY so our self-hosted customers can change it they want to. We will be applying thanks for sharing greystell
        nijave 4y ago
        Some services also use prefixes to provide additional context like account type and token validity length. I think Slack does this (service accounts have different prefixes than user accounts and I think temporary tokens have another prefix)
        4y ago
  - artursapek 4y ago
    That's attention to detail right there. Very nice.
  - deleted 4y ago
    [deleted]
  - insane_dreamer 4y ago
    Awesome feature. Saved the day for us some months back when an AWS token was accidentally committed and pushed. (AWS itself also immediately notified us.)
    - scarface74 4y ago
      Rant time: this isn’t directed at you. I am just replying to your comment because you said something that triggered me. Also the “you” below is the generic you - not you personally. Disclaimer: I work at AWS in Professional Services, all rants are my own. Now with that out of the way, I hate the fact that there are way too many code samples floating around on the internet that have you explicitly put your access key and secret key in the initialization code for the AWS SDK. s3 = boto3.resource(‘s3’,aws_accesskey_id=ccxx,aws_secret_access_key_id=cccc) Even if you put the access keys in a separate config file in your repo, this is wrong, unnecessary, and can easily lead to checking credentials in. When all they have to do is s3=boto3.resource(‘s3’) All of the SDKs will automatically find your credentials locally in your .config file that is in your home directory when you run “aws configure”. But really, you shouldn’t do that, you should use temporary access keys. When you do get ready to run on AWS, the SDK will automatically get the credentials from the attached role. Even when I’m integrating AWS with Azure DevOps, Microsoft provides a separate secure store that you can attach to your pipeline for your AWS credentials.
      - Mutjake 4y ago
        Hindsight is 20/20, but definitely one of those places where flat out giving the credentials should not even be an option (or it should be made artificially tedious and/or explicitly clear that it’s a bad idea by e.g. naming the param _this_is_a_bad_idea_use_credentials_file_instead_secret_key or so). Of course there are always edge cases in the vein of running notebooks in containers (probably not an optimal example, but some edge case like that) where you might need the escape hatch of embedding the credentials straight to the code. But yeah, if the wrong thing is easier or more straightforward than the right way, people tend to follow it when they have a deadline to meet. To end on a positive note, at least cli v2 makes bootstrapping the credentials to a workstation a tad easier!
  - comboy 4y ago
    TIL: make private key for your service easy to match with regexps
    - hunter2_ 4y ago
      Reminds me of how Airbnb redacts Hawaiian street addresses because they look too much like phone numbers, literally replacing them with a "phone number hidden" string in the host|guest chat. Moral of the story: make your keys regexable without likelihood of false positives!
      - Breza 4y ago
        I spend a lot of time working with physician data. In the USA, physicians have a registration system called NPI. Apparently, NPI numbers are in the same format as some passport numbers. I know this because I started getting angry warnings about PII sharing until I got our tech team to turn them off.
    - echelon 4y ago
      The whole industry should adopt a convention to prefix production keys with a well known prefix, such as "prod_secret_". We should have our systems and precommit hooks then alert us when those enter places they shouldn't and help us automate rotation.
      - jve 4y ago
        Bad idea. Better do it in DEV like you would do in PROD, not to shoot yourself in the foot. If you do it right in DEV, no problem in PROD. And what if your DEV is not actually well isolated from PROD/other infra? And what if some real data sneaked into DEV? Etc.
        mewpmewp2 4y ago
        I think prod_ might not be the important part there, so something like __secret__ should be enough.
      - 4y ago
  - psaux 4y ago
    I had a couple questions, as this feature is awesome! How long does it take to get the response vs external bots pulling the data? What mechanisms does GitHub have in place to stop bots who monitor repo changes? I ask, as I have been there and it is super scary how fast someone/bot pulls repo data changes, as in minutes, and the repo we had back then was not popular.
    - ls15 4y ago
      As long as search results can be sorted by date, anyone can see updates pretty much instantly if they monitor the search results. The repos don't have to be popular for that. Bots can just check such a feed every few seconds for example. https://github.com/search?o=desc&q=secret&s=updated&type=Repositories https://github.com/search?o=desc&q=secret&s=updated&type=Rep...
  - pabs3 4y ago
    Would blocking commits containing such tokens/keys be a better option?
    - JoeMattiello 4y ago
      https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
    - hnlmorg 4y ago
      You cannot block what someone commits (they can block it themselves with tools like gitleaks invoked on a pre-commit hook) so the only thing you can do as a 3rd party is to scan and react when you do notice a secret published.
      - pabs3 4y ago
        GitHub certainly could block push requests, at least git itself can via hooks, there are a number of hooks invoked by git-receive-pack that can influence what it does.
        hnlmorg 4y ago
        > GitHub certainly could block push requests But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature. > at least git itself can via hooks I already said that: > they can block it themselves with tools like gitleaks invoked on a pre-commit hook The problem with git hooks is that they're not cloned with the repo. So you're reliant on the user installing those git hooks locally (sure, some repos will have helper scripts to install the hooks for you. But you're still reliant on the user running that script).
        pabs3 4y ago
        > code you cannot push to origin. Definitely not a desirable feature. If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. > The problem with git hooks I was talking about GitHub's own git hooks that run on their servers, not about any local ones. > is that they're not cloned with the repo. It would be a terrible security issue if they were automatically enabled after cloning.
  - kenperkins 4y ago
    What are the thoughts around capabilities like this for private/enterprise customers? Is the code available in an action that could be connected to private runners perhaps?
- vultour 4y ago
  Great that they finally do that. I accidentally checked one into a public GitHub repo a long time ago and about 2 years later someone found it. The infinite spam wasn't even the worst part about this, bajillion emojis in every message just caused the Discord client to crash instantly upon opening, so I couldn't even figure out what's happening at first.
- shepherdjerred 4y ago
  I’ve had this happen to me too! No less than a second after pushing to GitHub did I receive a message about publicizing my auth key. It was amazing, and I’m sure this saves a lot of stolen keys from people just getting into programming
deleted 4y ago
[deleted]
paxys 4y ago
Github supports this out of the box – https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning https://docs.github.com/en/code-security/secret-scanning/abo..., and recognizes tokens from a lot of services.
- derefr 4y ago
  In fact, you can apply as a Github "secret scanning partner" to have your own secret's format (regexp) be a part of this secret scanning, with a webhook to your servers whenever they find one, so that you can do the credential-invalidation on your own backend + send the kindly-worded email from your own domain. Mind you, your secrets need to have a distinctive format in order for this to work. Probably a distinctive prefix is enough. An Unethical Life Pro-Tip (that the word is already out on anyway, so I don't feel too bad): • The content of Github public repos is all continuously loaded (by Github themselves) as a public dataset into BigQuery — https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset https://console.cloud.google.com/marketplace/details/github/.... • For about $500, you can use BigQuery to extract all matches of a particular regexp, from every file, in every commit, in every public Github repo. Whether or not Github themselves use this to power their secret scanning, arbitrary third parties (benevolent or not) certainly can use it for such. And likely already do.
  - lupire 4y ago
    Does GitHub not postpone publishing new verisons until after the secret scanning is done? Also I'd hope that Google is scanning BigQuery queries for that abuse signal.
  - dantiberian 4y ago
    The GitHub public events API is delayed by 5 minutes, presumably to give secret scanning partners time to react before commits are made public. https://github.blog/changelog/2018-08-01-new-delay-public-events-api/ https://github.blog/changelog/2018-08-01-new-delay-public-ev... Disclosure: I'm an ex-GitHub employee but was not involved in the secret scanning API.
JoeMattiello 4y ago
I worked for a big startup last year and was on a contract deadline for integrating a vendor framework into a React Native app. It was taking too long to get a new temp demo license key and GitHub search with clever filters helped me track down a demo key that was recently uploaded to a test repo. This is also why I use git-secrets in my repos. https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
fsckboy 4y ago
or it was being abused and they noticed the abuse

Natsukvshii 4y ago

I actually had something similar happen to me last month. I accidentally published a discord API key to GitHub and within minutes I got a nice message from “Safety Jim” to my personal discord account letting me know they’ve found my key on a public repo and have gone ahead and revoked it. I felt like a bit of a dope but it was neat to have it happen to me. Lesson learned for sure.

greysteil 4y ago
GitHub PM here. Glad that was a good experience! We work with ~50 partners (details in the link below) to notify them when tokens for their service are exposed in public repos, so that they can notify you. https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns#supported-secrets-for-partner-patterns https://docs.github.com/en/code-security/secret-scanning/sec...
- fiddlerwoaroof 4y ago
  I wish I could set this up to block pushes proactively instead of reacting to pushed secrets.
  - kadoban 4y ago
    You could set up something like https://github.com/godaddy/tartufo https://github.com/godaddy/tartufo in a pre-commit hook. Not sure if github has a way to hook into the push hooks on server side, they might though.
    - fiddlerwoaroof 4y ago
      Yeah, the issue with pre-commit hooks is you have to remember to set them up client-side. I tend to push to GitHub through a gitolite mirror, though, so I could probably put this in the hooks in my gitolite middlebox.
      - mypalmike 4y ago
        What do you have to set up client side? They can be committed with the project. Or do I misunderstand?
        kadoban 4y ago
        Pre-commit hooks can't be automatically set up on the client side. If they could, this would mean that any repo you clone could run arbitrary code on your machine. It can be as simple as a script you have to run once, but it can't be automatic. Which also means you can't really trust contributors to do it, even if they're well-meaning some will forget.
  - iancarroll 4y ago
    This exists on GitHub but is not free. https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/protecting-pushes-with-secret-scanning#using-secret-scanning-as-a-push-protection-from-the-command-line https://docs.github.com/en/enterprise-cloud@latest/code-secu...
  - neuronexmachina 4y ago
    Yelp has a "detect-secrets" project that can detect potential secrets and can be used as a pre-commit hook: https://github.com/Yelp/detect-secrets https://github.com/Yelp/detect-secrets
  - JoeMattiello 4y ago
    https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
  - cwinq 4y ago
    you can try ggshield - https://github.com/GitGuardian/ggshield https://github.com/GitGuardian/ggshield -
- cebert 4y ago
  This is awesome!
- DiggyJohnson 4y ago
  Top proactive security feature of the year, for me. Nice stuff.
- tough 4y ago
  Is this really expensive? We're a small startup providing API keys, to our customers.
  - greysteil 4y ago
    It's totally free - there are details of how to join the program at https://docs.github.com/en/developers/overview/secret-scanning-partner-program#joining-the-secret-scanning-program-on-github https://docs.github.com/en/developers/overview/secret-scanni...
    - josephg 4y ago
      Hm - this would work better if keys were easy to scan with regular expressions. Next time I implement api keys I wonder if it’s worth going out of my way to make them easy to identify. Eg, by prefixing every key with a few well known characters. Like FMLA_xxxxx for a fastmail app key.
      - crazysim 4y ago
        That's exactly what GitHub did with their own keys and their new keys fit this format. https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...
        tough 4y ago
        I just implemented our API with a PREFIX_KEY so our self-hosted customers can change it they want to. We will be applying thanks for sharing greystell
      - nijave 4y ago
        Some services also use prefixes to provide additional context like account type and token validity length. I think Slack does this (service accounts have different prefixes than user accounts and I think temporary tokens have another prefix)
      - 4y ago
- artursapek 4y ago
  That's attention to detail right there. Very nice.
- deleted 4y ago
  [deleted]
- insane_dreamer 4y ago
  Awesome feature. Saved the day for us some months back when an AWS token was accidentally committed and pushed. (AWS itself also immediately notified us.)
  - scarface74 4y ago
    Rant time: this isn’t directed at you. I am just replying to your comment because you said something that triggered me. Also the “you” below is the generic you - not you personally. Disclaimer: I work at AWS in Professional Services, all rants are my own. Now with that out of the way, I hate the fact that there are way too many code samples floating around on the internet that have you explicitly put your access key and secret key in the initialization code for the AWS SDK. s3 = boto3.resource(‘s3’,aws_accesskey_id=ccxx,aws_secret_access_key_id=cccc) Even if you put the access keys in a separate config file in your repo, this is wrong, unnecessary, and can easily lead to checking credentials in. When all they have to do is s3=boto3.resource(‘s3’) All of the SDKs will automatically find your credentials locally in your .config file that is in your home directory when you run “aws configure”. But really, you shouldn’t do that, you should use temporary access keys. When you do get ready to run on AWS, the SDK will automatically get the credentials from the attached role. Even when I’m integrating AWS with Azure DevOps, Microsoft provides a separate secure store that you can attach to your pipeline for your AWS credentials.
    - Mutjake 4y ago
      Hindsight is 20/20, but definitely one of those places where flat out giving the credentials should not even be an option (or it should be made artificially tedious and/or explicitly clear that it’s a bad idea by e.g. naming the param _this_is_a_bad_idea_use_credentials_file_instead_secret_key or so). Of course there are always edge cases in the vein of running notebooks in containers (probably not an optimal example, but some edge case like that) where you might need the escape hatch of embedding the credentials straight to the code. But yeah, if the wrong thing is easier or more straightforward than the right way, people tend to follow it when they have a deadline to meet. To end on a positive note, at least cli v2 makes bootstrapping the credentials to a workstation a tad easier!
- comboy 4y ago
  TIL: make private key for your service easy to match with regexps
  - hunter2_ 4y ago
    Reminds me of how Airbnb redacts Hawaiian street addresses because they look too much like phone numbers, literally replacing them with a "phone number hidden" string in the host|guest chat. Moral of the story: make your keys regexable without likelihood of false positives!
    - Breza 4y ago
      I spend a lot of time working with physician data. In the USA, physicians have a registration system called NPI. Apparently, NPI numbers are in the same format as some passport numbers. I know this because I started getting angry warnings about PII sharing until I got our tech team to turn them off.
  - echelon 4y ago
    The whole industry should adopt a convention to prefix production keys with a well known prefix, such as "prod_secret_". We should have our systems and precommit hooks then alert us when those enter places they shouldn't and help us automate rotation.
    - jve 4y ago
      Bad idea. Better do it in DEV like you would do in PROD, not to shoot yourself in the foot. If you do it right in DEV, no problem in PROD. And what if your DEV is not actually well isolated from PROD/other infra? And what if some real data sneaked into DEV? Etc.
      - mewpmewp2 4y ago
        I think prod_ might not be the important part there, so something like __secret__ should be enough.
    - 4y ago
- psaux 4y ago
  I had a couple questions, as this feature is awesome! How long does it take to get the response vs external bots pulling the data? What mechanisms does GitHub have in place to stop bots who monitor repo changes? I ask, as I have been there and it is super scary how fast someone/bot pulls repo data changes, as in minutes, and the repo we had back then was not popular.
  - ls15 4y ago
    As long as search results can be sorted by date, anyone can see updates pretty much instantly if they monitor the search results. The repos don't have to be popular for that. Bots can just check such a feed every few seconds for example. https://github.com/search?o=desc&q=secret&s=updated&type=Repositories https://github.com/search?o=desc&q=secret&s=updated&type=Rep...
- pabs3 4y ago
  Would blocking commits containing such tokens/keys be a better option?
  - JoeMattiello 4y ago
    https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets
  - hnlmorg 4y ago
    You cannot block what someone commits (they can block it themselves with tools like gitleaks invoked on a pre-commit hook) so the only thing you can do as a 3rd party is to scan and react when you do notice a secret published.
    - pabs3 4y ago
      GitHub certainly could block push requests, at least git itself can via hooks, there are a number of hooks invoked by git-receive-pack that can influence what it does.
      - hnlmorg 4y ago
        > GitHub certainly could block push requests But the commit still exists locally (since git is decentralized) so you now end up with a weird state that you have code you cannot push to origin. Definitely not a desirable feature. > at least git itself can via hooks I already said that: > they can block it themselves with tools like gitleaks invoked on a pre-commit hook The problem with git hooks is that they're not cloned with the repo. So you're reliant on the user installing those git hooks locally (sure, some repos will have helper scripts to install the hooks for you. But you're still reliant on the user running that script).
        pabs3 4y ago
        > code you cannot push to origin. Definitely not a desirable feature. If there is data that should never be pushed to origin, then it is a highly desirable feature that the server block pushes that include that private data. > The problem with git hooks I was talking about GitHub's own git hooks that run on their servers, not about any local ones. > is that they're not cloned with the repo. It would be a terrible security issue if they were automatically enabled after cloning.
- kenperkins 4y ago
  What are the thoughts around capabilities like this for private/enterprise customers? Is the code available in an action that could be connected to private runners perhaps?
vultour 4y ago
Great that they finally do that. I accidentally checked one into a public GitHub repo a long time ago and about 2 years later someone found it. The infinite spam wasn't even the worst part about this, bajillion emojis in every message just caused the Discord client to crash instantly upon opening, so I couldn't even figure out what's happening at first.
shepherdjerred 4y ago
I’ve had this happen to me too! No less than a second after pushing to GitHub did I receive a message about publicizing my auth key. It was amazing, and I’m sure this saves a lot of stolen keys from people just getting into programming

deleted 4y ago

[deleted]

paxys 4y ago

Github supports this out of the box – https://docs.github.com/en/code-security/secret-scanning/about-secret-scanning https://docs.github.com/en/code-security/secret-scanning/abo..., and recognizes tokens from a lot of services.

derefr 4y ago
In fact, you can apply as a Github "secret scanning partner" to have your own secret's format (regexp) be a part of this secret scanning, with a webhook to your servers whenever they find one, so that you can do the credential-invalidation on your own backend + send the kindly-worded email from your own domain. Mind you, your secrets need to have a distinctive format in order for this to work. Probably a distinctive prefix is enough. An Unethical Life Pro-Tip (that the word is already out on anyway, so I don't feel too bad): • The content of Github public repos is all continuously loaded (by Github themselves) as a public dataset into BigQuery — https://console.cloud.google.com/marketplace/details/github/github-repos?filter=solution-type:dataset https://console.cloud.google.com/marketplace/details/github/.... • For about $500, you can use BigQuery to extract all matches of a particular regexp, from every file, in every commit, in every public Github repo. Whether or not Github themselves use this to power their secret scanning, arbitrary third parties (benevolent or not) certainly can use it for such. And likely already do.
- lupire 4y ago
  Does GitHub not postpone publishing new verisons until after the secret scanning is done? Also I'd hope that Google is scanning BigQuery queries for that abuse signal.
- dantiberian 4y ago
  The GitHub public events API is delayed by 5 minutes, presumably to give secret scanning partners time to react before commits are made public. https://github.blog/changelog/2018-08-01-new-delay-public-events-api/ https://github.blog/changelog/2018-08-01-new-delay-public-ev... Disclosure: I'm an ex-GitHub employee but was not involved in the secret scanning API.

JoeMattiello 4y ago

I worked for a big startup last year and was on a contract deadline for integrating a vendor framework into a React Native app. It was taking too long to get a new temp demo license key and GitHub search with clever filters helped me track down a demo key that was recently uploaded to a test repo. This is also why I use git-secrets in my repos. https://github.com/awslabs/git-secrets https://github.com/awslabs/git-secrets

fsckboy 4y ago

or it was being abused and they noticed the abuse