4 ms·

Searching for creds can be tricky if they can't be readily distinguished from other text. Can anyone think of a problem with generating customer API keys that

Searching for creds can be tricky if they can't be readily distinguished from other text. Can anyone think of a problem with generating customer API keys that have a known prefix that makes them more detectable? For example, a key like "FooSecret.ZTNiMGM0NDI5OGZjMWMxNDlhZmJmNGM4OTk2ZmI5". I wouldn't think that'd open up any new attacks, but I'm no expert on the matter.

tnorthcutt 4y ago
This is exactly what Stripe and I believe some other companies do, partially for this exact reason.
letmeinhere 4y ago
Another approach is to identify likely ones based on the entropy of the strings. I used a tool that did precisely this once and found some, but can't find it anymore.
dghlsakjg 4y ago
Not much of a downside, but it means that they are really easy to detect for attackers as well. Really easy to just grep through something looking for that prefix
- jamesfinlayson 4y ago
  Yep - this way my thought. I've been bitten by this is in the past: an endpoint was throwing an error and dumping out environment variables (which included API keys) - the prefixed API keys were found by crawlers and abused but the unprefixed keys were untouched (though they were cycled just in case).
nikeee 4y ago
GitHub does this with the tokens they issue. They even have a checksum in the token, so they can check if the token is syntactically valid: https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica... They have a list of supported secrets they can find via automated scans: https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns https://docs.github.com/en/code-security/secret-scanning/sec...
greysteil 4y ago
GitHub PM here. We switched our own token format to something similar to the above in April of last year and have been encouraging other service providers to do the same. The big benefit of highly identifiable tokens is not just that we can alert on them, but that we can scan for them at pre-receive time and prevent them from leaking (by rejecting the push). We already have that functionality as part of GitHub Advanced Security, and are planning to make it available (for free) on public repos in 2023. [1] https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...
- ericpauley 4y ago
  One of the big issues with secret scanning now is that it’s opt-in from platforms, and from the list of supported platforms it seems like small ones may not be able to be included. The holy grail here would be to introduce a standardized token format that encodes a disclosure endpoint. Then platforms can issue tokens to this standard and receive notifications without needing to explicitly opt in.
  - yencabulator 4y ago
    > standardized token format that encodes a disclosure endpoint This should be relatively easy... secret:example.com:entropy-goes-here secret:subdomain.example.com:entropy-goes-here secret:example.com/path/optional:entropy-goes-here and then a Well-Known URI (https://en.wikipedia.org/wiki/Well-known_URI https://en.wikipedia.org/wiki/Well-known_URI) based on the embedded URL for the disclosure endpoint.
  - greysteil 4y ago
    For the secret scanning partner program we're happy to work with partners of any size - there are details of the program, including how to get in touch, at the link below.[1] However, with secret scanning alerts we look for credentials from service providers we _don't_ have a partnership with, too. Our partnerships team are pretty good, so the delta isn't that big, but Asana, Notion, Intercom and Artifactory are a few of the service providers whose tokens we scan for where we don't (yet!) have a relationship to send detections. We also scan for tokens where a partnership isn't possible or would be much harder (like HashiCorp Vault service tokens). On standardized formats, if one existed we would scan for it! However, as we've worked with dozens of service providers to update their formats we've found many have specific constraints and everyone has different preferences - as a result, for now, we're pursuing a broad church approach, rather than pushing a standard. If you haven't already read Thomas Ptacek's survey (for fly.io) I recommend it.[2] [1] https://docs.github.com/en/developers/overview/secret-scanning-partner-program https://docs.github.com/en/developers/overview/secret-scanni... [2] https://fly.io/blog/api-tokens-a-tedious-survey/ https://fly.io/blog/api-tokens-a-tedious-survey/
remram 4y ago
I argued for something like that previously on HN, like adding a domain prefix 'myservice.com_secretkeyhere'. This would allow automatic discovery of the reporting/revocation endpoint from the key. Then someone pointed out that you could just use an actual URL as your secret key and have that be the URL you visit to revoke it, and I think that is genius. Next service I make that has API keys, I will make them look like `https://secret.myservice.org/ZTNiMGM0NDI5OGZjMWM https://secret.myservice.org/ZTNiMGM0NDI5OGZjMWM`. POSTing to that URL revokes the key, a GET shows a form explaining what it is and a button to revoke the key. One issue is that some email services mangle URLs specifically, and that would be bad for keys. (edit: sudhirj is the genius: https://news.ycombinator.com/item?id=28299624 https://news.ycombinator.com/item?id=28299624)

tnorthcutt 4y ago

This is exactly what Stripe and I believe some other companies do, partially for this exact reason.

letmeinhere 4y ago

Another approach is to identify likely ones based on the entropy of the strings. I used a tool that did precisely this once and found some, but can't find it anymore.

dghlsakjg 4y ago

Not much of a downside, but it means that they are really easy to detect for attackers as well. Really easy to just grep through something looking for that prefix

jamesfinlayson 4y ago
Yep - this way my thought. I've been bitten by this is in the past: an endpoint was throwing an error and dumping out environment variables (which included API keys) - the prefixed API keys were found by crawlers and abused but the unprefixed keys were untouched (though they were cycled just in case).

nikeee 4y ago

GitHub does this with the tokens they issue. They even have a checksum in the token, so they can check if the token is syntactically valid: https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica... They have a list of supported secrets they can find via automated scans: https://docs.github.com/en/code-security/secret-scanning/secret-scanning-patterns https://docs.github.com/en/code-security/secret-scanning/sec...

greysteil 4y ago

GitHub PM here. We switched our own token format to something similar to the above in April of last year and have been encouraging other service providers to do the same. The big benefit of highly identifiable tokens is not just that we can alert on them, but that we can scan for them at pre-receive time and prevent them from leaking (by rejecting the push). We already have that functionality as part of GitHub Advanced Security, and are planning to make it available (for free) on public repos in 2023. [1] https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/ https://github.blog/2021-04-05-behind-githubs-new-authentica...

ericpauley 4y ago
One of the big issues with secret scanning now is that it’s opt-in from platforms, and from the list of supported platforms it seems like small ones may not be able to be included. The holy grail here would be to introduce a standardized token format that encodes a disclosure endpoint. Then platforms can issue tokens to this standard and receive notifications without needing to explicitly opt in.
- yencabulator 4y ago
  > standardized token format that encodes a disclosure endpoint This should be relatively easy... secret:example.com:entropy-goes-here secret:subdomain.example.com:entropy-goes-here secret:example.com/path/optional:entropy-goes-here and then a Well-Known URI (https://en.wikipedia.org/wiki/Well-known_URI https://en.wikipedia.org/wiki/Well-known_URI) based on the embedded URL for the disclosure endpoint.
- greysteil 4y ago
  For the secret scanning partner program we're happy to work with partners of any size - there are details of the program, including how to get in touch, at the link below.[1] However, with secret scanning alerts we look for credentials from service providers we _don't_ have a partnership with, too. Our partnerships team are pretty good, so the delta isn't that big, but Asana, Notion, Intercom and Artifactory are a few of the service providers whose tokens we scan for where we don't (yet!) have a relationship to send detections. We also scan for tokens where a partnership isn't possible or would be much harder (like HashiCorp Vault service tokens). On standardized formats, if one existed we would scan for it! However, as we've worked with dozens of service providers to update their formats we've found many have specific constraints and everyone has different preferences - as a result, for now, we're pursuing a broad church approach, rather than pushing a standard. If you haven't already read Thomas Ptacek's survey (for fly.io) I recommend it.[2] [1] https://docs.github.com/en/developers/overview/secret-scanning-partner-program https://docs.github.com/en/developers/overview/secret-scanni... [2] https://fly.io/blog/api-tokens-a-tedious-survey/ https://fly.io/blog/api-tokens-a-tedious-survey/

remram 4y ago

I argued for something like that previously on HN, like adding a domain prefix 'myservice.com_secretkeyhere'. This would allow automatic discovery of the reporting/revocation endpoint from the key. Then someone pointed out that you could just use an actual URL as your secret key and have that be the URL you visit to revoke it, and I think that is genius. Next service I make that has API keys, I will make them look like `https://secret.myservice.org/ZTNiMGM0NDI5OGZjMWM https://secret.myservice.org/ZTNiMGM0NDI5OGZjMWM`. POSTing to that URL revokes the key, a GET shows a form explaining what it is and a button to revoke the key. One issue is that some email services mangle URLs specifically, and that would be bad for keys. (edit: sudhirj is the genius: https://news.ycombinator.com/item?id=28299624 https://news.ycombinator.com/item?id=28299624)