6 ms·

> Using their initial foothold of OAuth user tokens for GitHub.com, the actor was able to exfiltrate a set of private npm repositories, some of which included s

> Using their initial foothold of OAuth user tokens for GitHub.com, the actor was able to exfiltrate a set of private npm repositories, some of which included secrets such as AWS access keys. Keep those keys out of source control, folks. There are a lot of options for secrets management these days, and making it harder for attackers to totally own you if they only manage to crack one piece of your infrastructure is key to limiting damage from this sort of attack.

dgb23 4y ago
This is also a thing that will not get past some interviewers.
urda 4y ago
A rule of thumb is: if the key is pushed into a git remote you should consider it compromised and roll a new key.
- celticninja 4y ago
  What if they key is in a git-crypted file? I get what you are saying about an open file, but surely best practice is to use encrypted files to store secrets that are needed e.g for deployment
  - urda 4y ago
    An encrypted file stored along with the git repo (without the decryption key) has a different attack surface. My original comment was more targeted towards users storing their keys in plaintext.
joshstrange 4y ago
I agree fully with you but I don't think secret management is as easy/cheap as some people pretend. On AWS, for example, each secret you store is $0.40 + $0.05 per 10,000 API calls and that can add up if you only have 1 api key/password/etc per "Secret" (for an individual at least, I hate bleeding off $10+/mo to store tiny bits of strings). Then, once you have the secret stored, you need setup roles/policies to be able to retrieve them. I have this setup pretty well in my code now but getting there wasn't simple or easy from my perspective and keeping the list of secrets your IAM user can access up to date can be a pain as well. I'm working in a lambda environment so my options might be more limited but I'm interested to see how other people are solving this issue (maybe specifically for small/side projects). As it stands my lambdas all get a role applied to them that gives them access to the secrets but something not AWS-specific would need a "bootstrap secret" to be injected before the code could call out to the third-party to get the other secrets. For Lambdas I suppose I could inject that "bootstrap secret" in via environmental variables but now I've got a new issue to deal with. Injecting at build time via something like GitHub Actions Secrets is an option I guess. All that to say, while I agree secrets should never be in source, in practice it's not super easy (I'd love to be proven wrong, maybe I'm not doing it right).
- jeremymcanally 4y ago
  We use Doppler and some other custom built tooling, and it works really well. The Doppler pricing is really fair IMO, and their tooling adds a lot of value for us so I'm OK with paying for it. Software security is rarely free (even with an OSS tool you've got infra and management costs), but the cost is almost always cheaper than a major breach that could stem from something like this incident, which fortunately was pretty contained.
- danenania 4y ago
  You could check out https://www.envkey.com https://www.envkey.com (disclaimer: I'm the founder). Our goal is to simplify this as much as possible. We use client-side end-to-end encryption so you don't have to trust a third party, and pricing on our paid plans is based on number of users, not number of secrets.
  - philsnow 4y ago
    Having helped build the production secret management bits at Airbnb [0], I would encourage most people to buy, not build. It was a large task and I'm 95% sure we didn't add more value than the cost of all the people working on it for as long as it took, compared to buying a solution. Just like the rest of Airbnb started before VPCs were GA and thus required a large engineering investment to move everything to VPCs, we started on the secret management stuff before there were a lot of good other options available (though arguably Hashicorp Vault was around and mature enough at the time, and would have been the best alternative). I haven't looked at envkey for production use but I've definitely considered it for home use since it's just so deliciously simple. [0] https://medium.com/airbnb-engineering/production-secret-management-at-airbnb-ad230e1bc0f6 https://medium.com/airbnb-engineering/production-secret-mana...
  - eins1234 4y ago
    I was looking at this with great interest, until I looked at the pricing and found out the lowest hosted pricing tier starts at $150/month. I know there's a free "community" hosted version, but I'm not sure what the differences is outside of the limits and support, and I'd prefer to see the pricing scale up a bit more gently than 0 -> $150 as soon as I reach the limits of the free offering.

dgb23 4y ago

This is also a thing that will not get past some interviewers.

urda 4y ago

A rule of thumb is: if the key is pushed into a git remote you should consider it compromised and roll a new key.

celticninja 4y ago
What if they key is in a git-crypted file? I get what you are saying about an open file, but surely best practice is to use encrypted files to store secrets that are needed e.g for deployment
- urda 4y ago
  An encrypted file stored along with the git repo (without the decryption key) has a different attack surface. My original comment was more targeted towards users storing their keys in plaintext.

joshstrange 4y ago

I agree fully with you but I don't think secret management is as easy/cheap as some people pretend. On AWS, for example, each secret you store is $0.40 + $0.05 per 10,000 API calls and that can add up if you only have 1 api key/password/etc per "Secret" (for an individual at least, I hate bleeding off $10+/mo to store tiny bits of strings). Then, once you have the secret stored, you need setup roles/policies to be able to retrieve them. I have this setup pretty well in my code now but getting there wasn't simple or easy from my perspective and keeping the list of secrets your IAM user can access up to date can be a pain as well. I'm working in a lambda environment so my options might be more limited but I'm interested to see how other people are solving this issue (maybe specifically for small/side projects). As it stands my lambdas all get a role applied to them that gives them access to the secrets but something not AWS-specific would need a "bootstrap secret" to be injected before the code could call out to the third-party to get the other secrets. For Lambdas I suppose I could inject that "bootstrap secret" in via environmental variables but now I've got a new issue to deal with. Injecting at build time via something like GitHub Actions Secrets is an option I guess. All that to say, while I agree secrets should never be in source, in practice it's not super easy (I'd love to be proven wrong, maybe I'm not doing it right).

jeremymcanally 4y ago
We use Doppler and some other custom built tooling, and it works really well. The Doppler pricing is really fair IMO, and their tooling adds a lot of value for us so I'm OK with paying for it. Software security is rarely free (even with an OSS tool you've got infra and management costs), but the cost is almost always cheaper than a major breach that could stem from something like this incident, which fortunately was pretty contained.
danenania 4y ago
You could check out https://www.envkey.com https://www.envkey.com (disclaimer: I'm the founder). Our goal is to simplify this as much as possible. We use client-side end-to-end encryption so you don't have to trust a third party, and pricing on our paid plans is based on number of users, not number of secrets.
- philsnow 4y ago
  Having helped build the production secret management bits at Airbnb [0], I would encourage most people to buy, not build. It was a large task and I'm 95% sure we didn't add more value than the cost of all the people working on it for as long as it took, compared to buying a solution. Just like the rest of Airbnb started before VPCs were GA and thus required a large engineering investment to move everything to VPCs, we started on the secret management stuff before there were a lot of good other options available (though arguably Hashicorp Vault was around and mature enough at the time, and would have been the best alternative). I haven't looked at envkey for production use but I've definitely considered it for home use since it's just so deliciously simple. [0] https://medium.com/airbnb-engineering/production-secret-management-at-airbnb-ad230e1bc0f6 https://medium.com/airbnb-engineering/production-secret-mana...
- eins1234 4y ago
  I was looking at this with great interest, until I looked at the pricing and found out the lowest hosted pricing tier starts at $150/month. I know there's a free "community" hosted version, but I'm not sure what the differences is outside of the limits and support, and I'd prefer to see the pricing scale up a bit more gently than 0 -> $150 as soon as I reach the limits of the free offering.