40 ms·

"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking

by ajross 4y ago

"Production keys in source control" is right up there with "mistaken routing table entry" and "fat-fingered DNS config" on the list of critical company-breaking mistakes that you'd think would be easy to avoid, but aren't.

ElevenLathe 4y ago
Add "failure to rotate TLS certs before they expire."
- HWR_14 4y ago
  But at least that failure just makes the services fail, not opens a security hole.
  - cmeacham98 4y ago
    Except in the scenarios where the company's support starts telling users to click through the warning, which I've seen a few times.
    - HWR_14 4y ago
      Yuck. I hadn't considered that.
  - deleted 4y ago
    [deleted]
- deathanatos 4y ago
  My company has monitoring for this, but it still seems to be a law of nature that, 1. someone adds new service/server/infra in a submarine manner 2. it goes to prod 3. the cert expires and outage begins 4. my team is asked what to do, because "we're the cert experts" 5. we add it to the monitoring So it only happens once … per service. Which isn't great. But how do you get people to slow down and do simple shit, like add a monitor to your new service? (Or promulgate a design doc…)
  - ElevenLathe 4y ago
    I think the real answer is to only issue limited-duration certs and only via automated means (ACME or similar), thus requiring automation be in place from day 1. This still doesn't protect against the vector where somebody else in the company has managed to prove themselves to be responsible parties to another CA/issuer.
    - deathanatos 4y ago
      Oh I agree! Everything would be ACME if I could. A ridiculous amount of stuff still doesn't support it, though. And, like I said, usually it's someone who doesn't grok certs doing it without asking for help in the first place, so they're not going to get why ACME. (Because I am tired of doing cert renewals. I've had enough for a lifetime…)
  - deleted 4y ago
    [deleted]
  - HL33tibCe7 4y ago
    “If you keep smelling shit, look at your own shoe”. Your processes are failing your development teams, and you need to fix them, rather than blaming your teams, which achieves nothing.
    - deathanatos 4y ago
      I agree, but that level of authority is in a different level of the foodchain. I'm on duck tales, Larry.
  - D-Coder 4y ago
    1. Clearly describe the correct process in your other process documentation. 2. Email everyone who might be involved a note about this and a link to the documentation and why it is important. 3. Next time someone ignores it, rip them and their manager a new orifice. 4. Wait for word of #3 to spread. Might help...
    - 4y ago
coenhyde 4y ago
Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.
- deathanatos 4y ago
  And yet I see it get violated all the time. People should do a lot of things, but a lot of my coworkers are lazy and do not do quality work. Given that it happens, and that I can't prevent it, one must then ask how to guard against it. At my org, we even try to generate all secrets with a standardize prefix/suffix so as to make them very greppable. That doesn't stop "Architects", "Customer Solutions", "Analytics" types from … just working around the standard tooling and manually generating one by hand because … IDK, they think they know better? I really don't get it.
  - coenhyde 4y ago
    Doctors used to not wash their hands too. I get it though, and i've seen the same thing. Really it comes down to education and not granting access to secrets to people who aren't capable of handling them.
    - robswc 4y ago
      "fun" fact - there could potentially be thousands of deaths attributed to Drs simply not washing their hands. IIRC they even basically got some hospital admin fired for creating a hand washing mandate, despite it being proven to save lives. https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... (talking centuries ago, but maybe even today) Looks like its still a "recent" issue, lol https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.html https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.htm...
      - 4y ago
wereHamster 4y ago
> Production keys in source control IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.
- JauntyHatAngle 4y ago
  Huh? No, you use an external secrets manager or leave it to run time environment vars, or leave it to your cd servers to access/supply those details. Assuming you have been given the right tooling, there is no reason for it to be in source code.
  - danenania 4y ago
    If anyone out there is using environment variables currently, and is interested a quick path to plugging the leaks in their secrets management, check out EnvKey[1] (disclaimer: I'm the founder). Because EnvKey integrates tightly with environment variables, no app code changes are needed to switch, so it only takes a minute or two to import/integrate a typical app. EnvKey is designed to help avoid incidents exactly like the one that just hit Toyota, while staying out of your way and even making your life significantly easier as a developer by simplifying many aspects of config management. Give it a look if you know you have some room for improvement in this area and are looking for an easy, secure, open source, end-to-end encrypted solution :) 1 - https://envkey.com https://envkey.com
- moooo99 4y ago
  Even with IaC you should store secrets only in your environment variables for the CI servers
- coenhyde 4y ago
  You can put your keys in source control if you are encrypting them with another key which is not in source control. Otherwise you're doing it wrong.
  - Too 4y ago
bonestamp2 4y ago
I committed my google maps api key to a public github repository recently and github immediately sent me a warning about it. The thing is, I did it intentionally. The key is used on my website and the website is served by github pages. Now, it's an embedded maps api key, there's no cost to use it, nobody can use it from a domain other than mine, and it's easily visible in the page source if someone views that, so there's really no reason not to commit it since even if I didn't commit to somewhere publically, it's still publicly available on my website and nobody else can use it anyway. Is this a reasonable exception to this rule?
- tomschwiha 4y ago
  I just wondered what happens if you spoof locally the domain name. Could you still use the api key?
  - bonestamp2 4y ago
    Maybe. Thankfully there's no cost to use the key if someone does it.
- prepend 4y ago
  I could run your site locally with a customized host file so the referers all come from your domain. I don’t think it’s that much of a risk but I wouldn’t want to use a key associated with something that can bill me. You could use Google actions to build your pages site injecting the api key at build time. It’s stored a repo secret rather than in code. Of course since you deploy the site publicly, the key will still be visible.

by ajross 4y ago

ElevenLathe 4y ago

Add "failure to rotate TLS certs before they expire."

HWR_14 4y ago
But at least that failure just makes the services fail, not opens a security hole.
- cmeacham98 4y ago
  Except in the scenarios where the company's support starts telling users to click through the warning, which I've seen a few times.
  - HWR_14 4y ago
    Yuck. I hadn't considered that.
- deleted 4y ago
  [deleted]
deathanatos 4y ago
My company has monitoring for this, but it still seems to be a law of nature that, 1. someone adds new service/server/infra in a submarine manner 2. it goes to prod 3. the cert expires and outage begins 4. my team is asked what to do, because "we're the cert experts" 5. we add it to the monitoring So it only happens once … per service. Which isn't great. But how do you get people to slow down and do simple shit, like add a monitor to your new service? (Or promulgate a design doc…)
- ElevenLathe 4y ago
  I think the real answer is to only issue limited-duration certs and only via automated means (ACME or similar), thus requiring automation be in place from day 1. This still doesn't protect against the vector where somebody else in the company has managed to prove themselves to be responsible parties to another CA/issuer.
  - deathanatos 4y ago
    Oh I agree! Everything would be ACME if I could. A ridiculous amount of stuff still doesn't support it, though. And, like I said, usually it's someone who doesn't grok certs doing it without asking for help in the first place, so they're not going to get why ACME. (Because I am tired of doing cert renewals. I've had enough for a lifetime…)
- deleted 4y ago
  [deleted]
- HL33tibCe7 4y ago
  “If you keep smelling shit, look at your own shoe”. Your processes are failing your development teams, and you need to fix them, rather than blaming your teams, which achieves nothing.
  - deathanatos 4y ago
    I agree, but that level of authority is in a different level of the foodchain. I'm on duck tales, Larry.
- D-Coder 4y ago
  1. Clearly describe the correct process in your other process documentation. 2. Email everyone who might be involved a note about this and a link to the documentation and why it is important. 3. Next time someone ignores it, rip them and their manager a new orifice. 4. Wait for word of #3 to spread. Might help...
  - 4y ago

coenhyde 4y ago

Those three are not all equal. "Production keys in source control" is the equivalent of a surgeon not washing their hands between between surgeries. It's basic level of professional competency that should not be violated. The latter two are bad mistakes, which shouldn't happen but do.

deathanatos 4y ago
And yet I see it get violated all the time. People should do a lot of things, but a lot of my coworkers are lazy and do not do quality work. Given that it happens, and that I can't prevent it, one must then ask how to guard against it. At my org, we even try to generate all secrets with a standardize prefix/suffix so as to make them very greppable. That doesn't stop "Architects", "Customer Solutions", "Analytics" types from … just working around the standard tooling and manually generating one by hand because … IDK, they think they know better? I really don't get it.
- coenhyde 4y ago
  Doctors used to not wash their hands too. I get it though, and i've seen the same thing. Really it comes down to education and not granting access to secrets to people who aren't capable of handling them.
  - robswc 4y ago
    "fun" fact - there could potentially be thousands of deaths attributed to Drs simply not washing their hands. IIRC they even basically got some hospital admin fired for creating a hand washing mandate, despite it being proven to save lives. https://www.npr.org/sections/health-shots/2015/01/12/375663920/the-doctor-who-championed-hand-washing-and-saved-women-s-lives https://www.npr.org/sections/health-shots/2015/01/12/3756639... (talking centuries ago, but maybe even today) Looks like its still a "recent" issue, lol https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.html https://www.nytimes.com/2006/09/24/magazine/24wwln_freak.htm...
    - 4y ago

wereHamster 4y ago

> Production keys in source control IaC, right? If you don't put keys into the Code you can't have Infrastructure as Code. Without keys the code only partially defines your infrastructure.

JauntyHatAngle 4y ago
Huh? No, you use an external secrets manager or leave it to run time environment vars, or leave it to your cd servers to access/supply those details. Assuming you have been given the right tooling, there is no reason for it to be in source code.
- danenania 4y ago
  If anyone out there is using environment variables currently, and is interested a quick path to plugging the leaks in their secrets management, check out EnvKey[1] (disclaimer: I'm the founder). Because EnvKey integrates tightly with environment variables, no app code changes are needed to switch, so it only takes a minute or two to import/integrate a typical app. EnvKey is designed to help avoid incidents exactly like the one that just hit Toyota, while staying out of your way and even making your life significantly easier as a developer by simplifying many aspects of config management. Give it a look if you know you have some room for improvement in this area and are looking for an easy, secure, open source, end-to-end encrypted solution :) 1 - https://envkey.com https://envkey.com
moooo99 4y ago
Even with IaC you should store secrets only in your environment variables for the CI servers
coenhyde 4y ago
You can put your keys in source control if you are encrypting them with another key which is not in source control. Otherwise you're doing it wrong.
- Too 4y ago

bonestamp2 4y ago

I committed my google maps api key to a public github repository recently and github immediately sent me a warning about it. The thing is, I did it intentionally. The key is used on my website and the website is served by github pages. Now, it's an embedded maps api key, there's no cost to use it, nobody can use it from a domain other than mine, and it's easily visible in the page source if someone views that, so there's really no reason not to commit it since even if I didn't commit to somewhere publically, it's still publicly available on my website and nobody else can use it anyway. Is this a reasonable exception to this rule?

tomschwiha 4y ago
I just wondered what happens if you spoof locally the domain name. Could you still use the api key?
- bonestamp2 4y ago
  Maybe. Thankfully there's no cost to use the key if someone does it.
prepend 4y ago
I could run your site locally with a customized host file so the referers all come from your domain. I don’t think it’s that much of a risk but I wouldn’t want to use a key associated with something that can bill me. You could use Google actions to build your pages site injecting the api key at build time. It’s stored a repo secret rather than in code. Of course since you deploy the site publicly, the key will still be visible.