5 ms·

Is the root issue here that they stored remote server database passwords in their source code? I'm always hesitant to have source code with hard coded password

by kator 12y ago

Is the root issue here that they stored remote server database passwords in their source code? I'm always hesitant to have source code with hard coded passwords, I know at some point you have to have them "somewhere" but anything you can do to reduce attack vectors for them is helpful. What do people think should be best practices for storing sensitive configuration data? I know some people will have just a config repo with very limited access but that still seems dangerous at scale when that ends up being a lot of people etc.

nknighthb 12y ago
I have yet to find any solution to this problem that I like. A separate config repo is least bad of the various bad options. The others have all been error-prone and failed to play nicely with version control. If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.
- jarrett 12y ago
  > If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal. You could probably roll your own. A decent middleground would be to keep the data in the central repo encrypted, but keep it decrypted on the engineers' workstations. Though a better approach is just to not check in sensitive values at all.
  - nknighthb 12y ago
    "Roll your own" is exactly what causes trouble. It's never right and always painful. We need a general, widely-vetted solution that doesn't make me want to yank my hair out. "Don't check in sensitive values" is not an option. Credentials must be kept track of and maintained somewhere. Storing them in an ad-hoc manner without version control doesn't just invite trouble, it demands it. We have password managers for desktops and mobile devices. We need an equivalent for servers that accounts for the fact that the credentials must be highly available, multiple people must be able to use them (whether or not they get to directly view them), some of them will be technicians with limited training following someone else's procedures, and regardless of knowledge and skill, there will be half-asleep people taking emergency actions at 3AM when shit has hit the fan.
- pavel_lishin 12y ago
  I wonder if AWS allows security-group-controlled access to databases, similar to the way you can only allow communication between two ec2 instances based on their security groups.
kachnuv_ocasek 12y ago
What about storing the credentials in environment?
jarrett 12y ago
Some people prefer to keep the keys on specialized servers when a) the need for security is especially acute, and b) the project has the budget. For more day-to-day stuff, environment variables on the server are an option. If you want to be able to provision a new server 100% automatically and keep the credentials out of source control, you have a few options. You can add a quick interactive phase to the deploy script, wherein the engineer must enter the key manually. Or the engineer could enter a value which is then passed to a KDF.
andrewjshults 12y ago
We use git-crypt (https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt) in our ansible repo for credentials. Credentials are stored as variable YAML files so anyone can work with them in templates, but only our deploy box and SRE team can encrypt/decrypt the credential files themselves. Generally works pretty well, but you do need to remember to run credential deploys on new boxes. Eventually we'll probably move to a HSM (hardware security module) for the actual storage, but those are pricey (especially since you need two of them).

by kator 12y ago

nknighthb 12y ago

I have yet to find any solution to this problem that I like. A separate config repo is least bad of the various bad options. The others have all been error-prone and failed to play nicely with version control. If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal.

jarrett 12y ago
> If someone could make git store all on-disk data in encrypted form with an ssh-agent-like solution for on-the-fly (de|en)cryption, that would seem ideal. You could probably roll your own. A decent middleground would be to keep the data in the central repo encrypted, but keep it decrypted on the engineers' workstations. Though a better approach is just to not check in sensitive values at all.
- nknighthb 12y ago
  "Roll your own" is exactly what causes trouble. It's never right and always painful. We need a general, widely-vetted solution that doesn't make me want to yank my hair out. "Don't check in sensitive values" is not an option. Credentials must be kept track of and maintained somewhere. Storing them in an ad-hoc manner without version control doesn't just invite trouble, it demands it. We have password managers for desktops and mobile devices. We need an equivalent for servers that accounts for the fact that the credentials must be highly available, multiple people must be able to use them (whether or not they get to directly view them), some of them will be technicians with limited training following someone else's procedures, and regardless of knowledge and skill, there will be half-asleep people taking emergency actions at 3AM when shit has hit the fan.
pavel_lishin 12y ago
I wonder if AWS allows security-group-controlled access to databases, similar to the way you can only allow communication between two ec2 instances based on their security groups.

kachnuv_ocasek 12y ago

What about storing the credentials in environment?

jarrett 12y ago

Some people prefer to keep the keys on specialized servers when a) the need for security is especially acute, and b) the project has the budget. For more day-to-day stuff, environment variables on the server are an option. If you want to be able to provision a new server 100% automatically and keep the credentials out of source control, you have a few options. You can add a quick interactive phase to the deploy script, wherein the engineer must enter the key manually. Or the engineer could enter a value which is then passed to a KDF.

andrewjshults 12y ago

We use git-crypt (https://github.com/AGWA/git-crypt https://github.com/AGWA/git-crypt) in our ansible repo for credentials. Credentials are stored as variable YAML files so anyone can work with them in templates, but only our deploy box and SRE team can encrypt/decrypt the credential files themselves. Generally works pretty well, but you do need to remember to run credential deploys on new boxes. Eventually we'll probably move to a HSM (hardware security module) for the actual storage, but those are pricey (especially since you need two of them).