4 ms·

When we migrate databases, we execute a Docker container with a specific environment variable to let it know that it's time to migrate. The CI environment doesn

When we migrate databases, we execute a Docker container with a specific environment variable to let it know that it's time to migrate. The CI environment doesn't actually have a network connection to the DB; it just updates the Docker images and then when the deploy task executes it starts this Docker task remotely and polls for success/failure. I think Kubernetes has some nicer features that could make this easier but I haven't had a chance to try this out yet. In fact, the CI environment and production environments are on different AWS accounts. This way if you accidentally execute tests with the wrong connection strings (and we have done this before), they just fail because they can't connect.

shoo 8y ago
> the CI environment and production environments are on different AWS accounts yep, that's one way to structurally enforce isolation (unless you go out of your way to configure cross account permissions) there's probably a bit of a tradeoff between things being safe and isolated, and things being seamlessly automated.
- gtsteve 8y ago
  Indeed, I didn't find it all that hard to configure a role in the production account that has the permission to launch containers that the CI environment can assume. I locked it down as much as possible but if the CI environment gets hacked someone could launch a container in the production environment. I did think of a way to make this more secure but I haven't done it yet - you could write a script that waits for input from a website where you have to log in via SAML. The script then passes the AWS credentials for the role back to the deployment task. Effectively, it's a second step authentication process where you'd need to be authenticated using a security key for the deployment to proceed. I haven't quite figured out all the details yet but that might help you get the assurance you need.

shoo 8y ago

> the CI environment and production environments are on different AWS accounts yep, that's one way to structurally enforce isolation (unless you go out of your way to configure cross account permissions) there's probably a bit of a tradeoff between things being safe and isolated, and things being seamlessly automated.

gtsteve 8y ago
Indeed, I didn't find it all that hard to configure a role in the production account that has the permission to launch containers that the CI environment can assume. I locked it down as much as possible but if the CI environment gets hacked someone could launch a container in the production environment. I did think of a way to make this more secure but I haven't done it yet - you could write a script that waits for input from a website where you have to log in via SAML. The script then passes the AWS credentials for the role back to the deployment task. Effectively, it's a second step authentication process where you'd need to be authenticated using a security key for the deployment to proceed. I haven't quite figured out all the details yet but that might help you get the assurance you need.