5 ms·

I was trying to deploy a CI/CD pipeline for a small project and testing some changes in the test step before deploying. I wanted to collect env variables into a

I was trying to deploy a CI/CD pipeline for a small project and testing some changes in the test step before deploying. I wanted to collect env variables into a global location in the pipeline and put some defaults for the tests, but I have used the same variable names for them, so the global ones that were for production deployment were being used for the tests as well, and the tests were starting by wiping out the connected database. As if this was not enough, I didn't understand what was going on because tests that used to take 5 seconds were taking 3 minutes at that point, so I ran them at least 10 times to make sure this was not a pipeline glitch. This also helped me to make sure that the database was completely wiped.

shoo 8y ago
how could we structure things differently to prevent anything remotely like this from ever happening?
- gtsteve 8y ago
  When we migrate databases, we execute a Docker container with a specific environment variable to let it know that it's time to migrate. The CI environment doesn't actually have a network connection to the DB; it just updates the Docker images and then when the deploy task executes it starts this Docker task remotely and polls for success/failure. I think Kubernetes has some nicer features that could make this easier but I haven't had a chance to try this out yet. In fact, the CI environment and production environments are on different AWS accounts. This way if you accidentally execute tests with the wrong connection strings (and we have done this before), they just fail because they can't connect.
  - shoo 8y ago
    > the CI environment and production environments are on different AWS accounts yep, that's one way to structurally enforce isolation (unless you go out of your way to configure cross account permissions) there's probably a bit of a tradeoff between things being safe and isolated, and things being seamlessly automated.
    - gtsteve 8y ago
      Indeed, I didn't find it all that hard to configure a role in the production account that has the permission to launch containers that the CI environment can assume. I locked it down as much as possible but if the CI environment gets hacked someone could launch a container in the production environment. I did think of a way to make this more secure but I haven't done it yet - you could write a script that waits for input from a website where you have to log in via SAML. The script then passes the AWS credentials for the role back to the deployment task. Effectively, it's a second step authentication process where you'd need to be authenticated using a security key for the deployment to proceed. I haven't quite figured out all the details yet but that might help you get the assurance you need.
- chris11 8y ago
  One safeguard you can add is to use a different database for the tests with different environmental variable names and db roles. So the unit-tests should never be able to access a non-test db.
- deleted 8y ago
  [deleted]
luckman212 8y ago
No db backups?

shoo 8y ago

how could we structure things differently to prevent anything remotely like this from ever happening?

gtsteve 8y ago
When we migrate databases, we execute a Docker container with a specific environment variable to let it know that it's time to migrate. The CI environment doesn't actually have a network connection to the DB; it just updates the Docker images and then when the deploy task executes it starts this Docker task remotely and polls for success/failure. I think Kubernetes has some nicer features that could make this easier but I haven't had a chance to try this out yet. In fact, the CI environment and production environments are on different AWS accounts. This way if you accidentally execute tests with the wrong connection strings (and we have done this before), they just fail because they can't connect.
- shoo 8y ago
  > the CI environment and production environments are on different AWS accounts yep, that's one way to structurally enforce isolation (unless you go out of your way to configure cross account permissions) there's probably a bit of a tradeoff between things being safe and isolated, and things being seamlessly automated.
  - gtsteve 8y ago
    Indeed, I didn't find it all that hard to configure a role in the production account that has the permission to launch containers that the CI environment can assume. I locked it down as much as possible but if the CI environment gets hacked someone could launch a container in the production environment. I did think of a way to make this more secure but I haven't done it yet - you could write a script that waits for input from a website where you have to log in via SAML. The script then passes the AWS credentials for the role back to the deployment task. Effectively, it's a second step authentication process where you'd need to be authenticated using a security key for the deployment to proceed. I haven't quite figured out all the details yet but that might help you get the assurance you need.
chris11 8y ago
One safeguard you can add is to use a different database for the tests with different environmental variable names and db roles. So the unit-tests should never be able to access a non-test db.
deleted 8y ago
[deleted]

luckman212 8y ago

No db backups?