4 ms·

Couple things: 1) It is so weird to me that every cloud provider deletes backups when you delete the SQL instance. We take offsite backups that are decoupled

Couple things: 1) It is so weird to me that every cloud provider deletes backups when you delete the SQL instance. We take offsite backups that are decoupled from this process. Fortunately someone made a shell script to do this that has worked quite well: https://github.com/ovotech/cloud_sql_backup https://github.com/ovotech/cloud_sql_backup (you can then just copy the Cloud Storage buckets to S3, so when you Google account gets banned you still have your database). 2) I hate to be "that guy", but I'm starting to wonder about gitops. I really, really like having all infrastructure changes recorded in machine-readable format with history. Pry it out of my cold, dead hands. But you also lose a ton of important tools, like the diff and the sanity check before you deploy. You could have a CI rule that does the diffs, but then CI has to touch production which is not ideal. You could have CI do a dry run of HEAD and a dry run against your PR, and diff those two, but honestly no CI systems really let you check out multiple branches and use them as an input to the script, so have fun hacking that up. You end up with workarounds for workarounds and the net result is that your auditable infrastructure comes at the cost of taking a human out of the loop (and slows down experimentation). I don't think the model is quite right quite yet.

easton 5y ago
Isn't the answer to 2 to have a pre-production environment to calculate the diffs/changesets/etc against, or even to deploy the change to to test it? I know with CloudFormation, for all its faults, is pretty good at making a hypothetical changeset before it runs, and if that's not safe enough, you can just deploy it to your dev environment and use the changes succeeding and your service being up as a integration test.
whimsicalism 5y ago
I don't understand the problem. Use something like `terraform plan` as a comment on the pull request and manually look over the changes before pulling in.
- spondyl 5y ago
  I don't use Terraform but I think you're describing something like Atlantis? https://www.runatlantis.io/ https://www.runatlantis.io/ I remember seeing a team running this at a previous company I worked at
  - a012 5y ago
    Atlantis is just like the GP's mentioned: a CI to run terraform plan then a person needs to examine the dryrun output. But it's still not ideal because Atlantis also has persmissions on production, and a (human) mistake could brought down your infrastucture.
  - rmetzler 5y ago
    There is also documentation on how to integrate Terraform with Gitlab Merge Requests https://docs.gitlab.com/ee/user/infrastructure/mr_integration.html https://docs.gitlab.com/ee/user/infrastructure/mr_integratio...
- viraptor 5y ago
  Not even before pulling in - the state may change in between the comment and the applying. A safer way is to dump the plan from after the merge and not apply that specific plan until after it's manually approved.
- sontek 5y ago
  Yeah, terraform cloud from hashicorp does this
lmm 5y ago
> honestly no CI systems really let you check out multiple branches and use them as an input to the script, so have fun hacking that up. Doesn't seem hard? It would be a one-liner "git worktree" pre-build step in Jenkins, for example.
dilyevsky 5y ago
What is unideal about CI (actually CD) touching prod? In my view what’s unideal is humans touching prod. Most problems with “gitops” ime come from git itself which is terrible vcs for just about any project except the kernel and the fact that tooling is not fully idempotent or automic (eg terraform applies failing mid-way)
quickthrower2 5y ago
Sounds like you have no test or staging environment! Just apply the changes against staging. Test everything is kosher. Then merge into master to apply to prod.
rmetzler 5y ago
> 1) It is so weird to me that every cloud provider deletes backups when you delete the SQL instance. Not actually true for Google Cloud CloudSQL (at least MySQL). You can delete the instance and you don't see any backups any more in the Google Cloud Console, but they are actually there and you can restore from them. You need to know the instance name though. I think the backups will be kept for 30 days. This is the same time you can't reuse the instance name. Google could improve the UI about this a lot and display also deleted instances.
- yegle 5y ago
  Is this documented anywhere? Or is it just happen to work and may break in the future?

easton 5y ago

Isn't the answer to 2 to have a pre-production environment to calculate the diffs/changesets/etc against, or even to deploy the change to to test it? I know with CloudFormation, for all its faults, is pretty good at making a hypothetical changeset before it runs, and if that's not safe enough, you can just deploy it to your dev environment and use the changes succeeding and your service being up as a integration test.

whimsicalism 5y ago

I don't understand the problem. Use something like `terraform plan` as a comment on the pull request and manually look over the changes before pulling in.

spondyl 5y ago
I don't use Terraform but I think you're describing something like Atlantis? https://www.runatlantis.io/ https://www.runatlantis.io/ I remember seeing a team running this at a previous company I worked at
- a012 5y ago
  Atlantis is just like the GP's mentioned: a CI to run terraform plan then a person needs to examine the dryrun output. But it's still not ideal because Atlantis also has persmissions on production, and a (human) mistake could brought down your infrastucture.
- rmetzler 5y ago
  There is also documentation on how to integrate Terraform with Gitlab Merge Requests https://docs.gitlab.com/ee/user/infrastructure/mr_integration.html https://docs.gitlab.com/ee/user/infrastructure/mr_integratio...
viraptor 5y ago
Not even before pulling in - the state may change in between the comment and the applying. A safer way is to dump the plan from after the merge and not apply that specific plan until after it's manually approved.
sontek 5y ago
Yeah, terraform cloud from hashicorp does this

lmm 5y ago

> honestly no CI systems really let you check out multiple branches and use them as an input to the script, so have fun hacking that up. Doesn't seem hard? It would be a one-liner "git worktree" pre-build step in Jenkins, for example.

dilyevsky 5y ago

What is unideal about CI (actually CD) touching prod? In my view what’s unideal is humans touching prod. Most problems with “gitops” ime come from git itself which is terrible vcs for just about any project except the kernel and the fact that tooling is not fully idempotent or automic (eg terraform applies failing mid-way)

quickthrower2 5y ago

Sounds like you have no test or staging environment! Just apply the changes against staging. Test everything is kosher. Then merge into master to apply to prod.

rmetzler 5y ago

> 1) It is so weird to me that every cloud provider deletes backups when you delete the SQL instance. Not actually true for Google Cloud CloudSQL (at least MySQL). You can delete the instance and you don't see any backups any more in the Google Cloud Console, but they are actually there and you can restore from them. You need to know the instance name though. I think the backups will be kept for 30 days. This is the same time you can't reuse the instance name. Google could improve the UI about this a lot and display also deleted instances.

yegle 5y ago
Is this documented anywhere? Or is it just happen to work and may break in the future?