4 ms·

The questions are always: is it better than what we had before? Are the "problems" common to other tools? Are there better alternatives? This article doesn't ad

by pquki4 3y ago

The questions are always: is it better than what we had before? Are the "problems" common to other tools? Are there better alternatives? This article doesn't address any of these. > I heard a story once of an engineer running a “terraform destroy” on his laptop — in the wrong tab. He was in the Prod directory — and starting nuking production. Realizing his mistake, he Control-C’d out of it, but it was too late. That's the stupidest argument I have seen. If you do 'rm rf /' or 'DROP TABLE' in production, nothing can save you.

jpdb 3y ago
And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs. You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.
- jansommer 3y ago
  Yes, and usually you can make rules that prevent databases from being deleted again from IaC.
- zaphar 3y ago
  The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production. Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense. This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
  - zzyzxd 3y ago
    GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race). To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra. If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline. It should have never been called infra-as-code, but infra-as-data. See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/
    - zaphar 3y ago
      This does not change my statement at all though? You fundamentally can't really predict the impact of some changes in a given environment until it's deployed. Just because you can obtain the current state of the environment and reconcile some stuff doesn't change this.
      - zzyzxd 3y ago
        That's why you should call what you store in git the _desired_ state, not anything else. A git repository is not a live database. It's a collection of static text files that change less often than your live system. There will be bugs and misconfiguration, and sometimes the desired state is just technically not reachable, and that's fine. What the actual state is doesn't matter. Leave that to the controller. State drifting is a problem your gitops engine should detect, and should be fixed by the owner of controller code. Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.
        zaphar 3y ago
        This is correct. You need some kind of running check on the environment and when possible code that handle exceptional cases. Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".
thomascgalvin 3y ago
This article is basically one long "be careful, that hammer can hit your thumb." While that's true, it doesn't mitigate the hammer's usefulness, nor does it propose a better alternative. Technology is hard. Good tools make it easier, but they never make it trivial. You still need to know what you're doing.
deleted 3y ago
[deleted]
nonethewiser 3y ago
It doesnt absolve the whole article but a delete command for your entire infrastructure is obviously a major footgun that doesnt exist with alternatives (to use one of your own lenses).
- groestl 3y ago
  "It takes an admin to take down a server, it takes a configuration management system to take down a site".
- jolux 3y ago
  It doesn't exist with Terraform either if you are following least privilege and using a cloud provider.
  - pksebben 3y ago
    Or if you just use the terraform cli properly and actually glance at the report it gives you before the “hey guy are you really, really sure about this?” Safety prompt. You not paying attention is not terraforms fault.
- adwf 3y ago
  Eh, Azure has a big delete button for any Subscription. Seen it happen and take down a production system. Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?
  - kthejoker2 3y ago
    To be fair when you click the button: * it throws a.big warning text * it lists out all the resources you're deleting * you have to type out the name of the subscription * click another button * and then another yes I'm sure button with more warning text At the end of the day they have to provide a button.
zmgsabst 3y ago
In particular, IaC replaced “thousands of lines of YAML and regex” or “magic shell script” solutions. IaC is much better than what it replaced, even if there are still many sharp corners.
cjonas 3y ago
The "as code" means the quality and stability of the implementation varies wildly. If you use a declarative style with proper SDLC in place, then it's fantastic. On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.
SideburnsOfDoom 3y ago
> If you do 'DROP TABLE' in production, nothing can save you. Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'
nojvek 3y ago
That’s not infrastructure as code means. It means engineers wouldn’t have ability to run terraform on laptop at all. They have to make a PR, get it reviewed and the code lands in some prod level branch where CI would run terraform for them. And if things got messed up, revert the commit and we are back (mostly - May need backup restores). It means your infrastructure is a commit that you can see history and forward/reverse. Git represents your infra. If it’s not in the repo, it’s not there.

by pquki4 3y ago

jpdb 3y ago

And if you're pairing your infra-as-code with a gitops model then you can help prevent these kinds of issues with PRs. You can also use your git history to restore the infrastructure itself. You may lose some data, but it's also possible to have destroyed resources retain their data or backup before destroy.

jansommer 3y ago
Yes, and usually you can make rules that prevent databases from being deleted again from IaC.
zaphar 3y ago
The problem with infra-as-code and gitops is that it's often nearly impossible to tell what will actually happen with a PR without running it somewhere. Which is 1. expensive. and 2. nearly impossible to get to mirror production. Production and staging are the farthest you can get from pure immutable environment that you can get. They carry state around all over the place. It's their entire reason for existing in some sense. This means that while git-ops can be helpful in some ways it can also be incredibly dangerous in others. I'm not entirely sure it doesn't all come out in the wash in the end.
- zzyzxd 3y ago
  GitOps is just like "DevOps" -- you don't really know what it means to a specific org until you talk to them, because people interpret it differently based on their own understanding (or if they have a horse in this race). To me it always means describing the desired state of your infra in structured data, storing that in git, and run controller to reconcile it against the actual infra. If your GitOps engine has to compile/run the "code" to uncover the desired state, that defeats the purpose of GitOps and is no better than running your hand crafted release bash script in a CI/CD pipeline. It should have never been called infra-as-code, but infra-as-data. See "The Rendered Manifests Pattern": https://akuity.io/blog/the-rendered-manifests-pattern/ https://akuity.io/blog/the-rendered-manifests-pattern/
  - zaphar 3y ago
    This does not change my statement at all though? You fundamentally can't really predict the impact of some changes in a given environment until it's deployed. Just because you can obtain the current state of the environment and reconcile some stuff doesn't change this.
    - zzyzxd 3y ago
      That's why you should call what you store in git the _desired_ state, not anything else. A git repository is not a live database. It's a collection of static text files that change less often than your live system. There will be bugs and misconfiguration, and sometimes the desired state is just technically not reachable, and that's fine. What the actual state is doesn't matter. Leave that to the controller. State drifting is a problem your gitops engine should detect, and should be fixed by the owner of controller code. Some companies practice infra-as-code, point to their git repo and tell me "this is our single source of truth" of our infrastructure. And I have to tell them that statement is wrong.
      - zaphar 3y ago
        This is correct. You need some kind of running check on the environment and when possible code that handle exceptional cases. Sometimes that's as simple as a service that shoots other services in the head to restart them. Othertimes it's more complicated. But lot's of places can't afford to get more complicated than "alert a human and have them look at it".

thomascgalvin 3y ago

This article is basically one long "be careful, that hammer can hit your thumb." While that's true, it doesn't mitigate the hammer's usefulness, nor does it propose a better alternative. Technology is hard. Good tools make it easier, but they never make it trivial. You still need to know what you're doing.

deleted 3y ago

[deleted]

nonethewiser 3y ago

It doesnt absolve the whole article but a delete command for your entire infrastructure is obviously a major footgun that doesnt exist with alternatives (to use one of your own lenses).

groestl 3y ago
"It takes an admin to take down a server, it takes a configuration management system to take down a site".
jolux 3y ago
It doesn't exist with Terraform either if you are following least privilege and using a cloud provider.
- pksebben 3y ago
  Or if you just use the terraform cli properly and actually glance at the report it gives you before the “hey guy are you really, really sure about this?” Safety prompt. You not paying attention is not terraforms fault.
adwf 3y ago
Eh, Azure has a big delete button for any Subscription. Seen it happen and take down a production system. Thankfully they also must've realised this was a stupid button and provide a 90-day cooldown period on deleted subscriptions... but why have the easy button in the first place?
- kthejoker2 3y ago
  To be fair when you click the button: * it throws a.big warning text * it lists out all the resources you're deleting * you have to type out the name of the subscription * click another button * and then another yes I'm sure button with more warning text At the end of the day they have to provide a button.

zmgsabst 3y ago

In particular, IaC replaced “thousands of lines of YAML and regex” or “magic shell script” solutions. IaC is much better than what it replaced, even if there are still many sharp corners.

cjonas 3y ago

The "as code" means the quality and stability of the implementation varies wildly. If you use a declarative style with proper SDLC in place, then it's fantastic. On the other hand, I'm currently reviewing a project using python CDK where the branch logic and imperatives style makes it extremely hard to build a mental model.

SideburnsOfDoom 3y ago

> If you do 'DROP TABLE' in production, nothing can save you. Not having permission to do that in production unless you specifically elevate, can save you. But yes, the exact same precaution holds for 'DELETE INFRA'

nojvek 3y ago

That’s not infrastructure as code means. It means engineers wouldn’t have ability to run terraform on laptop at all. They have to make a PR, get it reviewed and the code lands in some prod level branch where CI would run terraform for them. And if things got messed up, revert the commit and we are back (mostly - May need backup restores). It means your infrastructure is a commit that you can see history and forward/reverse. Git represents your infra. If it’s not in the repo, it’s not there.