7 ms·

I'm curious why people use configuration management software in 2020. All of that seems like the old way of approaching problems to me. What I prefer to do is

I'm curious why people use configuration management software in 2020. All of that seems like the old way of approaching problems to me. What I prefer to do is use Terraform to create immutable infrastructure from code. CoreOS and most Linux variants can be configured at boot time (cloud-config, Ignition, etc) to start and run a certain workload. Ideally, all of your workloads would be containerised, so there's no need for configuration drift, or for any management software to be running on the box. If you need to update something, create the next version of your immutable machine and replace the existing ones.

notyourday 7y ago
> What I prefer to do is use Terraform to create immutable infrastructure from code. Can you mount all your volumes read-only and run all of your stack? If you cannot, then you do not have immutable infrastructure. You simply happen to agree that no one write anything useful, which with time will absolutely fail because someone, somewhere is going to start storing state on a stateless system giving you "a cow #378 called 'Betsy'"
- tilolebo 7y ago
  In the current state of infrastructure, an accepted definition of "immutable infrastructure" is that: 1. You deploy a completely fresh instance/container, instead of in-place updates 2. You don't actively push changes on a running instance/container Of course you might have stuff written to disk, such as logs, temp files, etc. But it should be non-essential data, and potentially pushed to a central place in near real-time.
  - HelloNurse 7y ago
    Interesting. How would you do that if your deployment is, say, a couple of new tables in a 50TB Oracle database?
    - tilolebo 7y ago
      It only works with stateless resources. There's no point in trying to manage a database or similar resources this way.
king_phil 7y ago
But... How do you configure the hosts where your containers are running on? How do you configure your storage (NAS/SAN)? How do you configure your routers and switches? ...
- grrywlsn 7y ago
  The original question didn't have much context, and I guess my answer assumed someone would be using a cloud provider as opposed to anything on premise. Are Ansible/Puppet/Chef any good for managing the hardware you mentioned?
  - notacoward 7y ago
    > Are Ansible/Puppet/Chef any good for managing the hardware you mentioned? Yes. Well, OK, maybe not good but better than ad hoc.
  - throw0101a 7y ago
    > Are Ansible/Puppet/Chef any good for managing the hardware you mentioned? Ansible is used in networking, which many vendors having official modules: * https://github.com/PaloAltoNetworks/ansible-pan https://github.com/PaloAltoNetworks/ansible-pan * https://github.com/aristanetworks/ansible-cvp https://github.com/aristanetworks/ansible-cvp * https://www.juniper.net/documentation/en_US/junos-ansible/information-products/pathway-pages/junos-ansible.html https://www.juniper.net/documentation/en_US/junos-ansible/in... * https://github.com/F5Networks/f5-ansible https://github.com/F5Networks/f5-ansible There are even frameworks if you want to write things in 'raw' Python as well.
- jcrben 7y ago
  Build the underlying vms with packer. Or use cloud-init as the parent mentioned - I think it has a bunch of knobs.
dillonmckay 7y ago
Not all of us have the luxury of our projects being greenfield.
- grrywlsn 7y ago
  The question asks what I would consider to be the right approach for 2020, and also what my team is doing. This is the design pattern I've been following for 5 years, but obviously your mileage may vary, it won't work for everyone, etc.
whatsmyusername 7y ago
I mostly don't for new stuff (all in on Docker/ECS), however we have a lot of old stuff and things in the process of being migrated where it makes sense. There's also always the odd bird thing you use that needs to run on a regular host.
- grrywlsn 7y ago
  (Genuinely curious) what old stuff do you think doesn't make sense to be set up immutably? and what odd stuff needs to run on a regular host?
  - detaro 7y ago
    Example: How do you do "immutable" management of Mac OS machines? Taking what's typically described as such there, you've just turned a 30s deploy of software into a multi-hour "lets reimage the entire machine"? (although that's of course not strictly "old stuff")
    - grrywlsn 7y ago
      Were Macs in scope of the original post? I assumed it was server side stuff, rather than office hardware. For that, though, I'd use Jamf (Pro) or some other MDM option.
      - detaro 7y ago
        Even if you exclude "office hardware" from configuration management, our Mac OS build and test farm is "servers" I'd say. Not everyone running servers is doing so to run an online service on a platform of their choice.
  - jcheng 7y ago
    Not the GP, but some proprietary software requires license activation and you only get a certain (small) number of activates/deactivates.
  - deleted 7y ago
    [deleted]
notacoward 7y ago
"Immutable infrastructure" what a laugh. In a large deployment, configuration somewhere is always changing - preferably without restarting tasks because they're constantly loaded. We have (most) configuration under source control, and during the west-coast work day it is practically impossible to commit a change without hitting conflicts and having to rebase. Then there are machines not running production workloads, such as development machines or employees' laptops, which still need to have their configuration managed. Are you going to "immutable infrastructure" everyone's laptops? (Context: my team manages dozens of clusters, each with a score of services across thousands of physical hosts. Every minute of every day, multiple things are being scaled up or down, tuned, rearranged to deal with hardware faults or upgrades, new features rolled out, etc. Far from being immutable, this infrastructure is remarkably fluid because that's the only way to run things at such scale.) Beware of Chesterton's Fence. Just because you haven't learned the reasons for something doesn't mean it's wrong, and the new shiny often re-introduces problems that were already solved (along with some of its own) because of that attitude.
- nickthemagicman 7y ago
  Sometimes you have to work with what you're given in a brownfield env and a config managment tool is useful in that case, but it's possible that you are working with a less than ideal architecture with less than ideal time/money to make changes. State is always the enemy in technology. I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.
  - notacoward 7y ago
    > State is always the enemy in technology. I work in data storage. Am I the enemy, then? ;) > can't even imagine managing hundreds of servers whose state is unpredictable at any moment Be careful not to conflate immutability with predictability. The state of these servers is predictable. All of the information necessary to reconstruct them is on a single continuous timeline in source control. But that doesn't mean they're immutable because the head of that timeline is moving very rapidly. > can't be terminated and replaced with a fresh instance for fear of losing something. No, there's (almost) no danger of losing any data because everything's erasure-coded at a level of redundancy that most people find surprising until they learn the reasons (e.g. large-scale electrical outages). But there's definitely a danger of losing availability. You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming. Rolling changes are an absolute requirement. Some take minutes. Some take hours. Some take days. Many of these services have run continuously for years, barely resembling the code or config they had when they first started, and their users wouldn't have it any other way. It might be hard to imagine, but it's an every-day reality for my team.
deleted 7y ago
[deleted]
detaro 7y ago
So, Linux-only? ;)
- grrywlsn 7y ago
  Yes, but to be clear, some of those containers have been .Net Core containers (running in Kubernetes) for me. I appreciate not having Windows in an estate isn't common to all setups.
mneil 7y ago
I'm going to agree with you. In 2020 (and really the last few years), configuration management is outdated. IaC (infrastructure as code) is the current approach. Containerize everything you can, use terraform or cloudformation, or azure devops. Avoid managing the underlying os as much as possible. Use vanilla or prebuilt images to deploy these containers on, coreos, Amazon's new bottle rocket (maybe). Or use a service like fargate when possible. All configuration should be declarative to avoid errors. If you need to build images tools like packer are great. AWS has a recommended "golden Ami pipeline" pattern and a new image builder service if you can't use community images. I'm speaking imperatively but read these as my own directives. I work for a company that consults and actively helps fortune 500's migrate to the cloud. So some of what I'm saying is not possible or harder on prem and I recognize that. If I had to, I still like Chef, puppet second favorite mostly because of familiarity. Ansiblee can be used with either of these. And tools like serverspec to validate your images. I don't really use any of this anymore though.
skywhopper 7y ago
But you still need to configure things, even if they are immutable at runtime. And you need to manage that configuration over time in some systematic way. You always have a configuration management system.
- mleonhard 7y ago
  I'm using Terraform to deploy Docker containers. Terraform's docker_container resource has a lovely 'upload' feature which one can use to upload files into the container. I make Terraform load server config files (or use multi-line strings in the .tf file), perform variable replacement, then destroy Docker containers and recreate them with updated config files. All persistent data is stored in directories bind-mounted into the Docker container. Terraform has some limitations. For example, one deployment cannot deploy hosts and their containers [1]. And there is no usable support for rolling deployments [2, 3]. So I've ended up with a 4-stage deployment: host-set1, containers on host-set1, host-set2, containers on host-set2. I also use Terraform to deploy the servers to my laptop during development. Docker for Mac works well. Someday, Kubernetes will get some usable documentation on how to do normal things [4]. Then I will use it for deploying containers, load balancers, and persistent volumes. For now, it's too big of a complexity jump over plain Docker. [1] https://github.com/hashicorp/terraform/issues/2430 https://github.com/hashicorp/terraform/issues/2430 [2] https://github.com/hashicorp/terraform/issues/23735#issuecomment-568047226 https://github.com/hashicorp/terraform/issues/23735#issuecom... [3] https://github.com/hashicorp/terraform/issues?q=is%3Aissue+%22Error%3A+Cycle%3A%22+create_before_destroy+ https://github.com/hashicorp/terraform/issues?q=is%3Aissue+%... [4] https://github.com/kubernetes/website/issues/19139 https://github.com/kubernetes/website/issues/19139
inshadows 7y ago
What about the system that runs the containers? "Amazon/Google/Azure takes care of that" is not the answer, unless your comment is predicated on a world where compute can only be rented from big corps... and their methods of managing underlying infrastructure are sacred secrets for which we are to unworthy to comprehend.
z9e 7y ago
For mutable infra that holds state. IMO not all infra is gonna to end up in k8s and some still needs to be self hosted.
mleonhard 7y ago
Terraform keeps track of resources it creates. One can remove resources (VMs, managed databases, persistent volumes, DNS records) from the config file and Terraform will cleanly delete them. This is a crucial feature for most deployments. For example, I deployed an app backend to DigitalOcean with a load balancer, 2x replicated API server, 2x replicated worker process, managed database, file storage, and static website. Terraform is tracking 114 resources for each deployment. It seems that automatic removal is poorly supported by Ansible, Chef, Puppet, Salt, etc. One can explicitly add items to a "to_remove" list, but this is error-prone. Terraform has many limitations and problems, but I have found no better tool.

notyourday 7y ago

> What I prefer to do is use Terraform to create immutable infrastructure from code. Can you mount all your volumes read-only and run all of your stack? If you cannot, then you do not have immutable infrastructure. You simply happen to agree that no one write anything useful, which with time will absolutely fail because someone, somewhere is going to start storing state on a stateless system giving you "a cow #378 called 'Betsy'"

tilolebo 7y ago
In the current state of infrastructure, an accepted definition of "immutable infrastructure" is that: 1. You deploy a completely fresh instance/container, instead of in-place updates 2. You don't actively push changes on a running instance/container Of course you might have stuff written to disk, such as logs, temp files, etc. But it should be non-essential data, and potentially pushed to a central place in near real-time.
- HelloNurse 7y ago
  Interesting. How would you do that if your deployment is, say, a couple of new tables in a 50TB Oracle database?
  - tilolebo 7y ago
    It only works with stateless resources. There's no point in trying to manage a database or similar resources this way.

king_phil 7y ago

But... How do you configure the hosts where your containers are running on? How do you configure your storage (NAS/SAN)? How do you configure your routers and switches? ...

grrywlsn 7y ago
The original question didn't have much context, and I guess my answer assumed someone would be using a cloud provider as opposed to anything on premise. Are Ansible/Puppet/Chef any good for managing the hardware you mentioned?
- notacoward 7y ago
  > Are Ansible/Puppet/Chef any good for managing the hardware you mentioned? Yes. Well, OK, maybe not good but better than ad hoc.
- throw0101a 7y ago
  > Are Ansible/Puppet/Chef any good for managing the hardware you mentioned? Ansible is used in networking, which many vendors having official modules: * https://github.com/PaloAltoNetworks/ansible-pan https://github.com/PaloAltoNetworks/ansible-pan * https://github.com/aristanetworks/ansible-cvp https://github.com/aristanetworks/ansible-cvp * https://www.juniper.net/documentation/en_US/junos-ansible/information-products/pathway-pages/junos-ansible.html https://www.juniper.net/documentation/en_US/junos-ansible/in... * https://github.com/F5Networks/f5-ansible https://github.com/F5Networks/f5-ansible There are even frameworks if you want to write things in 'raw' Python as well.
jcrben 7y ago
Build the underlying vms with packer. Or use cloud-init as the parent mentioned - I think it has a bunch of knobs.

dillonmckay 7y ago

Not all of us have the luxury of our projects being greenfield.

grrywlsn 7y ago
The question asks what I would consider to be the right approach for 2020, and also what my team is doing. This is the design pattern I've been following for 5 years, but obviously your mileage may vary, it won't work for everyone, etc.

whatsmyusername 7y ago

I mostly don't for new stuff (all in on Docker/ECS), however we have a lot of old stuff and things in the process of being migrated where it makes sense. There's also always the odd bird thing you use that needs to run on a regular host.

grrywlsn 7y ago
(Genuinely curious) what old stuff do you think doesn't make sense to be set up immutably? and what odd stuff needs to run on a regular host?
- detaro 7y ago
  Example: How do you do "immutable" management of Mac OS machines? Taking what's typically described as such there, you've just turned a 30s deploy of software into a multi-hour "lets reimage the entire machine"? (although that's of course not strictly "old stuff")
  - grrywlsn 7y ago
    Were Macs in scope of the original post? I assumed it was server side stuff, rather than office hardware. For that, though, I'd use Jamf (Pro) or some other MDM option.
    - detaro 7y ago
      Even if you exclude "office hardware" from configuration management, our Mac OS build and test farm is "servers" I'd say. Not everyone running servers is doing so to run an online service on a platform of their choice.
- jcheng 7y ago
  Not the GP, but some proprietary software requires license activation and you only get a certain (small) number of activates/deactivates.
- deleted 7y ago
  [deleted]

notacoward 7y ago

"Immutable infrastructure" what a laugh. In a large deployment, configuration somewhere is always changing - preferably without restarting tasks because they're constantly loaded. We have (most) configuration under source control, and during the west-coast work day it is practically impossible to commit a change without hitting conflicts and having to rebase. Then there are machines not running production workloads, such as development machines or employees' laptops, which still need to have their configuration managed. Are you going to "immutable infrastructure" everyone's laptops? (Context: my team manages dozens of clusters, each with a score of services across thousands of physical hosts. Every minute of every day, multiple things are being scaled up or down, tuned, rearranged to deal with hardware faults or upgrades, new features rolled out, etc. Far from being immutable, this infrastructure is remarkably fluid because that's the only way to run things at such scale.) Beware of Chesterton's Fence. Just because you haven't learned the reasons for something doesn't mean it's wrong, and the new shiny often re-introduces problems that were already solved (along with some of its own) because of that attitude.

nickthemagicman 7y ago
Sometimes you have to work with what you're given in a brownfield env and a config managment tool is useful in that case, but it's possible that you are working with a less than ideal architecture with less than ideal time/money to make changes. State is always the enemy in technology. I can't even imagine managing hundreds of servers whose state is unpredictable at any moment and they can't be terminated and replaced with a fresh instance for fear of losing something.
- notacoward 7y ago
  > State is always the enemy in technology. I work in data storage. Am I the enemy, then? ;) > can't even imagine managing hundreds of servers whose state is unpredictable at any moment Be careful not to conflate immutability with predictability. The state of these servers is predictable. All of the information necessary to reconstruct them is on a single continuous timeline in source control. But that doesn't mean they're immutable because the head of that timeline is moving very rapidly. > can't be terminated and replaced with a fresh instance for fear of losing something. No, there's (almost) no danger of losing any data because everything's erasure-coded at a level of redundancy that most people find surprising until they learn the reasons (e.g. large-scale electrical outages). But there's definitely a danger of losing availability. You can't just cold-restart a whole service that's running on thousands of hosts and being used continuously by even more thousands without a lot of screaming. Rolling changes are an absolute requirement. Some take minutes. Some take hours. Some take days. Many of these services have run continuously for years, barely resembling the code or config they had when they first started, and their users wouldn't have it any other way. It might be hard to imagine, but it's an every-day reality for my team.

deleted 7y ago

[deleted]

detaro 7y ago

So, Linux-only? ;)

grrywlsn 7y ago
Yes, but to be clear, some of those containers have been .Net Core containers (running in Kubernetes) for me. I appreciate not having Windows in an estate isn't common to all setups.

mneil 7y ago

I'm going to agree with you. In 2020 (and really the last few years), configuration management is outdated. IaC (infrastructure as code) is the current approach. Containerize everything you can, use terraform or cloudformation, or azure devops. Avoid managing the underlying os as much as possible. Use vanilla or prebuilt images to deploy these containers on, coreos, Amazon's new bottle rocket (maybe). Or use a service like fargate when possible. All configuration should be declarative to avoid errors. If you need to build images tools like packer are great. AWS has a recommended "golden Ami pipeline" pattern and a new image builder service if you can't use community images. I'm speaking imperatively but read these as my own directives. I work for a company that consults and actively helps fortune 500's migrate to the cloud. So some of what I'm saying is not possible or harder on prem and I recognize that. If I had to, I still like Chef, puppet second favorite mostly because of familiarity. Ansiblee can be used with either of these. And tools like serverspec to validate your images. I don't really use any of this anymore though.

skywhopper 7y ago

But you still need to configure things, even if they are immutable at runtime. And you need to manage that configuration over time in some systematic way. You always have a configuration management system.

mleonhard 7y ago
I'm using Terraform to deploy Docker containers. Terraform's docker_container resource has a lovely 'upload' feature which one can use to upload files into the container. I make Terraform load server config files (or use multi-line strings in the .tf file), perform variable replacement, then destroy Docker containers and recreate them with updated config files. All persistent data is stored in directories bind-mounted into the Docker container. Terraform has some limitations. For example, one deployment cannot deploy hosts and their containers [1]. And there is no usable support for rolling deployments [2, 3]. So I've ended up with a 4-stage deployment: host-set1, containers on host-set1, host-set2, containers on host-set2. I also use Terraform to deploy the servers to my laptop during development. Docker for Mac works well. Someday, Kubernetes will get some usable documentation on how to do normal things [4]. Then I will use it for deploying containers, load balancers, and persistent volumes. For now, it's too big of a complexity jump over plain Docker. [1] https://github.com/hashicorp/terraform/issues/2430 https://github.com/hashicorp/terraform/issues/2430 [2] https://github.com/hashicorp/terraform/issues/23735#issuecomment-568047226 https://github.com/hashicorp/terraform/issues/23735#issuecom... [3] https://github.com/hashicorp/terraform/issues?q=is%3Aissue+%22Error%3A+Cycle%3A%22+create_before_destroy+ https://github.com/hashicorp/terraform/issues?q=is%3Aissue+%... [4] https://github.com/kubernetes/website/issues/19139 https://github.com/kubernetes/website/issues/19139

inshadows 7y ago

What about the system that runs the containers? "Amazon/Google/Azure takes care of that" is not the answer, unless your comment is predicated on a world where compute can only be rented from big corps... and their methods of managing underlying infrastructure are sacred secrets for which we are to unworthy to comprehend.

z9e 7y ago

For mutable infra that holds state. IMO not all infra is gonna to end up in k8s and some still needs to be self hosted.

mleonhard 7y ago

Terraform keeps track of resources it creates. One can remove resources (VMs, managed databases, persistent volumes, DNS records) from the config file and Terraform will cleanly delete them. This is a crucial feature for most deployments. For example, I deployed an app backend to DigitalOcean with a load balancer, 2x replicated API server, 2x replicated worker process, managed database, file storage, and static website. Terraform is tracking 114 resources for each deployment. It seems that automatic removal is poorly supported by Ansible, Chef, Puppet, Salt, etc. One can explicitly add items to a "to_remove" list, but this is error-prone. Terraform has many limitations and problems, but I have found no better tool.