8 ms·

Deploying Kubernetes with Ansible

55 points by kdubb 4y ago

kdubb 4y ago
As your cluster grows, so does the complexity of managing it. That's where Ansible comes in.
- sleepybrett 4y ago
  I've never worked anywhere sizeable that uses ansible to manage kubernetes. My one brush with it was at a small shop and it caused more problems that it solved. It was just a couple of guys that happened to use ansible for everything else, right or wrong, it seemed like a lot of "i've got a hammer..." In the pre-managed kubernetes world I did see an ansible stack for standing up a cluster that seemed ok, but once you get past setting up a base cluster ready to take 'customer' workloads ansible is just not a great tool.
- darken 4y ago
  I completely agree that for managing the host nodes/machines, Ansible is a huge win. Though this article focuses on deploying ON the cluster. Ideally the complexity of deploying ON a cluster (i.e. 'kubectl apply') doesn't increase with its size. In some ways it actually gets simpler, as you can worry less about per-node scheduling due to resource constraints/packing.
water554 4y ago
My conclusion is that using Ansible with Kubernetes is the WRONG way to go about it. So now instead of a directory of yaml or helm charts we have a bunch of ansible uselessly wrapping the yaml for application.
- Spivak 4y ago
  The problem is when you're in a situation where "over here we need YAML document A, and over there we need YAML document B." You either shove a templating system in front of your YAML and fight with that or throw in the towel just use a programming language where constructs in your YAML map to objects.
  - hdjjhhvvhga 4y ago
    Helm3 made it a solved problem years ago.
  - unxdfa 4y ago
    I think we need to burn this whole thing down and rethink it at this point. A huge amount of the total operational cost of our business is working around weird YAML problems and helm charts. Had one a while back where quoted strings containing padded numbers in were used for something. "01" fine ... "05" fine, "06" fine, "07" fine, "08" kaboom. 01-07 were treated as strings and 08 was suddenly being treated as octal, which it's not and exploded.
    - jpgvm 4y ago
      Helm is a pile of garbage. k8s speaks JSON just fine as expected. Try Tanka + Jsonnet to restore some of your sanity, also allows you to keep using Helm during your transition to a better place.
      - unxdfa 4y ago
        This is just the same problem in JSON, another loose schema’ed mess. Got my fair share of problems with that at the moment as well.
        jpgvm 4y ago
        If you feel really strongly about getting type safety then there is Pulumi + Typescript though I really hate how it tries to be a "better" kubectl apply and fails miserably. Definitely room for someone to make something better than Helm but I think you should give Jsonnet a shot if you haven't already.
        jen20 4y ago
        > tries to be a "better" kubectl apply and fails How does it try? How does it fail? I don't see any interactive CLI commands that could be seen as a replacement for kubectl.
Spivak 4y ago
I think the lesson people need to start taking is that any form of declarative form of configuration will spawn an ad-hoc ecosystem to turn it imperative. The complexity of situations that can be expressed is by a configuration system is proportional to the complexity of that system. Simple and declarative is an oxymoron for anything but simple cases. Imperative systems can express arbitrary complexity and you will need it. It's why troposphere, terraform-cdk, pulumi, a billion templatating systems over k8s yaml files, and Ansible exist.
- TomBombadildoze 4y ago
  The lesson is that shockingly few people have realized is that the problem space is not, nor has it ever been, served by declarative solutions. You can describe the state you want a system to be in but there will inevitably be procedures involved to do some of the work. It is unavoidable. At the point people began developing templating around their configuration, ah, templates... which themselves have imperative language constructs awkwardly (and badly, cough Golang) embedded in them, it should have been clear to everyone with a shred of talent that it wasn't a good solution. Declarative configuration is wholly expressible by imperative languages. The converse is not true. Declarative configuration is appropriate for declaring state (i.e., values), and not for describing procedures. Stop using half-baked YAML crap and start using real tools.
  - aero142 4y ago
    Do you have any examples of deploying to Kubernetes with something you consider real tools? I agree with what you are saying but have found very few real world companies using real languages for this.
    - iampims 4y ago
      At work we use typescript to generate yaml files which are then synced to their respective clusters via Gitops. It works fairly well. All cluster/namespace differences are encoded into our typescript code base, so we can take advantage of a real programming language to generate cluster specific yaml all sharing a common “base”.
denysvitali 4y ago
I thought this was about deploying a Kubernetes cluster with Ansible, but nope, it's really about deploying YAML files to Kubernetes, with Ansible. Just...Why?
- TomBombadildoze 4y ago
  When all you have is a hammer, everything looks like a nail.
- MuffinFlavored 4y ago
  Sounds like the title should be "Deploying To Kubernetes With Ansible", since this isn't about deploying a Kubernetes cluster?
- ikiris 4y ago
  ... oh god why. Also thank you for the very apt and red flagging summary kind sir.
- nijave 4y ago
  Yeah, title made me think kube-spray
- sprior 4y ago
  I had the same impression.
- KronisLV 4y ago
  > Just...Why? I can imagine some cases where you might want to setup infrastructure-related software on your cluster and to do so through Ansible (using manifests or Helm charts under the hood), so that you have one template that you can use for setting up clusters later, for various use cases. For example, you might want a custom ingress controller, or to deploy a service mesh like Istio, or maybe Kiali as well. Perhaps you want to launch Rancher on your cluster to give your engineers an arguably easier way to interact with the cluster, or to explore it in a more visual way than just using something like the kubectl CLI (though tools like k9s are great, too: https://k9scli.io/ https://k9scli.io/). Edit: Many suggest that using something like Argo CD would be a better tool for this, due to its specialization, though. I actually utilized a similar approach when working with Ansible + Docker Swarm (since I needed a more lightweight orchestrator that played nicely with Docker, but to manage the nodes with Ansible): I deployed my own "ingress" (just a web server container with some configuration), some supporting software like Sonatype Nexus for storing custom images, Portainer for graphic cluster management and some other software for similar concerns. Once that was done and other server configuration (e.g. user accounts, groups, permissions, folders, services etc.; though most less relevant for the containers themselves) were set up, other CI/CD pipelines could deploy apps across the cluster without Ansible being involved - but it was good for setting up the base for it all. But if you wanted to use Ansible for deploying almost all of your apps, like in the article? I guess you could do that if you have a lot of buy-in into using Ansible and just prefer the way you do things with it. Admittedly, using Helm or one of the alternatives in this case might be a better choice.
methods21 4y ago
Not really the best technical solution for this challenge. And this is from a big Ansible + Packer person that deploys 10Ks of VMs and infrastructure with Ansible.
- deleted 4y ago
  [deleted]
- MuffinFlavored 4y ago
  10,000+ VMs? What do you work on?
darken 4y ago
This went a different direction than I'd expected. That is, configuring your cluster resources rather than the host machines themselves. I managed to grab 5 or so Raspberry Pi 4s before the availability issues. I've been (slowly) getting a home Kubernetes cluster running over the last year or two, and Ansible helps immensely; I: * Write Ubuntu 20.04 (or now 22.04) to MicroSD card. * Configure the initial user + SSH-PK, hostname, and network via config added to the SD * Boot up the new Pi * Run Ansible playbook to get it ready for k3s (lightweight but "full" k8s); including bind-mounting all high use paths for containerd/logs/etc to a USB NVME-SSD, adding boot args to the Pi's GRUB, etc. * Push all my bashrc/tmux/vim/etc configs for consistent environment if I ssh in. * Use Ansible ad-hoc for misc changes or package updates. In terms of the "deploying to Kubernetes", I've recently got FluxCD working and love it. I think "GitOps" is the way to go in terms of clear reproducible config. I have GitLab running on a local Synology (NAS) which kicks off any container builds and hosts the Git and container registry. The FluxCD operator on the cluster polls my "cluster as code" repo and syncs state. Each project defines a Helm chart and Flux pulls the chart from each repo and applies any variables (as configured in the "cluster repo") or "kustomizations". Flux let's you setup "dependency" relationships for components. So I have it add the sealed-secrets operator early in, and put PK-encrypted secrets in directly the Git repo for it to"unpack" in the cluster. That being said, the author's Anisble-driven approach is a much quicker way get something on a cluster ASAP, versus setting up the initial Flux "pull from cluster" system. I'm primarily a software engineer, so this was also largely an exercise in getting more comfortable with the operations side of things. Just thought I'd share my experience since only so many people let me rant about my FluxCD/k3s/GitLab home dev environment.
- sleepybrett 4y ago
  `kubectl apply -f myshit/` is a faster way of getting things on your cluster, ansible is just a bunch of wrapper on top of that.
  - TomBombadildoze 4y ago
    Until you need to update it, or apply it to multiple environments, or any number of tasks where the concepts of configuration parameters and dependencies become important. Edit: I'm not saying Ansible is an appropriate substitute (it absolutely isn't), rather that `kubectl -f` is not a scalable solution.
c7DJTLrn 4y ago
Wat? If you really need to, you can use Ansible to bootstrap a cluster but I wouldn't use it for anything more than that - that's the responsibility of CD. Ideally you would just have Terraform apply a manifest that installs Argo or Flux and that's it.
cgeier 4y ago
I would suggest to look at something like Argo CD, Flux or Gitlab CI/CD's Kubernetes support instead.
guender 4y ago
[dead]
nforgerit 4y ago
Gosh, I wish the author wouldn't sound so arrogant presenting me his reinvented wheel. What your Ansible is doing (looping over the inventory, imperatively executing tasks) is reinventing what a Kubernetes-internal Operator is doing after you asked it to reconcile your changes. Thing is: You're not only reinventing the internal thing, you will eventually recreate internal logic, e.g. some internal dependencies and knowledge "what to do in which order". Good luck with further developments on the Kubernetes upstream. Not cool. Kubernetes does what it can, to give the user a declarative DX. You might not want to fight this. kustomize and some _trivial_ scripting is usually all you'll need. If you need more, consider writing custom Operators.
- MuffinFlavored 4y ago
  > kustomize and some _trivial_ scripting is usually all you'll need. So not just k8s alone, k8s + kustomize, no Helm charts, right? or can you do k8s + Helm, no kustomize? or can you do kustomize to deploy Helm charts which abstracts away and translates to underlying k8s API calls/generated YAML?
  - nijave 4y ago
    Yes
  - nforgerit 4y ago
    Hm take it with a grain of salt, but I'm not really a fan of Helm. It might make sense in some bigger corps where you have something like "Platform" teams with dozens of consuming teams to consolidate and version some kind of service. But I lack trust and frequently same-mindedness with Helm chart authors on how to deploy and configure certain things such that I'll use my own thingy anyway. I forgot to mention the CD tools we have, e.g. Flux or ArgoCD. Use customize to render a "final" yaml for your env/version/whatever and throw it at Flux/Argo (e.g. using GitOps) and be happy. That's imho the Kubernetes way of life.
    - schaum 4y ago
      couldn't agree more, but we are "stuck" in managed ansible + k8s although i think kustomize wouldn't make a nicer picture than ansible, but push vs pull .. i prefer pull
singingfish 4y ago
The art of ansible is knowing when not to use it. I'm guessing that ansible to K8s is an example of when not to do so, but I'm no expert.
emersonrsantos 4y ago
Does the author ever heard about using loops in Ansible? with_dict can save you around 20-30 lines of code.
Felminor 4y ago
I started with Ansible and k8s and.its.fine. Not really hard, you get it up and running fast enough (who ever wants to setup a k8s cluster like this has the week to do so). But honestly: don't! I switched to rke2. Alternativly try out gardener from sap or anything else between Ansible and managed. You do not want to take care of bare metal k8s. And rke2 is still not perfect but it's rancher support will be better and better and more than just you are using it. But be real: if you don't need air gap, go to a cloud provider. Ah and btw open nebula also started supporting k8s by deploying rke2!
- kryptn 4y ago
  I don't think you read the article. It's about deploying resources onto k8s with ansible, not deploying or maintaining k8s.
  - Felminor 4y ago
    Huh? Than that's even worse. Use argocd or flux
Felminor 4y ago
Use argocd! It's the best tool for the job. Can and should be configured as IaC. Argocd can manage itself!
- frizlab 4y ago
  Personally I prefer Flux, but YMMV.
frizlab 4y ago
Flux is a much better gitops option.
hbogert 4y ago
Ansible is the regular expression of devops, you can use it almost anywhere to try and solve a problem, but now you have 2 problems. Ansible makes you comfortable with mutable infra, but with enough time I always regret using it.

55 points by kdubb 4y ago

kdubb 4y ago

As your cluster grows, so does the complexity of managing it. That's where Ansible comes in.

sleepybrett 4y ago
I've never worked anywhere sizeable that uses ansible to manage kubernetes. My one brush with it was at a small shop and it caused more problems that it solved. It was just a couple of guys that happened to use ansible for everything else, right or wrong, it seemed like a lot of "i've got a hammer..." In the pre-managed kubernetes world I did see an ansible stack for standing up a cluster that seemed ok, but once you get past setting up a base cluster ready to take 'customer' workloads ansible is just not a great tool.
darken 4y ago
I completely agree that for managing the host nodes/machines, Ansible is a huge win. Though this article focuses on deploying ON the cluster. Ideally the complexity of deploying ON a cluster (i.e. 'kubectl apply') doesn't increase with its size. In some ways it actually gets simpler, as you can worry less about per-node scheduling due to resource constraints/packing.

water554 4y ago

My conclusion is that using Ansible with Kubernetes is the WRONG way to go about it. So now instead of a directory of yaml or helm charts we have a bunch of ansible uselessly wrapping the yaml for application.

Spivak 4y ago
The problem is when you're in a situation where "over here we need YAML document A, and over there we need YAML document B." You either shove a templating system in front of your YAML and fight with that or throw in the towel just use a programming language where constructs in your YAML map to objects.
- hdjjhhvvhga 4y ago
  Helm3 made it a solved problem years ago.
- unxdfa 4y ago
  I think we need to burn this whole thing down and rethink it at this point. A huge amount of the total operational cost of our business is working around weird YAML problems and helm charts. Had one a while back where quoted strings containing padded numbers in were used for something. "01" fine ... "05" fine, "06" fine, "07" fine, "08" kaboom. 01-07 were treated as strings and 08 was suddenly being treated as octal, which it's not and exploded.
  - jpgvm 4y ago
    Helm is a pile of garbage. k8s speaks JSON just fine as expected. Try Tanka + Jsonnet to restore some of your sanity, also allows you to keep using Helm during your transition to a better place.
    - unxdfa 4y ago
      This is just the same problem in JSON, another loose schema’ed mess. Got my fair share of problems with that at the moment as well.
      - jpgvm 4y ago
        If you feel really strongly about getting type safety then there is Pulumi + Typescript though I really hate how it tries to be a "better" kubectl apply and fails miserably. Definitely room for someone to make something better than Helm but I think you should give Jsonnet a shot if you haven't already.
        jen20 4y ago
        > tries to be a "better" kubectl apply and fails How does it try? How does it fail? I don't see any interactive CLI commands that could be seen as a replacement for kubectl.

Spivak 4y ago

I think the lesson people need to start taking is that any form of declarative form of configuration will spawn an ad-hoc ecosystem to turn it imperative. The complexity of situations that can be expressed is by a configuration system is proportional to the complexity of that system. Simple and declarative is an oxymoron for anything but simple cases. Imperative systems can express arbitrary complexity and you will need it. It's why troposphere, terraform-cdk, pulumi, a billion templatating systems over k8s yaml files, and Ansible exist.

TomBombadildoze 4y ago
The lesson is that shockingly few people have realized is that the problem space is not, nor has it ever been, served by declarative solutions. You can describe the state you want a system to be in but there will inevitably be procedures involved to do some of the work. It is unavoidable. At the point people began developing templating around their configuration, ah, templates... which themselves have imperative language constructs awkwardly (and badly, cough Golang) embedded in them, it should have been clear to everyone with a shred of talent that it wasn't a good solution. Declarative configuration is wholly expressible by imperative languages. The converse is not true. Declarative configuration is appropriate for declaring state (i.e., values), and not for describing procedures. Stop using half-baked YAML crap and start using real tools.
- aero142 4y ago
  Do you have any examples of deploying to Kubernetes with something you consider real tools? I agree with what you are saying but have found very few real world companies using real languages for this.
  - iampims 4y ago
    At work we use typescript to generate yaml files which are then synced to their respective clusters via Gitops. It works fairly well. All cluster/namespace differences are encoded into our typescript code base, so we can take advantage of a real programming language to generate cluster specific yaml all sharing a common “base”.

denysvitali 4y ago

I thought this was about deploying a Kubernetes cluster with Ansible, but nope, it's really about deploying YAML files to Kubernetes, with Ansible. Just...Why?

TomBombadildoze 4y ago
When all you have is a hammer, everything looks like a nail.
MuffinFlavored 4y ago
Sounds like the title should be "Deploying To Kubernetes With Ansible", since this isn't about deploying a Kubernetes cluster?
ikiris 4y ago
... oh god why. Also thank you for the very apt and red flagging summary kind sir.
nijave 4y ago
Yeah, title made me think kube-spray
sprior 4y ago
I had the same impression.
KronisLV 4y ago
> Just...Why? I can imagine some cases where you might want to setup infrastructure-related software on your cluster and to do so through Ansible (using manifests or Helm charts under the hood), so that you have one template that you can use for setting up clusters later, for various use cases. For example, you might want a custom ingress controller, or to deploy a service mesh like Istio, or maybe Kiali as well. Perhaps you want to launch Rancher on your cluster to give your engineers an arguably easier way to interact with the cluster, or to explore it in a more visual way than just using something like the kubectl CLI (though tools like k9s are great, too: https://k9scli.io/ https://k9scli.io/). Edit: Many suggest that using something like Argo CD would be a better tool for this, due to its specialization, though. I actually utilized a similar approach when working with Ansible + Docker Swarm (since I needed a more lightweight orchestrator that played nicely with Docker, but to manage the nodes with Ansible): I deployed my own "ingress" (just a web server container with some configuration), some supporting software like Sonatype Nexus for storing custom images, Portainer for graphic cluster management and some other software for similar concerns. Once that was done and other server configuration (e.g. user accounts, groups, permissions, folders, services etc.; though most less relevant for the containers themselves) were set up, other CI/CD pipelines could deploy apps across the cluster without Ansible being involved - but it was good for setting up the base for it all. But if you wanted to use Ansible for deploying almost all of your apps, like in the article? I guess you could do that if you have a lot of buy-in into using Ansible and just prefer the way you do things with it. Admittedly, using Helm or one of the alternatives in this case might be a better choice.

methods21 4y ago

Not really the best technical solution for this challenge. And this is from a big Ansible + Packer person that deploys 10Ks of VMs and infrastructure with Ansible.

deleted 4y ago
[deleted]
MuffinFlavored 4y ago
10,000+ VMs? What do you work on?

darken 4y ago

This went a different direction than I'd expected. That is, configuring your cluster resources rather than the host machines themselves. I managed to grab 5 or so Raspberry Pi 4s before the availability issues. I've been (slowly) getting a home Kubernetes cluster running over the last year or two, and Ansible helps immensely; I: * Write Ubuntu 20.04 (or now 22.04) to MicroSD card. * Configure the initial user + SSH-PK, hostname, and network via config added to the SD * Boot up the new Pi * Run Ansible playbook to get it ready for k3s (lightweight but "full" k8s); including bind-mounting all high use paths for containerd/logs/etc to a USB NVME-SSD, adding boot args to the Pi's GRUB, etc. * Push all my bashrc/tmux/vim/etc configs for consistent environment if I ssh in. * Use Ansible ad-hoc for misc changes or package updates. In terms of the "deploying to Kubernetes", I've recently got FluxCD working and love it. I think "GitOps" is the way to go in terms of clear reproducible config. I have GitLab running on a local Synology (NAS) which kicks off any container builds and hosts the Git and container registry. The FluxCD operator on the cluster polls my "cluster as code" repo and syncs state. Each project defines a Helm chart and Flux pulls the chart from each repo and applies any variables (as configured in the "cluster repo") or "kustomizations". Flux let's you setup "dependency" relationships for components. So I have it add the sealed-secrets operator early in, and put PK-encrypted secrets in directly the Git repo for it to"unpack" in the cluster. That being said, the author's Anisble-driven approach is a much quicker way get something on a cluster ASAP, versus setting up the initial Flux "pull from cluster" system. I'm primarily a software engineer, so this was also largely an exercise in getting more comfortable with the operations side of things. Just thought I'd share my experience since only so many people let me rant about my FluxCD/k3s/GitLab home dev environment.

sleepybrett 4y ago
`kubectl apply -f myshit/` is a faster way of getting things on your cluster, ansible is just a bunch of wrapper on top of that.
- TomBombadildoze 4y ago
  Until you need to update it, or apply it to multiple environments, or any number of tasks where the concepts of configuration parameters and dependencies become important. Edit: I'm not saying Ansible is an appropriate substitute (it absolutely isn't), rather that `kubectl -f` is not a scalable solution.

c7DJTLrn 4y ago

Wat? If you really need to, you can use Ansible to bootstrap a cluster but I wouldn't use it for anything more than that - that's the responsibility of CD. Ideally you would just have Terraform apply a manifest that installs Argo or Flux and that's it.

cgeier 4y ago

I would suggest to look at something like Argo CD, Flux or Gitlab CI/CD's Kubernetes support instead.

guender 4y ago

[dead]

nforgerit 4y ago

Gosh, I wish the author wouldn't sound so arrogant presenting me his reinvented wheel. What your Ansible is doing (looping over the inventory, imperatively executing tasks) is reinventing what a Kubernetes-internal Operator is doing after you asked it to reconcile your changes. Thing is: You're not only reinventing the internal thing, you will eventually recreate internal logic, e.g. some internal dependencies and knowledge "what to do in which order". Good luck with further developments on the Kubernetes upstream. Not cool. Kubernetes does what it can, to give the user a declarative DX. You might not want to fight this. kustomize and some _trivial_ scripting is usually all you'll need. If you need more, consider writing custom Operators.

MuffinFlavored 4y ago
> kustomize and some _trivial_ scripting is usually all you'll need. So not just k8s alone, k8s + kustomize, no Helm charts, right? or can you do k8s + Helm, no kustomize? or can you do kustomize to deploy Helm charts which abstracts away and translates to underlying k8s API calls/generated YAML?
- nijave 4y ago
  Yes
- nforgerit 4y ago
  Hm take it with a grain of salt, but I'm not really a fan of Helm. It might make sense in some bigger corps where you have something like "Platform" teams with dozens of consuming teams to consolidate and version some kind of service. But I lack trust and frequently same-mindedness with Helm chart authors on how to deploy and configure certain things such that I'll use my own thingy anyway. I forgot to mention the CD tools we have, e.g. Flux or ArgoCD. Use customize to render a "final" yaml for your env/version/whatever and throw it at Flux/Argo (e.g. using GitOps) and be happy. That's imho the Kubernetes way of life.
  - schaum 4y ago
    couldn't agree more, but we are "stuck" in managed ansible + k8s although i think kustomize wouldn't make a nicer picture than ansible, but push vs pull .. i prefer pull

singingfish 4y ago

The art of ansible is knowing when not to use it. I'm guessing that ansible to K8s is an example of when not to do so, but I'm no expert.

emersonrsantos 4y ago

Does the author ever heard about using loops in Ansible? with_dict can save you around 20-30 lines of code.

Felminor 4y ago

I started with Ansible and k8s and.its.fine. Not really hard, you get it up and running fast enough (who ever wants to setup a k8s cluster like this has the week to do so). But honestly: don't! I switched to rke2. Alternativly try out gardener from sap or anything else between Ansible and managed. You do not want to take care of bare metal k8s. And rke2 is still not perfect but it's rancher support will be better and better and more than just you are using it. But be real: if you don't need air gap, go to a cloud provider. Ah and btw open nebula also started supporting k8s by deploying rke2!

kryptn 4y ago
I don't think you read the article. It's about deploying resources onto k8s with ansible, not deploying or maintaining k8s.
- Felminor 4y ago
  Huh? Than that's even worse. Use argocd or flux

Felminor 4y ago

Use argocd! It's the best tool for the job. Can and should be configured as IaC. Argocd can manage itself!

frizlab 4y ago
Personally I prefer Flux, but YMMV.

frizlab 4y ago

Flux is a much better gitops option.

hbogert 4y ago

Ansible is the regular expression of devops, you can use it almost anywhere to try and solve a problem, but now you have 2 problems. Ansible makes you comfortable with mutable infra, but with enough time I always regret using it.