5 ms·

As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to upda

As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to update everything, Kubernetes has many small controllers which are essentially processes on the cluster that just run a control loop. Each controller corresponds to one resource type, so it will just loop over incoming events that pertain to the resource type in question and attempt to reconcile the state of the target resource instance with the desired state. If it fails initially, it will retry with back off. If something doesn’t stabilize after several minutes, an alert can notify a human. The key differences between the controller approach and the IaC approach are, I think, lots of little processes continuously reconciling state for all resources of a given type (many small loops that touch all resources of a given type on the entire cluster) versus a one-off process that tries to touch just the resources it cares about and if it fails it just gives up. One thing Kubernetes definitely improves upon Terraform is that Kubernetes uses a YAML “assembly language” for its infra as code, but that YAML could be generated by a real programming language. Terraform expects you to write HCL, which is an accidentally re-invented programming language (every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language). Terraform has a CDK that allows writing in other languages, but I’m skeptical that it liberated you from Terraform’s model of the world (e.g., if I rename a variable in CDK, does it try to destroy and recreate the underlying resource as with Terraform?). I’m also concerned that rather than allowing us to generate YAML in the obvious way, it will require bizarre inheritance patterns like the AWS CDK. I would be curious to hear from folks who have used the CDK.

steveb 5y ago
There are a lot of developments around using Kubernetes as an IaC platform for the reasons in your comment. The combination of a standard API model in CRDs + the controller model maps nicely to managing infrastructure and exposing resources to developers. <https://crossplane.io https://crossplane.io> just graduated to CNCF Incubation and each of the cloud providers are working on K8s controllers and code generators (like Amazon Controllers for Kubernetes, Google Config Connector, and the Azure service operator).
the_duke 5y ago
Terraform is also working on allowing actual code: https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
- throwaway894345 5y ago
  I know, I mention that in my comment :)
ClumsyPilot 5y ago
"every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language" I constantly see soo many people step on the same rake, it's incredible. Tools like Tilt let you use python, it's a much more sensible approach.
wernerb 5y ago
The reasoning you put in also means kubernetes is unsuited to be controlled by terraform. Too many lifecycles (resources) to centrally control. kubernetes custom resources can have dependencies on others which terraform either needs to support as well. Which is not doable to maintain.. keep your kubernetes manifests outside or your terraform state.
- throwaway894345 5y ago
  My company manages a lot of Kubernetes manifests with Terraform without issue. Terraform is just generating the manifests in this case; Kubernetes is doing the reconciliation work. More complex than is ideal (i.e., if we were starting out with Kubernetes we probably wouldn’t use Terraform) but it works reasonably well.
verdverm 5y ago
Terraform accepts JSON format as an alternative to HCL. I prefer CUE to JSON for TF and many other tools now
- throwaway894345 5y ago
  To be clear, the issue isn’t the HCL syntax. You could similarly use Cue to generate HCL. The problem is using Terraform’s dynamic features which were poorly designed.
leg100 5y ago
The Hashicorp co-founders considered alternative approaches when originally designing terraform. The actor model was considered but dismissed. That's not a million miles away from the kubernetes reconcile loop model: > Then we transitioned to an actor-based model where each resource was almost an actor, and there was a message-passing interface between them. > This allowed the system to be highly concurrent the way Terraform is today, but also confusing for users to deal with and very difficult to build a programming model around, because the ordering of execution was so random and everything was happening concurrently. https://www.hashicorp.com/resources/terraform-fireside-chat-with-mitchell-and-armon https://www.hashicorp.com/resources/terraform-fireside-chat-... They may still be right. Kubernetes' approach may seem more attractive but terraform is far more pragmatic in its design.

steveb 5y ago

There are a lot of developments around using Kubernetes as an IaC platform for the reasons in your comment. The combination of a standard API model in CRDs + the controller model maps nicely to managing infrastructure and exposing resources to developers. <https://crossplane.io https://crossplane.io> just graduated to CNCF Incubation and each of the cloud providers are working on K8s controllers and code generators (like Amazon Controllers for Kubernetes, Google Config Connector, and the Azure service operator).

the_duke 5y ago

Terraform is also working on allowing actual code: https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk

throwaway894345 5y ago
I know, I mention that in my comment :)

ClumsyPilot 5y ago

"every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language" I constantly see soo many people step on the same rake, it's incredible. Tools like Tilt let you use python, it's a much more sensible approach.

wernerb 5y ago

The reasoning you put in also means kubernetes is unsuited to be controlled by terraform. Too many lifecycles (resources) to centrally control. kubernetes custom resources can have dependencies on others which terraform either needs to support as well. Which is not doable to maintain.. keep your kubernetes manifests outside or your terraform state.

throwaway894345 5y ago
My company manages a lot of Kubernetes manifests with Terraform without issue. Terraform is just generating the manifests in this case; Kubernetes is doing the reconciliation work. More complex than is ideal (i.e., if we were starting out with Kubernetes we probably wouldn’t use Terraform) but it works reasonably well.

verdverm 5y ago

Terraform accepts JSON format as an alternative to HCL. I prefer CUE to JSON for TF and many other tools now

throwaway894345 5y ago
To be clear, the issue isn’t the HCL syntax. You could similarly use Cue to generate HCL. The problem is using Terraform’s dynamic features which were poorly designed.

leg100 5y ago

The Hashicorp co-founders considered alternative approaches when originally designing terraform. The actor model was considered but dismissed. That's not a million miles away from the kubernetes reconcile loop model: > Then we transitioned to an actor-based model where each resource was almost an actor, and there was a message-passing interface between them. > This allowed the system to be highly concurrent the way Terraform is today, but also confusing for users to deal with and very difficult to build a programming model around, because the ordering of execution was so random and everything was happening concurrently. https://www.hashicorp.com/resources/terraform-fireside-chat-with-mitchell-and-armon https://www.hashicorp.com/resources/terraform-fireside-chat-... They may still be right. Kubernetes' approach may seem more attractive but terraform is far more pragmatic in its design.