4 ms·

Surprising that the author is writing a tool for managing your servers, so writes a post about how Terraform isn't great at managing your servers... It seems l

Surprising that the author is writing a tool for managing your servers, so writes a post about how Terraform isn't great at managing your servers... It seems like the root of the problem is that the author wants to use Terraform to manage their AWS state, but also wants to use the web console to directly change things, so Terraform gets out of sync. Terraform has a command to handle this - https://www.terraform.io/docs/cli/commands/refresh.html https://www.terraform.io/docs/cli/commands/refresh.html

orf 5y ago
There’s also a refresh flag on plan. It’s always worth using before your apply CI step.
throwaway894345 5y ago
As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to update everything, Kubernetes has many small controllers which are essentially processes on the cluster that just run a control loop. Each controller corresponds to one resource type, so it will just loop over incoming events that pertain to the resource type in question and attempt to reconcile the state of the target resource instance with the desired state. If it fails initially, it will retry with back off. If something doesn’t stabilize after several minutes, an alert can notify a human. The key differences between the controller approach and the IaC approach are, I think, lots of little processes continuously reconciling state for all resources of a given type (many small loops that touch all resources of a given type on the entire cluster) versus a one-off process that tries to touch just the resources it cares about and if it fails it just gives up. One thing Kubernetes definitely improves upon Terraform is that Kubernetes uses a YAML “assembly language” for its infra as code, but that YAML could be generated by a real programming language. Terraform expects you to write HCL, which is an accidentally re-invented programming language (every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language). Terraform has a CDK that allows writing in other languages, but I’m skeptical that it liberated you from Terraform’s model of the world (e.g., if I rename a variable in CDK, does it try to destroy and recreate the underlying resource as with Terraform?). I’m also concerned that rather than allowing us to generate YAML in the obvious way, it will require bizarre inheritance patterns like the AWS CDK. I would be curious to hear from folks who have used the CDK.
- steveb 5y ago
  There are a lot of developments around using Kubernetes as an IaC platform for the reasons in your comment. The combination of a standard API model in CRDs + the controller model maps nicely to managing infrastructure and exposing resources to developers. <https://crossplane.io https://crossplane.io> just graduated to CNCF Incubation and each of the cloud providers are working on K8s controllers and code generators (like Amazon Controllers for Kubernetes, Google Config Connector, and the Azure service operator).
- the_duke 5y ago
  Terraform is also working on allowing actual code: https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
  - throwaway894345 5y ago
    I know, I mention that in my comment :)
- ClumsyPilot 5y ago
  "every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language" I constantly see soo many people step on the same rake, it's incredible. Tools like Tilt let you use python, it's a much more sensible approach.
- wernerb 5y ago
  The reasoning you put in also means kubernetes is unsuited to be controlled by terraform. Too many lifecycles (resources) to centrally control. kubernetes custom resources can have dependencies on others which terraform either needs to support as well. Which is not doable to maintain.. keep your kubernetes manifests outside or your terraform state.
  - throwaway894345 5y ago

orf 5y ago

There’s also a refresh flag on plan. It’s always worth using before your apply CI step.

throwaway894345 5y ago

As the sibling commenter notes, Terraform has a refresh flag, but I wonder if Kubernetes’ model is better here. Rather than a one-off process that tries to update everything, Kubernetes has many small controllers which are essentially processes on the cluster that just run a control loop. Each controller corresponds to one resource type, so it will just loop over incoming events that pertain to the resource type in question and attempt to reconcile the state of the target resource instance with the desired state. If it fails initially, it will retry with back off. If something doesn’t stabilize after several minutes, an alert can notify a human. The key differences between the controller approach and the IaC approach are, I think, lots of little processes continuously reconciling state for all resources of a given type (many small loops that touch all resources of a given type on the entire cluster) versus a one-off process that tries to touch just the resources it cares about and if it fails it just gives up. One thing Kubernetes definitely improves upon Terraform is that Kubernetes uses a YAML “assembly language” for its infra as code, but that YAML could be generated by a real programming language. Terraform expects you to write HCL, which is an accidentally re-invented programming language (every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language). Terraform has a CDK that allows writing in other languages, but I’m skeptical that it liberated you from Terraform’s model of the world (e.g., if I rename a variable in CDK, does it try to destroy and recreate the underlying resource as with Terraform?). I’m also concerned that rather than allowing us to generate YAML in the obvious way, it will require bizarre inheritance patterns like the AWS CDK. I would be curious to hear from folks who have used the CDK.

steveb 5y ago
There are a lot of developments around using Kubernetes as an IaC platform for the reasons in your comment. The combination of a standard API model in CRDs + the controller model maps nicely to managing infrastructure and exposing resources to developers. <https://crossplane.io https://crossplane.io> just graduated to CNCF Incubation and each of the cloud providers are working on K8s controllers and code generators (like Amazon Controllers for Kubernetes, Google Config Connector, and the Azure service operator).
the_duke 5y ago
Terraform is also working on allowing actual code: https://github.com/hashicorp/terraform-cdk https://github.com/hashicorp/terraform-cdk
- throwaway894345 5y ago
  I know, I mention that in my comment :)
ClumsyPilot 5y ago
"every IaC tool provider thought static configs like YAML would suffice as a human interface, but as they gradually realized the need for more dynamism, Terraform and others would bolt on one dynamic feature after another until they had a slow, unfamiliar, and counterintuitive programming language" I constantly see soo many people step on the same rake, it's incredible. Tools like Tilt let you use python, it's a much more sensible approach.
wernerb 5y ago
The reasoning you put in also means kubernetes is unsuited to be controlled by terraform. Too many lifecycles (resources) to centrally control. kubernetes custom resources can have dependencies on others which terraform either needs to support as well. Which is not doable to maintain.. keep your kubernetes manifests outside or your terraform state.
- throwaway894345 5y ago