6 ms·

I was initially a huge fan but now I hate it. It re-invents so many unnecessary wheels that it is downright obnoxious. There are no loops, if statements, variab

I was initially a huge fan but now I hate it. It re-invents so many unnecessary wheels that it is downright obnoxious. There are no loops, if statements, variables. Things that have been in programming languages since day 1. It re-invents modules, badly. Has weird hacks like null resources and external data providers to get around shortcomings in their "declarative" model. And a few other things I'm forgetting. Oh, now they're adding "environments". That's just dandy because they got everything else so right. Every place I've seen Terraform used invariably runs into these shortcomings and the workarounds range from using Erb and Jinja templates to generate Terraform templates to just ditching it entirely and using home-grown solutions. Terraform should have been a library in an actual programming language instead of a gimped external DSL that re-invents the wheel.

harlowja 10y ago
It also seems pretty much the same as ansible (which suffers the same problems as you mentioned). I pretty much agree with you, they should of provided 2 entrypoints IMHO, they can make there DSL and all that and make it (eventually) turing complete and at the same time they should just provide composeable objects in <some> language as well. I've never quite figured out who the target audience is for these kind of DSL(s), is it people that can't program, or people that can't program well/at all (without adult supervision)?
- dkarapetyan 10y ago
  I don't know either. Puppet, ansible, salt, and friends are in the same boat. The best part is that the special syntax doesn't buy them anything. They just end up being gimped languages for expressing some kind of dependency graph.
  - bostik 10y ago
    All configuration management systems suck. They all suck in equal measure, but each in a different fashion. You can only try to pick the one tool that sucks the least for your particular use-case and hope that two years down the line you made the right choice.
    - dkarapetyan 10y ago
      This isn't much of an argument. That's like saying all programming languages suck and it's a crapshoot. It's not. There are design principles that work and scale to large scale systems and then there are things that don't. As far as I'm concerned Chef is the right way to do configuration management. Everyone else with their YAML DSLs is doing it wrong. Expressing dependencies between resources does not require specialized external DSLs and sometimes you need to do imperative things to get to the end which is either impossible or just horribly convoluted with any tool that tries to hide complexity behind a markup/serialization language.
- hubert123 10y ago
  Apparently these people think it makes it "simpler". Well I'm sorry but me looking up how to format things with yaml and no autoformat and no intellisense just isnt good and doesnt make things simple. Even xml with a backing xsd would be better than that, but that's apparently not cool enough anymore.
  - dkarapetyan 10y ago
    I forgot about the lack of auto-complete and the horrible error messages. At least with a library when I have a syntax error or something blows up I have a debugger and the actual language at my disposal to figure stuff out. No such things are remotely available with all these external DSLs. If even there is any semblance of error reporting. Several human centuries of effort have been spent on improving these things (syntax highlighting, auto-complete, stack traces, error messages, interactive debuggers, etc.) and then all these DSL authors decide to just chuck it all out. Boggles the mind. More than a few times Terraform has wrapped an error message from AWS in the hopes of being helpful while hiding all the details of the actual error message and no hints how to rectify things in the DSL.
    - apatap 10y ago
      Syntax highlighting and auto complete is available for HCL (The language used by Terraform) e.g. https://plugins.jetbrains.com/plugin/7808-hcl-language-support https://plugins.jetbrains.com/plugin/7808-hcl-language-suppo... https://github.com/wholroyd/vscode-hcl https://github.com/wholroyd/vscode-hcl
      - bmurphy1976 10y ago
        Which is great and all but like the op said our files are actually jinja2 templates because HCL just doesn't cut it.
    - hubert123 10y ago
      I had the same experience with ansible. It also wraps the responses in some ultra verbose json. I really just want to see the remote commands being executed, if you return them wrapped in some kind of format then you need to provide a gui for it. Otherwise it's all worse than before.
jolynch 10y ago
We've really enjoyed templating terraform. It solves a lot of the problems you're mentioning, and we can write the complex logic we need without interacting _directly_ with cloud APIs... Compilers are like, useful.
skMed 10y ago
Some basic conditional logic was added in 0.8 [0]. The reliance on 'interesting' declarative tricks, e.g. counts for conditionals and loops, is a little annoying I agree. However, it looks like they're listening to the community and will build on these features in the future. There's a lot to work on :) 0: https://www.hashicorp.com/blog/terraform-0-8/#conditional https://www.hashicorp.com/blog/terraform-0-8/#conditional
zenlikethat 10y ago
I don't really agree that Terraform re-invents wheels. In fact it feels like a lot of the things you cite as issues are actually due to Terraform being pretty innovative. There's not much like it for "version-controlled" infrastructure, so the team and community are learning as they go, and can't be expected to come up with perfect solutions right away. I agree the looping hacks are very ugly, really dislike the whole split and join everywhere thing. As for DSL vs. programming languages, the halting problem provides a pretty compelling argument in favor of limiting your system if you want to have "verifiable convergence". Personally I'm a huge fan of HCL too. It fixes a lot of what's bad about YAML and TOML and has a built in pretty printer.
nunez 10y ago
I am actually okay with not having loops or advanced constructs. Those things made using Ansible for more complex provisioning scenarios (a) unreadable and (b) horribly kludgey. m The things that Terraform solves for, namely keeping environment state and infrastructure relationships, are really hard to do with other CM tools without diving deep i.e. writing python or ruby
- dkarapetyan 10y ago
  It's not the lack of "advanced" constructs as you say. It is also the total disregard for all the human centuries of effort that have gone into making programming languages and environments convenient vehicles for expressing computation. Things like stack traces, syntax highlighting editors, auto-complete, interactive debuggers, and a bunch of other stuff you get with an actual programming language that you don't get with an external DSL. There is a reason those things are not part of the package. It is because making those things is extremely hard. Whereas slapping together a parser is not but feels like actual work. You might consider those things advanced. I consider it the bare minimum to get proper work done.
  - nunez 10y ago
    Here's the thing: HCL isn't a programming language nor will it ever be. I hope. this problem isn't unique to Terraform; Chef and Puppet are in the aame boat. It's markup. It is meant to be easily readable by devs and non-devs alike while being reasonably capable. The beauty of these tools is that they are written and be expanded upon with popular languages. If I wanted to make Terraform deploy Dell servers, all I have to do is write a plug-in with Go. What's more is that I don't have to teach a TF user anything new since its usage pattern will be the same as any other TF plug-in.
    - dkarapetyan 10y ago
      Nope. Chef is an embedded DSL. It does not have the same shortcomings. In fact it is the only configuration management tool that approaches the problem from the right angle. Instead of re-inventing the wheel it gives me a library of composable components that I can glue together however I wish. You must not have seen many TF deployments. I've seen several and each one is a unique snowflake. Some use Jinja, some use Makesfiles, some use Erb, some just use heroic copy-pasting. There are no conventions or standards whatsoever. I don't know what you mean by readable either. When was the last time you navigated 20 Terraform modules while mentally filling in all the variables and thought to yourself "Ya, this is totally readable".
jdoss 10y ago
But this isn't a programming language. it's a DSL that lets you get the job done. I get that you want to use flow control statements to make your life easier, but I feel that part of the reason why Terraform was made and has gained a lot of popularity. For me, it doesn't make things overly complicated. Except for how they handle variables with modules.That stuff sucks.
arianvanp 10y ago
I would highly recommend Nixops. It allows you to describe your infrastructure in the Nix expression language. Which is a proper dict-based programming language. It supports many popular clouds. I'm not sure if it is Nixos only, but Nixos itself is awesome to use with it. You just put your provisioning configuration inside your "network" configuration and boom, all your servers and hard drives are created and fully provisioned. https://Nixos.org/nixops https://Nixos.org/nixops
- scrollaway 10y ago
  A cursory look at it tells me the nix expression language isn't that much more powerful than HCL. Inheritance is nice but that's about it.
  - arianvanp 10y ago
    it is a fully-fledged programming lazy purely-functional programming language. Which allows you to abstract a lot of boilerplate in functions and whatnot. https://nixcloud.io/tour/?id=1 https://nixcloud.io/tour/?id=1
skovorodkin 10y ago
You should try out http://jsonnet.org http://jsonnet.org, I love it.
ludwigvan 10y ago
Hear, hear! Devops right now is in the Middle Ages, kind of like what web frontend programming was a few years ago. Remember all those weird cobbled down logicless DSLs that were supposed to save us from ourselves. I am expecting the Renaissance you mentioned in devops shortly, just like how React transformed the scene. In short, DSLs should be inside programming languages and we shouldn't be afraid to write code.
scrollaway 10y ago
HCL is atrocious. Terraform's main issue is that it tries very hard to be declarative, then it has to invent a template language as a superset of the declarative JSON it uses to make up for it. And that superset is invariably shit. It's like you said; no loops, no proper variables etc. Terraform itself though is fantastic and powerful. It's lacking some pretty important stuff still (for example, not having to refresh the entire infrastructure state when you change one single variable for one single tiny little service), but that just makes it an alpha. It's still better than the alternatives by a mile and then some. But, yeah. Fuck HCL. The good news though is that I think this is "easily" fixable by someone who would really work on it: Because the underlying environment is 100% json, it's easy enough to generate; so you could implement a saner environment for specifying the state and feed that to terraform. That's enough for me to bet my infrastructure on it: knowing that today's issues are fixable, rather than the effort being doomed from the start.
- jen20 10y ago
  `terraform plan -refresh=false`
  - scrollaway 10y ago
    Granual refreshing is not the same thing as no refreshing.
    - jen20 10y ago
      `terraform refresh -target=resource_path`
      - scrollaway 10y ago
        I know it's possible to do manually. But you have to figure it out yourself then. It also doesn't properly dirty the other affected resources.
purelazy 10y ago
Take a look at jsonnet: http://jsonnet.org http://jsonnet.org. It extends JSON with loops, conditionals, variables and more in an elegant way. We don't write JSON anymore and treat it like an "assembly language." We use jsonnet to generate CloudFormation templates, terraform templates and Kubernetes resource definitions.
threatofrain 10y ago
Ahh just like Ansible with it's YAML'ish templating language. If you want to add variables and scoping, loops, a module resolution system, etc., then I'd rather use a real programming language.

harlowja 10y ago

It also seems pretty much the same as ansible (which suffers the same problems as you mentioned). I pretty much agree with you, they should of provided 2 entrypoints IMHO, they can make there DSL and all that and make it (eventually) turing complete and at the same time they should just provide composeable objects in <some> language as well. I've never quite figured out who the target audience is for these kind of DSL(s), is it people that can't program, or people that can't program well/at all (without adult supervision)?

dkarapetyan 10y ago
I don't know either. Puppet, ansible, salt, and friends are in the same boat. The best part is that the special syntax doesn't buy them anything. They just end up being gimped languages for expressing some kind of dependency graph.
- bostik 10y ago
  All configuration management systems suck. They all suck in equal measure, but each in a different fashion. You can only try to pick the one tool that sucks the least for your particular use-case and hope that two years down the line you made the right choice.
  - dkarapetyan 10y ago
    This isn't much of an argument. That's like saying all programming languages suck and it's a crapshoot. It's not. There are design principles that work and scale to large scale systems and then there are things that don't. As far as I'm concerned Chef is the right way to do configuration management. Everyone else with their YAML DSLs is doing it wrong. Expressing dependencies between resources does not require specialized external DSLs and sometimes you need to do imperative things to get to the end which is either impossible or just horribly convoluted with any tool that tries to hide complexity behind a markup/serialization language.
hubert123 10y ago
Apparently these people think it makes it "simpler". Well I'm sorry but me looking up how to format things with yaml and no autoformat and no intellisense just isnt good and doesnt make things simple. Even xml with a backing xsd would be better than that, but that's apparently not cool enough anymore.
- dkarapetyan 10y ago
  I forgot about the lack of auto-complete and the horrible error messages. At least with a library when I have a syntax error or something blows up I have a debugger and the actual language at my disposal to figure stuff out. No such things are remotely available with all these external DSLs. If even there is any semblance of error reporting. Several human centuries of effort have been spent on improving these things (syntax highlighting, auto-complete, stack traces, error messages, interactive debuggers, etc.) and then all these DSL authors decide to just chuck it all out. Boggles the mind. More than a few times Terraform has wrapped an error message from AWS in the hopes of being helpful while hiding all the details of the actual error message and no hints how to rectify things in the DSL.
  - apatap 10y ago
    Syntax highlighting and auto complete is available for HCL (The language used by Terraform) e.g. https://plugins.jetbrains.com/plugin/7808-hcl-language-support https://plugins.jetbrains.com/plugin/7808-hcl-language-suppo... https://github.com/wholroyd/vscode-hcl https://github.com/wholroyd/vscode-hcl
    - bmurphy1976 10y ago
      Which is great and all but like the op said our files are actually jinja2 templates because HCL just doesn't cut it.
  - hubert123 10y ago
    I had the same experience with ansible. It also wraps the responses in some ultra verbose json. I really just want to see the remote commands being executed, if you return them wrapped in some kind of format then you need to provide a gui for it. Otherwise it's all worse than before.

jolynch 10y ago

We've really enjoyed templating terraform. It solves a lot of the problems you're mentioning, and we can write the complex logic we need without interacting _directly_ with cloud APIs... Compilers are like, useful.

skMed 10y ago

Some basic conditional logic was added in 0.8 [0]. The reliance on 'interesting' declarative tricks, e.g. counts for conditionals and loops, is a little annoying I agree. However, it looks like they're listening to the community and will build on these features in the future. There's a lot to work on :) 0: https://www.hashicorp.com/blog/terraform-0-8/#conditional https://www.hashicorp.com/blog/terraform-0-8/#conditional

zenlikethat 10y ago

I don't really agree that Terraform re-invents wheels. In fact it feels like a lot of the things you cite as issues are actually due to Terraform being pretty innovative. There's not much like it for "version-controlled" infrastructure, so the team and community are learning as they go, and can't be expected to come up with perfect solutions right away. I agree the looping hacks are very ugly, really dislike the whole split and join everywhere thing. As for DSL vs. programming languages, the halting problem provides a pretty compelling argument in favor of limiting your system if you want to have "verifiable convergence". Personally I'm a huge fan of HCL too. It fixes a lot of what's bad about YAML and TOML and has a built in pretty printer.

nunez 10y ago

I am actually okay with not having loops or advanced constructs. Those things made using Ansible for more complex provisioning scenarios (a) unreadable and (b) horribly kludgey. m The things that Terraform solves for, namely keeping environment state and infrastructure relationships, are really hard to do with other CM tools without diving deep i.e. writing python or ruby

dkarapetyan 10y ago
It's not the lack of "advanced" constructs as you say. It is also the total disregard for all the human centuries of effort that have gone into making programming languages and environments convenient vehicles for expressing computation. Things like stack traces, syntax highlighting editors, auto-complete, interactive debuggers, and a bunch of other stuff you get with an actual programming language that you don't get with an external DSL. There is a reason those things are not part of the package. It is because making those things is extremely hard. Whereas slapping together a parser is not but feels like actual work. You might consider those things advanced. I consider it the bare minimum to get proper work done.
- nunez 10y ago
  Here's the thing: HCL isn't a programming language nor will it ever be. I hope. this problem isn't unique to Terraform; Chef and Puppet are in the aame boat. It's markup. It is meant to be easily readable by devs and non-devs alike while being reasonably capable. The beauty of these tools is that they are written and be expanded upon with popular languages. If I wanted to make Terraform deploy Dell servers, all I have to do is write a plug-in with Go. What's more is that I don't have to teach a TF user anything new since its usage pattern will be the same as any other TF plug-in.
  - dkarapetyan 10y ago
    Nope. Chef is an embedded DSL. It does not have the same shortcomings. In fact it is the only configuration management tool that approaches the problem from the right angle. Instead of re-inventing the wheel it gives me a library of composable components that I can glue together however I wish. You must not have seen many TF deployments. I've seen several and each one is a unique snowflake. Some use Jinja, some use Makesfiles, some use Erb, some just use heroic copy-pasting. There are no conventions or standards whatsoever. I don't know what you mean by readable either. When was the last time you navigated 20 Terraform modules while mentally filling in all the variables and thought to yourself "Ya, this is totally readable".

jdoss 10y ago

But this isn't a programming language. it's a DSL that lets you get the job done. I get that you want to use flow control statements to make your life easier, but I feel that part of the reason why Terraform was made and has gained a lot of popularity. For me, it doesn't make things overly complicated. Except for how they handle variables with modules.That stuff sucks.

arianvanp 10y ago

I would highly recommend Nixops. It allows you to describe your infrastructure in the Nix expression language. Which is a proper dict-based programming language. It supports many popular clouds. I'm not sure if it is Nixos only, but Nixos itself is awesome to use with it. You just put your provisioning configuration inside your "network" configuration and boom, all your servers and hard drives are created and fully provisioned. https://Nixos.org/nixops https://Nixos.org/nixops

scrollaway 10y ago
A cursory look at it tells me the nix expression language isn't that much more powerful than HCL. Inheritance is nice but that's about it.
- arianvanp 10y ago
  it is a fully-fledged programming lazy purely-functional programming language. Which allows you to abstract a lot of boilerplate in functions and whatnot. https://nixcloud.io/tour/?id=1 https://nixcloud.io/tour/?id=1

skovorodkin 10y ago

You should try out http://jsonnet.org http://jsonnet.org, I love it.

ludwigvan 10y ago

Hear, hear! Devops right now is in the Middle Ages, kind of like what web frontend programming was a few years ago. Remember all those weird cobbled down logicless DSLs that were supposed to save us from ourselves. I am expecting the Renaissance you mentioned in devops shortly, just like how React transformed the scene. In short, DSLs should be inside programming languages and we shouldn't be afraid to write code.

scrollaway 10y ago

HCL is atrocious. Terraform's main issue is that it tries very hard to be declarative, then it has to invent a template language as a superset of the declarative JSON it uses to make up for it. And that superset is invariably shit. It's like you said; no loops, no proper variables etc. Terraform itself though is fantastic and powerful. It's lacking some pretty important stuff still (for example, not having to refresh the entire infrastructure state when you change one single variable for one single tiny little service), but that just makes it an alpha. It's still better than the alternatives by a mile and then some. But, yeah. Fuck HCL. The good news though is that I think this is "easily" fixable by someone who would really work on it: Because the underlying environment is 100% json, it's easy enough to generate; so you could implement a saner environment for specifying the state and feed that to terraform. That's enough for me to bet my infrastructure on it: knowing that today's issues are fixable, rather than the effort being doomed from the start.

jen20 10y ago
`terraform plan -refresh=false`
- scrollaway 10y ago
  Granual refreshing is not the same thing as no refreshing.
  - jen20 10y ago
    `terraform refresh -target=resource_path`
    - scrollaway 10y ago
      I know it's possible to do manually. But you have to figure it out yourself then. It also doesn't properly dirty the other affected resources.

purelazy 10y ago

Take a look at jsonnet: http://jsonnet.org http://jsonnet.org. It extends JSON with loops, conditionals, variables and more in an elegant way. We don't write JSON anymore and treat it like an "assembly language." We use jsonnet to generate CloudFormation templates, terraform templates and Kubernetes resource definitions.

threatofrain 10y ago

Ahh just like Ansible with it's YAML'ish templating language. If you want to add variables and scoping, loops, a module resolution system, etc., then I'd rather use a real programming language.