6 ms·

Terraform Collaboration for Everyone

103 points by mwarkentin 8y ago

mwarkentin 8y ago
If you’re familiar with Atlantis and think that this sounds similar - you’re right! They’ve also hired the Atlantis developer: https://medium.com/runatlantis/joining-hashicorp-200ee9572dc5 https://medium.com/runatlantis/joining-hashicorp-200ee9572dc...
- lars_francke 8y ago
  I hadn't heard of Atlantis before, thank you. Very interesting. Do you happen to know of a self-service tool (CLI and/or Web UI) that allows us to do the following: Run these Ansible playbooks, then apply these Terraform modules, then run these Ansible... etc. We often have to build different environments for all kinds of use-cases (usually for testing, debugging or demonstration purposes). Two examples: We need a three node Kafka cluster on the Hetzner Cloud (Terraform + Ansible), we need a five node Elasticsearch cluster on AWS etc. For that we'd like to specify which Terraform modules to run with which variables, we then read the Terraform state file to generate an Ansible inventory and run a set of Ansible plays/roles.
  - dkhenry 8y ago
    I usually accomplish that task indirectly by running Ansible first as a provisioner of Packer to create the images I need. Then I run terraform to deploy everything I have just created.
  - orf 8y ago
    Airflow or Luigi could be what you are looking for.
  - Bombthecat 8y ago
    Maybe use rundeck as a gui? Still a lot of custom code necessary though.
  - fishnchips 8y ago
    As a hobby project, I'm building a tool somewhat similar to Atlantis/TFE. It's free though not yet open source - https://docs.geopoiesis.io/manual https://docs.geopoiesis.io/manual. Not sure about your exact use case, but it's distributed as a Docker image and apart from Terraform it allows you to run arbitrary scripts (for example, `tflint`). We're using the project internally at Deliveroo, so it's production ready, but I haven't had the time to do all the proper open-source work around the landing page, and the documentation is still a bit meh. Still, give me a shout (email in my profile) if it ticks your box, and I'm happy to help you get started.
    - lars_francke 8y ago
      Thanks! I've put it on my list of things to look at. It doesn't exactly fit our use-case but it still might be interesting for us. Also thank you very much for the offer of help.
  - sciurus 8y ago
    We have a custom python tool that does something similar, except for terraform and kubectl. This is the sort of thing where you probably have to write it from scratch to match your workflow, but it should end up ~500 LOC in your favorite scripting language. Then we wrap that tool in jenkin pipelines so we can easily trigger it in response to events like code being pushed.
    - lars_francke 8y ago
      That's what we ended up doing, yes. It's simple but works for us. Every time we want to add a new feature I think that there must be something like this out there already...
  - kokey 8y ago
    I've been using Makefiles for that, and a plenty of parameters like REGION=us-north-2 NODES=5 etc. This is run from a dedicated shell server.
- manojlds 8y ago
  It's there in the article.
IloveHN84 8y ago
Finally
SteveNuts 8y ago
This is great news, the state file was always the worst part about TF.
- DerpyBaby123 8y ago
  In modest, small team use, a remote state file on s3 served me well. Did you run into problems with it?
  - SteveNuts 8y ago
    As a general rule I try to avoid putting sensitive information onto services that could accidentally be turned public facing.
    - mike-cardwell 8y ago
      You can (should) use client side encryption on the state terraform state bucket.
      - kitotik 8y ago
        Exactly. I’m a little surprised to hear that this one of the biggest feature requests from the community. Once you are in hashicorp-land it’s not a big jump to spin up a consul+vault cluster and store your encrypted state remotely.
        jen20 8y ago
        There is still a chicken-and-egg problem assuming you want to use Terraform to spin up the Consul+Vault clusters, though.
        marenkay 8y ago
        Why worry about that? Has anyone ever solved the chicken-and-egg problem in it? Can it even be solved? Seriously, is there any other approach from declaring a new egg as new baseline chicken?
outworlder 8y ago
It would be great to have an intermediate step between "you are on your own" and "ENTERPRISE – contact us!". Some 'business' plan, perhaps? I know there's 'pro', but it still has the 'request info' button instead of pricing. I am hoping this is what they want to do with the SaaS offering.
- tylersmith 8y ago
  That's what it sounds like from the description: > Going forward collaboration features will be available for free to practitioners and small teams, at an affordable price to businesses, and Terraform Enterprise will remain our world-class platform for organizations adopting Terraform at scale. It's not released yet though, so it won't be reflected in the pricing page yet. The beta starts "early next year".
ris 8y ago
Looking at the two diagrams shown lower down in the article, I can't be the only one that thinks the "Before" looks infinitely superior to the "After". Yet another third party service to depend on which has unknown reliability or future business model plans.
- ronjouch 8y ago
  Yes! To me, that sounds similar (or one step further) to dev/ops folks exclaiming "Docker!" at the mildest trouble with builds / CI / isolation. After spending time with Docker, I like it when it's deserved, but my first reaction is one of "Uh, it's one more layer, it will bring new tooling and required knowledge and error conditions, let's only do it if we have to".
- reptation 8y ago
  Yeah, this appears to be accomplishing the same purpose as the 'Terragrunt' wrapper which uses DynamoDB for locking state IIRC https://github.com/gruntwork-io/terragrunt https://github.com/gruntwork-io/terragrunt
  - jen20 8y ago
    If using the S3 remote state backend, Terraform itself has supported this with no wrapper for quite some time now.
    - kitotik 8y ago
      Doesn’t the consul backend support this as well?
      - jen20 8y ago
        Several do - I'm not sure which offhand though. The docs for each backend (e.g. [1]) explain what is supported. [1]: https://www.terraform.io/docs/backends/types/azurerm.html https://www.terraform.io/docs/backends/types/azurerm.html
- 8y ago
carlsborg 8y ago
What are the arguments for TF being the better option for a new infrastructure build today? Given that a) Cloudformation with Custom Resources and Macros is extensible, AWS supported, and free and b) its hard to leverage many of the AWS services while still maintaining provider-neutrality (e.g Serverless Framework 2.0 on AWS runs on top of cloudformation templates now)
- choochootrain 8y ago
  i last used cloudformation over a year ago and it was a pretty shitty experience overall. bread and butter resources like vpcs would fail to provision in time which would cause a cascading failure and result in the entire stack failing and rolling back. also cleaning up stacks that failed to roll back was a bit of a pain when there were intermingled dependencies - the dashboard was about as unhelpful as possible in determining what order to delete things so i could carry on doing what i was actually trying to do. i chalk that up to aws resources being backed by an eventually consistent db and the fact that aws has never built dashboards that are better than "acceptable". it's possible that i'm missing something here but i've found terraform to be orders of magnitude better on both fronts. it takes a much more defensive stance on resource provisioning with retry logic and picking up on a previous `terraform apply` which timed out instead of rolling back the entire thing. and for whatever reason, terraform is usually way faster than cloudformation for provisioning the same resources.
- chucky_z 8y ago
  I used TF very early on, for versions 0.3 and 0.4. It was unpleasant at best, but very functional. I've began using it again with 0.11, and with 0.12 on the horizon it's much easier to use than it ever has been in the past. It used to be very difficult to do one-off tasks with Terraform but I've began managing things piecemeal and the community around importing everything into existing TF things are great. The statefile borking itself from time to time is still an issue, but no more than any other operational tool of this magnitude. The biggest downside to Terraform currently (0.11) is that you need to get extremely creative in ugly ways to accomplish some tasks. This can been seen very specifically in the AWS Security Group official TF module, which I could not live without, but would not want to write/maintain.
magd 8y ago
Well, I'm now concerned that they'll kill of Atlantis.
- marenkay 8y ago
  Why would they if they can just provide a Terraform for Github application for a few bucks per month per team with it?
deleted 8y ago
[deleted]
paulgrant999 8y ago
Terraform is pretty good; but its strength is its weakness; the declarative syntax (DAG planner) isn't perfect and once you start hinting it, it causes all sort of chaos. The solution to complex deploys that re-use portions of the deployments you would want to put a context boundary on, is a series of modules that maintain their own state. But this forces you to pass large amounts of the same data (couples the modules strongly to the main file) over and over again. As for collaborative, terragrunt works pretty good ;) provided you can give a home to the statefile. The real problem I have with terraform/aws (or insert provider) is that its still time-intensive to actually do the deploys. The parallelization of the terraform graph, is "simple" at best. If anyone wants to collaborate on a fork of packer/terraform, let me know on thread.
- marenkay 8y ago
  Wouldn't the fact that passing resource to modules works in 0.12 help with the mass exodus of output variables for modules? What is time-intensive in terms of deploys? It has been my experience with tens of providers that the time loss occurs when the requests have been handed off to a providers API but not so much in Terraform. As for Terraform itself, my biggest issue is that it has turned around the view of resources and providers. It should have been going in the direction of https://github.com/google/go-cloud https://github.com/google/go-cloud where you talk in terms of resources, and then a provider loaded would just provide the implementation. Now I have to build Terraform modules to export resource and hide away provider specifics in the module.
  - paulgrant999 8y ago
    > Wouldn't the fact that passing resource to modules works in 0.12 help with the mass exodus of output variables for modules? might. I skipped it, use soft-links, and hard outputs (variable files) to get around it. cuts down on the graph, I can set explicit boundaries (on whats getting eval'd) without having to go through the terraform DSL. equivalent to running separate terraform scripts, on separate portions of the infrastructure. data source providers (understandably) are not static; there is no way to cache the output of data providers between runs; which means any query retriggers dependencies eval which rebuilds perfectly fine infrastructure. hence why I just output static files. Now there is no chance for terraform to think this variable might change, and ergo, no chance for rebuild of infrastructure that doesn't need it. > It has been my experience with tens of providers that the time loss occurs when the requests have been handed off to a providers API but not so much in Terraform. Agree. So this is sort of the point. I don't need to actually bring up machines in a typical DAG fashion - so some of the work is parallelizable. But other parts aren't. If I need 20 machines, I know I need 20 machines. If the AMI's are loaded on the cloud provider, there is absolutely no reason to have them loaded sequentially (such as a "post-config" might require) i.e. incurring large wait times. Once you start using "terraform" variables (off of resources), that is what you get. So I prefer a different variable resolution mechanism (i.e. lazy/evented). Particularly now that you can ghetto-leverage the cloud providers tag systems, in order to do "proto" service discovery (tag machines for eventual service discovery/config i.e. similar to ansible roles). I'm twisting the hell out of terraform (for things it is not intended to do). but that is because I want to avoid adding yet another tool to the toolchain. More declarative DSL's = more issues to diagnose. > As for Terraform itself, my biggest issue is that it has turned around the view of resources and providers. not so much a problem for me (not multi-cloud). I like knowing (at a glance) what provider-specific features I can expect to be supported. The problem with trying to abstract out cloud provider, is you go with the lowest common denominator (in terms of declarative syntax). > It should have been going in the direction of https://github.com/google/go-cloud https://github.com/google/go-cloud Never used it. Also I've never written a terraform module. So its entirely plausible, this is where my issues would be resolved (custom provider) and an "enhanced" dsl.
marenkay 8y ago
As a long-term user (2014/2015), I shall say this is nice but... it is kind of late. I always wondered why Terraform Enterprise / Atlas was in its simple incarnation a tool to be used in CI/CD tooling in form of a containerized app with a selection of storage drivers. That would be a use case applying to hundreds - if not thousands - of companies, and even big players on the market. Those might want the UI with policies too. Who wants to give a tool / backend full access to their infrastructure and not be the one to control all aspects of that tool?

103 points by mwarkentin 8y ago

If you’re familiar with Atlantis and think that this sounds similar - you’re right! They’ve also hired the Atlantis developer: https://medium.com/runatlantis/joining-hashicorp-200ee9572dc5 https://medium.com/runatlantis/joining-hashicorp-200ee9572dc...

lars_francke 8y ago
I hadn't heard of Atlantis before, thank you. Very interesting. Do you happen to know of a self-service tool (CLI and/or Web UI) that allows us to do the following: Run these Ansible playbooks, then apply these Terraform modules, then run these Ansible... etc. We often have to build different environments for all kinds of use-cases (usually for testing, debugging or demonstration purposes). Two examples: We need a three node Kafka cluster on the Hetzner Cloud (Terraform + Ansible), we need a five node Elasticsearch cluster on AWS etc. For that we'd like to specify which Terraform modules to run with which variables, we then read the Terraform state file to generate an Ansible inventory and run a set of Ansible plays/roles.
- dkhenry 8y ago
  I usually accomplish that task indirectly by running Ansible first as a provisioner of Packer to create the images I need. Then I run terraform to deploy everything I have just created.
- orf 8y ago
  Airflow or Luigi could be what you are looking for.
- Bombthecat 8y ago
  Maybe use rundeck as a gui? Still a lot of custom code necessary though.
- fishnchips 8y ago
  As a hobby project, I'm building a tool somewhat similar to Atlantis/TFE. It's free though not yet open source - https://docs.geopoiesis.io/manual https://docs.geopoiesis.io/manual. Not sure about your exact use case, but it's distributed as a Docker image and apart from Terraform it allows you to run arbitrary scripts (for example, `tflint`). We're using the project internally at Deliveroo, so it's production ready, but I haven't had the time to do all the proper open-source work around the landing page, and the documentation is still a bit meh. Still, give me a shout (email in my profile) if it ticks your box, and I'm happy to help you get started.
  - lars_francke 8y ago
    Thanks! I've put it on my list of things to look at. It doesn't exactly fit our use-case but it still might be interesting for us. Also thank you very much for the offer of help.
- sciurus 8y ago
  We have a custom python tool that does something similar, except for terraform and kubectl. This is the sort of thing where you probably have to write it from scratch to match your workflow, but it should end up ~500 LOC in your favorite scripting language. Then we wrap that tool in jenkin pipelines so we can easily trigger it in response to events like code being pushed.
  - lars_francke 8y ago
    That's what we ended up doing, yes. It's simple but works for us. Every time we want to add a new feature I think that there must be something like this out there already...
- kokey 8y ago
  I've been using Makefiles for that, and a plenty of parameters like REGION=us-north-2 NODES=5 etc. This is run from a dedicated shell server.
manojlds 8y ago
It's there in the article.

IloveHN84 8y ago

Finally

SteveNuts 8y ago

This is great news, the state file was always the worst part about TF.

DerpyBaby123 8y ago
In modest, small team use, a remote state file on s3 served me well. Did you run into problems with it?
- SteveNuts 8y ago
  As a general rule I try to avoid putting sensitive information onto services that could accidentally be turned public facing.
  - mike-cardwell 8y ago
    You can (should) use client side encryption on the state terraform state bucket.
    - kitotik 8y ago
      Exactly. I’m a little surprised to hear that this one of the biggest feature requests from the community. Once you are in hashicorp-land it’s not a big jump to spin up a consul+vault cluster and store your encrypted state remotely.
      - jen20 8y ago
        There is still a chicken-and-egg problem assuming you want to use Terraform to spin up the Consul+Vault clusters, though.
        marenkay 8y ago
        Why worry about that? Has anyone ever solved the chicken-and-egg problem in it? Can it even be solved? Seriously, is there any other approach from declaring a new egg as new baseline chicken?

outworlder 8y ago

It would be great to have an intermediate step between "you are on your own" and "ENTERPRISE – contact us!". Some 'business' plan, perhaps? I know there's 'pro', but it still has the 'request info' button instead of pricing. I am hoping this is what they want to do with the SaaS offering.

tylersmith 8y ago
That's what it sounds like from the description: > Going forward collaboration features will be available for free to practitioners and small teams, at an affordable price to businesses, and Terraform Enterprise will remain our world-class platform for organizations adopting Terraform at scale. It's not released yet though, so it won't be reflected in the pricing page yet. The beta starts "early next year".

ris 8y ago

Looking at the two diagrams shown lower down in the article, I can't be the only one that thinks the "Before" looks infinitely superior to the "After". Yet another third party service to depend on which has unknown reliability or future business model plans.

ronjouch 8y ago
Yes! To me, that sounds similar (or one step further) to dev/ops folks exclaiming "Docker!" at the mildest trouble with builds / CI / isolation. After spending time with Docker, I like it when it's deserved, but my first reaction is one of "Uh, it's one more layer, it will bring new tooling and required knowledge and error conditions, let's only do it if we have to".
reptation 8y ago
Yeah, this appears to be accomplishing the same purpose as the 'Terragrunt' wrapper which uses DynamoDB for locking state IIRC https://github.com/gruntwork-io/terragrunt https://github.com/gruntwork-io/terragrunt
- jen20 8y ago
  If using the S3 remote state backend, Terraform itself has supported this with no wrapper for quite some time now.
  - kitotik 8y ago
    Doesn’t the consul backend support this as well?
    - jen20 8y ago
      Several do - I'm not sure which offhand though. The docs for each backend (e.g. [1]) explain what is supported. [1]: https://www.terraform.io/docs/backends/types/azurerm.html https://www.terraform.io/docs/backends/types/azurerm.html
8y ago

carlsborg 8y ago

What are the arguments for TF being the better option for a new infrastructure build today? Given that a) Cloudformation with Custom Resources and Macros is extensible, AWS supported, and free and b) its hard to leverage many of the AWS services while still maintaining provider-neutrality (e.g Serverless Framework 2.0 on AWS runs on top of cloudformation templates now)

choochootrain 8y ago
i last used cloudformation over a year ago and it was a pretty shitty experience overall. bread and butter resources like vpcs would fail to provision in time which would cause a cascading failure and result in the entire stack failing and rolling back. also cleaning up stacks that failed to roll back was a bit of a pain when there were intermingled dependencies - the dashboard was about as unhelpful as possible in determining what order to delete things so i could carry on doing what i was actually trying to do. i chalk that up to aws resources being backed by an eventually consistent db and the fact that aws has never built dashboards that are better than "acceptable". it's possible that i'm missing something here but i've found terraform to be orders of magnitude better on both fronts. it takes a much more defensive stance on resource provisioning with retry logic and picking up on a previous `terraform apply` which timed out instead of rolling back the entire thing. and for whatever reason, terraform is usually way faster than cloudformation for provisioning the same resources.
chucky_z 8y ago
I used TF very early on, for versions 0.3 and 0.4. It was unpleasant at best, but very functional. I've began using it again with 0.11, and with 0.12 on the horizon it's much easier to use than it ever has been in the past. It used to be very difficult to do one-off tasks with Terraform but I've began managing things piecemeal and the community around importing everything into existing TF things are great. The statefile borking itself from time to time is still an issue, but no more than any other operational tool of this magnitude. The biggest downside to Terraform currently (0.11) is that you need to get extremely creative in ugly ways to accomplish some tasks. This can been seen very specifically in the AWS Security Group official TF module, which I could not live without, but would not want to write/maintain.

magd 8y ago

Well, I'm now concerned that they'll kill of Atlantis.

marenkay 8y ago
Why would they if they can just provide a Terraform for Github application for a few bucks per month per team with it?

deleted 8y ago

[deleted]

paulgrant999 8y ago

Terraform is pretty good; but its strength is its weakness; the declarative syntax (DAG planner) isn't perfect and once you start hinting it, it causes all sort of chaos. The solution to complex deploys that re-use portions of the deployments you would want to put a context boundary on, is a series of modules that maintain their own state. But this forces you to pass large amounts of the same data (couples the modules strongly to the main file) over and over again. As for collaborative, terragrunt works pretty good ;) provided you can give a home to the statefile. The real problem I have with terraform/aws (or insert provider) is that its still time-intensive to actually do the deploys. The parallelization of the terraform graph, is "simple" at best. If anyone wants to collaborate on a fork of packer/terraform, let me know on thread.

marenkay 8y ago
Wouldn't the fact that passing resource to modules works in 0.12 help with the mass exodus of output variables for modules? What is time-intensive in terms of deploys? It has been my experience with tens of providers that the time loss occurs when the requests have been handed off to a providers API but not so much in Terraform. As for Terraform itself, my biggest issue is that it has turned around the view of resources and providers. It should have been going in the direction of https://github.com/google/go-cloud https://github.com/google/go-cloud where you talk in terms of resources, and then a provider loaded would just provide the implementation. Now I have to build Terraform modules to export resource and hide away provider specifics in the module.
- paulgrant999 8y ago
  > Wouldn't the fact that passing resource to modules works in 0.12 help with the mass exodus of output variables for modules? might. I skipped it, use soft-links, and hard outputs (variable files) to get around it. cuts down on the graph, I can set explicit boundaries (on whats getting eval'd) without having to go through the terraform DSL. equivalent to running separate terraform scripts, on separate portions of the infrastructure. data source providers (understandably) are not static; there is no way to cache the output of data providers between runs; which means any query retriggers dependencies eval which rebuilds perfectly fine infrastructure. hence why I just output static files. Now there is no chance for terraform to think this variable might change, and ergo, no chance for rebuild of infrastructure that doesn't need it. > It has been my experience with tens of providers that the time loss occurs when the requests have been handed off to a providers API but not so much in Terraform. Agree. So this is sort of the point. I don't need to actually bring up machines in a typical DAG fashion - so some of the work is parallelizable. But other parts aren't. If I need 20 machines, I know I need 20 machines. If the AMI's are loaded on the cloud provider, there is absolutely no reason to have them loaded sequentially (such as a "post-config" might require) i.e. incurring large wait times. Once you start using "terraform" variables (off of resources), that is what you get. So I prefer a different variable resolution mechanism (i.e. lazy/evented). Particularly now that you can ghetto-leverage the cloud providers tag systems, in order to do "proto" service discovery (tag machines for eventual service discovery/config i.e. similar to ansible roles). I'm twisting the hell out of terraform (for things it is not intended to do). but that is because I want to avoid adding yet another tool to the toolchain. More declarative DSL's = more issues to diagnose. > As for Terraform itself, my biggest issue is that it has turned around the view of resources and providers. not so much a problem for me (not multi-cloud). I like knowing (at a glance) what provider-specific features I can expect to be supported. The problem with trying to abstract out cloud provider, is you go with the lowest common denominator (in terms of declarative syntax). > It should have been going in the direction of https://github.com/google/go-cloud https://github.com/google/go-cloud Never used it. Also I've never written a terraform module. So its entirely plausible, this is where my issues would be resolved (custom provider) and an "enhanced" dsl.

marenkay 8y ago

As a long-term user (2014/2015), I shall say this is nice but... it is kind of late. I always wondered why Terraform Enterprise / Atlas was in its simple incarnation a tool to be used in CI/CD tooling in form of a containerized app with a selection of storage drivers. That would be a use case applying to hundreds - if not thousands - of companies, and even big players on the market. Those might want the UI with policies too. Who wants to give a tool / backend full access to their infrastructure and not be the one to control all aspects of that tool?