5 ms·

An alternative opinion: When I worked at a large tech company, my team made a conscious decision to not use terraform. This gave us some key advantages- we are

An alternative opinion: When I worked at a large tech company, my team made a conscious decision to not use terraform. This gave us some key advantages- we are able to adopt new cloud features immediately, months before they were available in tf, and our direct cloud access let us build features that would surprise the teams using tf within the company. If your core competency isn't dependent on your cloud platform tf is a great tool. But using cloud APIs directly was great for us.

CSDude 5y ago
What new cloud features was not available in Terraform for months?
- pvtmert 5y ago
  eg. AWS ChatBot not available in TF yet. TBH AWS haven't even added it to their Go SDK. So, I cannot blame TF. But anyway that's one of the inheret problems of TF plugin system. Compare to kubectl. Where you can write plugins in bash/shell and mark with execute bit, put it in somewhere in your $PATH as kubectl-blabla and use it as "kubectl blabla".
  - CSDude 5y ago
    It's not fair to compare imperative simple shell scripts with the things Terraform does. It has schema validation, state comparison, retries, failure handlers etc. Also, just as you can write extensions to kubectl, you can write your own provider in Terraform if it does not exists. See https://registry.terraform.io/modules/waveaccounting/chatbot-slack-configuration/aws/latest https://registry.terraform.io/modules/waveaccounting/chatbot... Also, Chatbot does not have a public API, that's why, it's only configured via Cloudformation. So the expectation is not fair either. I've seen Cloudformation getting features years later. i.e 2021 - https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dynamodb-global-tables-now-support-aws-cloudformation/ https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dy... 2015 - https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dynamodb-available-now-cross-region-replication-triggers-and-streams/ https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dy...
    - jen20 5y ago
      NAT Gateways is another notable feature that took CloudFormation months yet Terraform had on day 1. If you can configure something via CloudFormation you can integrate it via Terraform et al also, since they have resources representing CloudFormation stacks.
      - lincler 5y ago
        This! Is not like you can't go beyond what Terraform offers by default. Running CloudFormation stacks from Terraform is a neat way of solving missing apis/integration. And that's exactly what my team did when Terraform was missing a lot of lambdas functionalities. We just declared the CloudFormation Stack for lambdas and then call it from Terraform.
  - jen20 5y ago
    There’s no reason you can’t do something similar with Terraform either - plugins speak GRPC and thus could be implemented in Python, with Node.js or with Rust. However, if AWS have not published metadata for a given service to be used across their various SDKs, it’s hard to take that service particularly seriously, so I’m not sure I’d bother with this.
- dharmab 5y ago
  Private preview features for partnered organizations. We had access up to 6 months before the public.
iddqd 5y ago
It usually takes 1-2 years for AWS to roll out their latest updates to the regions I use and by then Terraform is stable.
jrsdav 5y ago
> But using cloud APIs directly was great for us This is fine, I've done it extensively myself for some of the bleeding-edge cloud stuff, but the importance of things like tracking state, managing hierarchical resource dependencies, or retry/back-off logic shouldn't be tossed aside simply because there are gaps in what's available in the Terraform providers. Especially where change management is important (basically any enterprise company). I'd caution others reading this against abandoning something altogether and writing bespoke IaC tooling simply because the stable approach doesn't cover every (bleeding) edge case. You'll spend a lot of time reinventing the wheel, and while it's fine for certain situations (like when you only care about desired state, not known state, for instance), you'll move faster (and likely safer) by sticking with tools like Terraform for the bulk of your infra, and augmenting here there with cloud APIs/SDKs when needed.
- dharmab 5y ago
  Yes, we did have to implement our own state tracking, retries/recovery, etc- but since we were focused on a limited subset of the cloud API, this was pretty easy.
  - Thaxll 5y ago
    So you re-implemented terraform but worse most likely. Also you could have added those missing features and re-use the TF engine, it's very simple to include new API of an existing provider.
    - dharmab 5y ago
      No, we used an entirely different architecture/paradigm not possible with tf, and had capabilities tf doesn't attempt to provide (such as coordinating migrations with both cloud and application APIs, or managing capacity while upgrading 10000s of CPU cores worth of compute).
goodpoint 5y ago
Same here. It's incredible how much efforts developers are willing to put on popular "devops" tools when the job could be done faster with 200 lines of Python.
- dharmab 5y ago
  This comment Gets It.
oneplane 5y ago
Would the time spent re-implementing a specialised Terraform subset be better spent simply maintaining a private branch of the AWS Provider? You can add your secret/special API without having to do all the other heavy lifting as well. This makes your own effort for customisation minimal, keeps your knowledge portable and because your added features can be separated in to different files and the provider API is stable you can also easily backport/fast-forward new changes.
- dharmab 5y ago
  Our custom code was fewer LOC, more robust and faster than tf. Honestly, it's not hard to beat OSS if you have a team of great engineers. This was far from the only time our internal projects were better than OSS- it gave us advantages for removing bottlenecks in the critical paths.
  - oneplane 5y ago
    So when engineers left the company, was their knowledge portable? And attracting new talent, did they come with X years knowledge for the custom code? With fewer LOC I'm not sure what you mean, the provider code is pretty small, smaller than custom ansible, salt or puppet modules. Smaller than CDK and Pulumni as well. Sure, you'll have to write Go, but that's about the only hurdle. Everyone doing a round of NIH for internal tooling is ultimately not making the tide rise. Edit: don't get me wrong here, writing internal tools to do a job the right way for the right needs isn't "invalid" or something like that, but people often dismiss the rest of the lifecycle of knowledge and maintenance when making something completely custom.

CSDude 5y ago

What new cloud features was not available in Terraform for months?

pvtmert 5y ago
eg. AWS ChatBot not available in TF yet. TBH AWS haven't even added it to their Go SDK. So, I cannot blame TF. But anyway that's one of the inheret problems of TF plugin system. Compare to kubectl. Where you can write plugins in bash/shell and mark with execute bit, put it in somewhere in your $PATH as kubectl-blabla and use it as "kubectl blabla".
- CSDude 5y ago
  It's not fair to compare imperative simple shell scripts with the things Terraform does. It has schema validation, state comparison, retries, failure handlers etc. Also, just as you can write extensions to kubectl, you can write your own provider in Terraform if it does not exists. See https://registry.terraform.io/modules/waveaccounting/chatbot-slack-configuration/aws/latest https://registry.terraform.io/modules/waveaccounting/chatbot... Also, Chatbot does not have a public API, that's why, it's only configured via Cloudformation. So the expectation is not fair either. I've seen Cloudformation getting features years later. i.e 2021 - https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dynamodb-global-tables-now-support-aws-cloudformation/ https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-dy... 2015 - https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dynamodb-available-now-cross-region-replication-triggers-and-streams/ https://aws.amazon.com/about-aws/whats-new/2015/07/amazon-dy...
  - jen20 5y ago
    NAT Gateways is another notable feature that took CloudFormation months yet Terraform had on day 1. If you can configure something via CloudFormation you can integrate it via Terraform et al also, since they have resources representing CloudFormation stacks.
    - lincler 5y ago
      This! Is not like you can't go beyond what Terraform offers by default. Running CloudFormation stacks from Terraform is a neat way of solving missing apis/integration. And that's exactly what my team did when Terraform was missing a lot of lambdas functionalities. We just declared the CloudFormation Stack for lambdas and then call it from Terraform.
- jen20 5y ago
  There’s no reason you can’t do something similar with Terraform either - plugins speak GRPC and thus could be implemented in Python, with Node.js or with Rust. However, if AWS have not published metadata for a given service to be used across their various SDKs, it’s hard to take that service particularly seriously, so I’m not sure I’d bother with this.
dharmab 5y ago
Private preview features for partnered organizations. We had access up to 6 months before the public.

iddqd 5y ago

It usually takes 1-2 years for AWS to roll out their latest updates to the regions I use and by then Terraform is stable.

jrsdav 5y ago

> But using cloud APIs directly was great for us This is fine, I've done it extensively myself for some of the bleeding-edge cloud stuff, but the importance of things like tracking state, managing hierarchical resource dependencies, or retry/back-off logic shouldn't be tossed aside simply because there are gaps in what's available in the Terraform providers. Especially where change management is important (basically any enterprise company). I'd caution others reading this against abandoning something altogether and writing bespoke IaC tooling simply because the stable approach doesn't cover every (bleeding) edge case. You'll spend a lot of time reinventing the wheel, and while it's fine for certain situations (like when you only care about desired state, not known state, for instance), you'll move faster (and likely safer) by sticking with tools like Terraform for the bulk of your infra, and augmenting here there with cloud APIs/SDKs when needed.

dharmab 5y ago
Yes, we did have to implement our own state tracking, retries/recovery, etc- but since we were focused on a limited subset of the cloud API, this was pretty easy.
- Thaxll 5y ago
  So you re-implemented terraform but worse most likely. Also you could have added those missing features and re-use the TF engine, it's very simple to include new API of an existing provider.
  - dharmab 5y ago
    No, we used an entirely different architecture/paradigm not possible with tf, and had capabilities tf doesn't attempt to provide (such as coordinating migrations with both cloud and application APIs, or managing capacity while upgrading 10000s of CPU cores worth of compute).

goodpoint 5y ago

Same here. It's incredible how much efforts developers are willing to put on popular "devops" tools when the job could be done faster with 200 lines of Python.

dharmab 5y ago
This comment Gets It.

oneplane 5y ago

Would the time spent re-implementing a specialised Terraform subset be better spent simply maintaining a private branch of the AWS Provider? You can add your secret/special API without having to do all the other heavy lifting as well. This makes your own effort for customisation minimal, keeps your knowledge portable and because your added features can be separated in to different files and the provider API is stable you can also easily backport/fast-forward new changes.

dharmab 5y ago
Our custom code was fewer LOC, more robust and faster than tf. Honestly, it's not hard to beat OSS if you have a team of great engineers. This was far from the only time our internal projects were better than OSS- it gave us advantages for removing bottlenecks in the critical paths.
- oneplane 5y ago
  So when engineers left the company, was their knowledge portable? And attracting new talent, did they come with X years knowledge for the custom code? With fewer LOC I'm not sure what you mean, the provider code is pretty small, smaller than custom ansible, salt or puppet modules. Smaller than CDK and Pulumni as well. Sure, you'll have to write Go, but that's about the only hurdle. Everyone doing a round of NIH for internal tooling is ultimately not making the tide rise. Edit: don't get me wrong here, writing internal tools to do a job the right way for the right needs isn't "invalid" or something like that, but people often dismiss the rest of the lifecycle of knowledge and maintenance when making something completely custom.