3 ms·

We have an internal provider that handles generation of IAM Roles and IAM users via an external API for various security reasons I may or may not agree with. Th

We have an internal provider that handles generation of IAM Roles and IAM users via an external API for various security reasons I may or may not agree with. The constant provider changes in 0.10 and upward were extremely painful. We actually ended up having several teams who so hopelessly fubared their tfstates trying to skip minor releases, we ended up having to take it away and put together a task force to do it ourselves across almost 200 repositories. I’m not sure what you mean by “all three major cloud providers rewrote their own APIs.” Azure[1][2], AWS[3], and Google[4] are all maintained by Hashicorp. In fact, if you peruse the issues you’ll often see PRs opened by employees of the respective providers trying to fix blocking issues and they often devolve into literal begging for Hashicorp to respond and at least tell them why something hasn’t been merged. I know one blocker[5] actually cost Azure a very substantial customer as it languished in Hashicorp’s queue. Hashicorp’s constant refrain of “Well it’s a 0 version software” while selling enterprise support and constantly shilling their wares as production ready across the entire DevOps space was dishonest. I appreciate the position they were in and I appreciate even more their attempt to at least put out a good PR move with their 1.0 release. We will see how well it holds up over the years. What you call “that good” I call “better than everything else but still byzantine and hellish to deal with every time someone DMs me, ‘hey, you know terraform right?’” 1: https://github.com/hashicorp/terraform-provider-azuread https://github.com/hashicorp/terraform-provider-azuread 2: https://github.com/hashicorp/terraform-provider-azurerm https://github.com/hashicorp/terraform-provider-azurerm 3: https://github.com/hashicorp/terraform-provider-aws https://github.com/hashicorp/terraform-provider-aws 4: https://github.com/hashicorp/terraform-provider-google https://github.com/hashicorp/terraform-provider-google 5: https://github.com/hashicorp/terraform-provider-azurerm/pull/5485#issuecomment-591608953 https://github.com/hashicorp/terraform-provider-azurerm/pull...

busterarm 5y ago
Yes, you're complaining about provider problems. All terraform does is translate HCL into commands against the providers' own APIs. Right around the transition periods between 0.11 through 0.14, AWS, Google and Azure were rewriting large sections of their own public-facing APIs, changing tons of behaviors and then rewriting their Terraform providers to match the API changes. That's an issue with those providers' APIs and how their Terraform provider was architected. Most of the other Terraform providers were smooth-sailing during the same period, save for the major challenges involved in updating 0.11 to 0.12. You're putting the blame in the wrong place. In the case of all three of those cloud providers, the companies' own employees plus outside contributors maintain the terraform providers, not HashiCorp. HashiCorp gets involved but mostly to resolve errors in Terraform itself. Saying they are maintained by HashiCorp is completely incorrect. They are part of HashiCorp's repos (because they are official) but in each case the core contributors are people from AWS, Azure and Google. My company has large accounts with all of these, I contribute to them myself and I know(/knew. Dana @ Google moved onto another role and Google hasn't introduced me to her replacements yet) the maintainers of all of them personally. Don't look at who owns the repo, look at the contributor lists.
- krinchan 5y ago
  I don’t see why I, the end user, should be forced to make that differentiation since Terraform is effectively useless without the providers from the big three. Statements like “you’re blaming the wrong people” is in the same spirit as what ZeroVer is lampooning. Like yes, our internal provider was a pain point and we own that but the rest of the drama around providers was just weird. The work to move over 200 repositories through the hoops to keep them updated, especially mature services that may not have been deployed for several months, was difficult to automate and very brittle even when it was. It broke down at scale and really no one should have been using it that wide spread before it was 1.0. Hashicorp’s whole treatment of terraform 0.x was horrendous and constantly broke everything, all while they said it was production ready. You can blame whoever you want but the total lack of stability and easy upgrade paths and constant manual fiddling and reviewing output from ‘upgrade0.11’ type commands was ridiculous and a massive time sink for our org.
  - busterarm 5y ago
    I've written more Terraform than most people outside of possibly the folks at HashiCorp themselves and probably Gruntwork. Have been using it at 10^5 scale of systems and haven't had nearly the kind of problems that you're describing. Also the treadmill is really not any different than integrating with _ANY_ Google service. In fact I'd say it's an order of magnitude better. Google has set a standard of breaking changes without notification and if that's one of the providers you're using then I understand. And well, if it was Azure (as it likely looks to be) the state of their public facing APIs is/was an absolute fucking mess and the preferred way to do anything in their system still seems to be using the UI. I've talked to several people at Microsoft at Azure teams responsible and there's multiple compounding problems there. For one you have 200+ engineering teams with no unified approach to exposing services. Then you've got multiple regions in their cloud that for years didn't have the same authentication system, didn't have consistent features between regions, etc. There's very little you can lay at Hashicorp's feet for this when the underlying systems themselves have very poor automation. And then you talk about having 200+ repos and services that haven't been deployed for months and all I can say is the consensus around the need for CI/CD is over a decade old now and infrastructure needs these things just as much as code does. 100% of my Terraform is in a CI/CD pipeline. Yes it was a lot of work to set up, but the alternative is nothing but problems. Terraform is just a tool. It's not a panacea. It will not make all of your problems go away -- it's up to the craftsman how good it is.

busterarm 5y ago

Yes, you're complaining about provider problems. All terraform does is translate HCL into commands against the providers' own APIs. Right around the transition periods between 0.11 through 0.14, AWS, Google and Azure were rewriting large sections of their own public-facing APIs, changing tons of behaviors and then rewriting their Terraform providers to match the API changes. That's an issue with those providers' APIs and how their Terraform provider was architected. Most of the other Terraform providers were smooth-sailing during the same period, save for the major challenges involved in updating 0.11 to 0.12. You're putting the blame in the wrong place. In the case of all three of those cloud providers, the companies' own employees plus outside contributors maintain the terraform providers, not HashiCorp. HashiCorp gets involved but mostly to resolve errors in Terraform itself. Saying they are maintained by HashiCorp is completely incorrect. They are part of HashiCorp's repos (because they are official) but in each case the core contributors are people from AWS, Azure and Google. My company has large accounts with all of these, I contribute to them myself and I know(/knew. Dana @ Google moved onto another role and Google hasn't introduced me to her replacements yet) the maintainers of all of them personally. Don't look at who owns the repo, look at the contributor lists.

krinchan 5y ago
I don’t see why I, the end user, should be forced to make that differentiation since Terraform is effectively useless without the providers from the big three. Statements like “you’re blaming the wrong people” is in the same spirit as what ZeroVer is lampooning. Like yes, our internal provider was a pain point and we own that but the rest of the drama around providers was just weird. The work to move over 200 repositories through the hoops to keep them updated, especially mature services that may not have been deployed for several months, was difficult to automate and very brittle even when it was. It broke down at scale and really no one should have been using it that wide spread before it was 1.0. Hashicorp’s whole treatment of terraform 0.x was horrendous and constantly broke everything, all while they said it was production ready. You can blame whoever you want but the total lack of stability and easy upgrade paths and constant manual fiddling and reviewing output from ‘upgrade0.11’ type commands was ridiculous and a massive time sink for our org.
- busterarm 5y ago
  I've written more Terraform than most people outside of possibly the folks at HashiCorp themselves and probably Gruntwork. Have been using it at 10^5 scale of systems and haven't had nearly the kind of problems that you're describing. Also the treadmill is really not any different than integrating with _ANY_ Google service. In fact I'd say it's an order of magnitude better. Google has set a standard of breaking changes without notification and if that's one of the providers you're using then I understand. And well, if it was Azure (as it likely looks to be) the state of their public facing APIs is/was an absolute fucking mess and the preferred way to do anything in their system still seems to be using the UI. I've talked to several people at Microsoft at Azure teams responsible and there's multiple compounding problems there. For one you have 200+ engineering teams with no unified approach to exposing services. Then you've got multiple regions in their cloud that for years didn't have the same authentication system, didn't have consistent features between regions, etc. There's very little you can lay at Hashicorp's feet for this when the underlying systems themselves have very poor automation. And then you talk about having 200+ repos and services that haven't been deployed for months and all I can say is the consensus around the need for CI/CD is over a decade old now and infrastructure needs these things just as much as code does. 100% of my Terraform is in a CI/CD pipeline. Yes it was a lot of work to set up, but the alternative is nothing but problems. Terraform is just a tool. It's not a panacea. It will not make all of your problems go away -- it's up to the craftsman how good it is.