12 ms·

Terraform is currently not reviewing community pull requests

249 points by jaxxstorm 5y ago

danpalmer 5y ago
These things happen, props to Hashicorp for communicating about it. It would be easy to say that this is a failure of open-source in some way, but to do so would be unfair to the huge amount of work that companies put into tools like this, and the stewardship that they offer, both of which take a lot of time and money. If periods of low activity while teams change are the cost the community needs to pay for that, I think that's very fair.
- leetrout 5y ago
  I had a conversation with my coworkers when I worked there about being upfront with folks that we wouldn't review or accept their PR and stop leaving people hanging (this was on the TFE provider). I'm glad this was added. I stand by my statement then and now: there is nothing worse than contributing to something open source and then have your PR completely ignored.
  - jdavis703 5y ago
    I’ve made PRs because I need the change for my day job. By the time I’ve traced an issue down to a third party library the cost of making a PR is minimal. But if they don’t want to take the changes, it doesn’t bother me. Of course I don’t work on open source out of passion, so I could imagine this is different for the true believers.
    - mrtweetyhack 5y ago
      If you spent the time and effort to hunt down the problem for them, the least they could do is look into it. repos that don't care enough are a waste of time.
      - makeitdouble 5y ago
        TBF, even if your PR solves an actual issue a number of users have, it still might not be a good fit for the project's future, and starting a discussion about it might not be worth anyone's time. More importantly, as you disclosed your solution, other affected by the issue can rely on your code in the meantime (= the rest of the project's life in some cases. I've been there). If it really is a critical issue that isn't solved, your fix can always be used in a different fork with better maintenance. All in all, the upstream repo not responding to a PR isn't the end of the world I think, and the openness of the system makes it an acceptable state in many ways IMHO.
    - mwaitjmp 5y ago
      Out of interest then how do you proceed? Do you fork the code and run your own patched version?
      - jdavis703 5y ago
        Yes, just use our own forked version.
        arthurcolle 5y ago
        So then when your upstream repo diverges would you just rebase and manually add in anything you want from the forked development tree on the upstream side? Not sure what's best practice so... just curious how people have handled this - I usually leave my forks of stuff pretty stale and focus on my own little sub-pieces to achieve what I want but not too much else.
        rad_gruchalski 5y ago
        I usually have my own branch which can be used to get a diff and maintain the diff on any major code structure change. Most often there is not much hassle but can be annoying.
        dharmab 5y ago
        Yes, we would branch from upstream again and apply the patch set against the new branch. Normally it is trivial but once in a while there are manual changes necessary.
        acidbaseextract 5y ago
        Maintaining the fork as a stack of patches using something like Stacked Git can make it a little easier than rebasing: https://stacked-git.github.io/ https://stacked-git.github.io/
        don-code 5y ago
        I do this. I wish I didn't. First of all, building a provider isn't straightforward. The best way I've found to do this is to wrap `terraform init`, and have it `docker run` a build process for a plugin version that never existed - then dumping the built provider into the `.terraform` directory for the project. It's prone to failure; new users of the Terraform project complain that the build eats 8GB of RAM and takes many minutes. Second, providers are constantly changing, and it's not always possible to cleanly rebase a set of community changes on top of master. Part of the trouble with letting PRs wither on the vine is that they themselves become stale - in one case, the code still compiles, but the end result is completely wrong. For what it's worth: my use case was needing to use Terraform with some more "unusual" features of CloudFront and ALBs. The 80% use case for support is great. There's a remaining 15% that's well implemented by unmerged PRs, and another 5% entirely that's completely unsuppored. I've kept it IaC by using the `null_resource` provisioner to shell out to the AWS CLI where absolutely necessary.
  - qbasic_forever 5y ago
    I don't understand why people think that just doing unsolicitied work and pressing a button means the people on the other side are obligated to spend time reviewing and integrating it. GitHub has weirdly unbalanced open source contributions with a heavier burden and burnout on maintainers. Contributing to a project doesn't mean just slinging code and calling it good. Communicate--talk to the people maintaining the code. Ask them, hey I have an idea and want to add this feature/fix this bug/etc.--do you have bandwidth for that change? Mailing lists, discussion forums, chat rooms, e-mails, etc. are the place to sort this out, not snarky or even angrier and angrier replies to an unsolicited pull request that goes unreviewed.
    - leetrout 5y ago
      To be explicit: I agree. But I still think it's basic decency to close the PR and tell them No. Or if you plan to leave it around make a comment to that effect.
      - geerlingguy 5y ago
        There are many projects where just doing that would be the equivalent of a full time position. Even saying no requires a skilled review.
        amyjess 5y ago
        Is there no way to just have a script add a comment on every new PR saying "Due to insufficient staff, your PR may not be reviewed for a considerable amount of time."?
        tmp538394722 5y ago
        Your code is important to us. Please stay on the line, and the next available reviewer will answer your code. Due to unusually high code volumes, this response may take longer than usual. Something like that?
  - sergiotapia 5y ago
    I disagree, when someone opens a PR on my project it's an imposition of my time. I appreciate the help but I will review when I have time and feel like it's a good time to do it.
  - junon 5y ago
    > I stand by my statement then and now: there is nothing worse than contributing to something open source and then have your PR completely ignored Really? Nothing worse?
    - raffraffraff 5y ago
      Jock itch is pretty bad. Might even be worse.
- benatkin 5y ago
  They don't happen to community projects like Node.js, Rust, Vue, or Debian. Perhaps this is apples and oranges, but I'm most interested in open source projects that are governed by a foundation.
  - wmf 5y ago
    Community projects fall behind on PRs all the time. Many of them are permanently behind. Also, foundation != community.
  - void_mint 5y ago
    Comparing programming languages to individual open source projects is silly. Lots of Node and Rust projects are behind on PRs. Also Vue just doesn't allow community development, which makes everything much easier.
- mpd 5y ago
  I think it's the likely reality of far more projects than just terraform, and I don't fault the project completely for reaching this state - attrition is a brutal thing. It's distressing personally though, that it's an issue unknown to me before I read this post, and affects a product I use and champion.
- throwaway09223 5y ago
  "It would be easy to say that this is a failure of open-source in some way" I have seen far more commercial, closed source products go through similar staffing crunches. The difference is that the problems are hidden away behind misdirecting sales teams and so on. I can't tell you how many times I've reached out to someone on the inside of a company to get a straight answer as to whether a product is being properly staffed and supported. Or, conversely, how many times I myself have had to decide to orphan some commercial, customer facing work to meet a goal with a higher priority. In my experience, useful open source products are less likely to suffer from inadequate staffing than closed.
lloydatkinson 5y ago
Wow. Glad I started using ARM/Bicep for my recent learning about devops.
- dagss 5y ago
  ARM is a complete joke, and Bicep compiles down to ARM. (More details: ARM is simply a dumb script that says "do this, do that", it doesn't interact with your cloud resources in any smart way. As one example: You can download a template for an Azure SQL instance from the Azure portal. That template will then randomly fail to execute, because it contains two "configuration" child resources of Azure SQL, which ARM will try to deploy in paralell, but oh, they touch the same parent sources, so they crash with an error... Then try to add depends_on in order to have it, perhaps, not fail. Or how about the Azure SQL feature where, if you block Azure SQL to only AD logins, you'd have to declare an administrator password on first ARM run (resource creation), then remove it on subsequent ARM runs (resource update) because having it is then prevented... These aren't minor issue or glitches; it's symptomatic of a system that's just built in the wrong way from ground up. ARM isn't a stateless description of your resources, it's an awkward scripting language (although since the APIs it interacts with are idempotent the difference isn't obvious at first). How Microsoft could decide to have Bicep compile to ARM is beyond me -- Azure has some good features (refer to resources by name rather than allocated ID) that could greatly simplify the approach that Terraform/Pulumi takes if they only targeted Azure -- why didn't Microsoft do that instead, give us Terraform/Pulumi for Azure without having to keep a statefile (especially a statefile with secrets in it).
  - jen20 5y ago
    This is only partly accurate, because it conflates ARM the CRUD API with ARM Templates. ARM’s CRUD API is used by Terraform’s AzureRM provider (I bootstrapped that provider in 2016). I’m not sure exactly what you’re looking for with regards to referencing by name rather than ID - names must be qualified with a resource group and so forth, and to actually unambiguously identify things, you’d need all components of the ID. Bicep looks like a solution in search of a problem to me, though fortunately I haven’t had to spent more than about 5 minutes looking at it.
shadycuz 5y ago
Maybe they are desperate enough to hire me
- mrtweetyhack 5y ago
  nobody's THAT desperate haha jk
braddeicide 5y ago
Surely community patches are such a good return on investment they should be pulling devs off other areas to keep someone on reviewing public prs. I've always been confused by companies that aren't over the moon to spend minimal review time to get the benefit of hours of work by free employees.
- devoutsalsa 5y ago
  Getting people to submit PRs that stand up to your requirements can be a nontrivial exercise.
  - YorickPeterse 5y ago
    In addition, many people will only contribute once or twice. The result is that you as a maintainer may need to invest a lot of time, while the results are minimal.
- Etheryte 5y ago
  There are no free lunches, pull requests are no exception. For starters, before merging every pull request needs to be reviewed at a minimum. That by itself can oftentimes be a very time-consuming activity, especially if the changes are from someone outside the circle of regular contributors. Outside of fixes for typos and other trivialities, pull requests generally require a lot of back and forth to get to a good state — does this change make sense architecturally, does it cover edge cases, does it come with tests? Additionally, oftentimes pull requests expand the scope of what you need to maintain, whether you want to take on that permanent burden is a critical question in and of itself. The list goes on. There are many projects that do make it work, but make no mistake that this takes a considerable amount of effort.
- OJFord 5y ago
  Isn't a good review at least as hard as a good PR?
  - Notanothertoo 5y ago
    No, reading code is far easier than writing it. Either way both have go be done regardless of the author.
bawolff 5y ago
I don't see anything wrong with that. The beauty of open source is you can run your project however you want: bazzar, cathedral, or something else. If people think its a bad choice, they can fork.
electroly 5y ago
Props to them for being honest, but it's still not a good look for Hashicorp. Their stewardship of Terraform leaves a lot to be desired. For years now I've watched Terraform PRs just wither on the vine. You get the impression that nobody is working on the AWS provider at all. Every PR to the AWS provider that I've ever cared about has taken years to be reviewed and merged despite lots of thumbs-ups and comments from users begging for it to be merged. This has been ongoing for years. There is clearly a priority problem here.
- OJFord 5y ago
  (Minor contributor, perhaps major issue-opener-commenter speaking) I have a lot of respect for the team personally (as in myself), though I do recognise what you say completely too. I just think there's a lot of focus on trying to do things right, and long-term-maintainably, that it often presents with a poor outlook or like there's a lack of interest. @apparentlymart from OP in particular - I have no idea who he is in Hashicorp hierarchy, I just recognise him from GitHub - in particular is really excellent in responding to good issues and adding information along the lines of 'yes we want this but unfortunately XYZ so we're hoping PQR is going to make this easier but first we need to ABC so yes but sorry not a priority right now' sort of thing.
  - nomoreplease 5y ago
    Ya, but I’ve seen those same responses 4+ years ago. I had to give up on using it for a project years ago as they didn’t support some things and closed the PRs
- jen20 5y ago
  A few years ago, there used to be community reviewers of pull requests to the AWS provider - indeed I reviewed dozens after leaving HashiCorp. My own access for this got removed in (IIRC) late 2018, and I’d assume this was across the board. I don’t think it’s conceivable that any reasonable number of employees could satisfy the demand of maintaining the first-party provider set in the current form, without leveraging the community. However, I for one will not sign a CLA that allows proprietary relicensing, and I’d guess most people who could give meaningful reviews are in a similar boat, or already work on the provider teams. However, I’m also not sure that there is a “priority problem” as such - most providers don’t make HashiCorp money from consumers or contributors, and employee time is better spent on products which contribute to a positive bottom line. The Terraform Provider Registry has made it much more palatable to run a fork of any given provider than it was previously - I’d recommend doing so if you have functionality you need that hasn’t been integrated.
topspin 5y ago
One of these days Hashicorp is going to cash in and one of the major cloud operators will own the lingua franca of cloud provisioning. Oracle, for instance, directs the cloud users to Terraform as its de facto first class provisioning tool. It works rather well BTW.
- zxcvbn4038 5y ago
  I've been very happy with Terraform. I also have a huge investment in Cloudformation and it doesn't take long to discover that Cloudformation isn't implemented consistently across services, it's full of bugs, and unless your bug happens to impact Amazon internally you can wait years for a fix. New functionality for existing resources can also take years to materialize. Back when I had an eight figure AWS budget they would offer to let me fund a Cloudformation fix (seriously! They wanted me to pay them to fix their errors) but since moving to a smaller shop they don't even offer that. With Terraform (and its AWS provider) I've found things are generally implemented consistently, I've not encountered any crashing bugs yet, and all of the manual steps I've had to do with the AWS CLI have Terraform equivalents.
  - jamesfinlayson 5y ago
    Looking at what CloudFormation can and can't do it seems like completely different teams work on a service and its CloudFormation support. I guess Amazon use the AWS API internally?
    - devcpp 5y ago
      I work at AWS. The service teams do own and write their CloudFormation providers, although if you've written a custom provider you'd see it is somewhat clunky, so it's sometimes considered more of an operational burden, and you can tell. We dogfood both the SDK and CloudFormation internally, we just deal with its numerous gripes much the same way you would externally (although we can also contact service teams directly if needed).
      - 5y ago
encryptluks2 5y ago
So Terraform manages to become the recommended DevOps tool for most cloud providers and now they won't even accept PRs to improve and add features from the community? I think there are greater issues here, first that we centralized around so few major providers instead of improving tooling to scale to any hosting provider and 2 that we let hosting providers create so much abstraction that each one requires an entirely different provider API. Clouds should be as simple as renting VMs or dedicated servers from hosting providers and having an image that automatically brings everything up for you.
- void_mint 5y ago
  > So Terraform manages to become the recommended DevOps tool for most cloud providers It's not the recommended devops tool for any cloud provider. > and now they won't even accept PRs to improve and add features from the community? Random people writing code doesn't make that code valuable or of any sound quality. They have employees whose jobs it is to improve and add features to their software. > Clouds should be as Clouds are and should be exactly as simple or complex as their customers request. Clouds should not be built around opinionated forum posts.
  - encryptluks2 5y ago
    > It's not the recommended devops tool for any cloud provider. The Deployment manager templates under CFT for GCP specifically state they recommend the Terraform modules. It is also the first provider listed under Oracle cloud. Pretty much every cloud list it as the preferred DevOps tool. > Random people writing code doesn't make that code valuable or of any sound quality. They have employees whose jobs it is to improve and add features to their software. It doesn't mean that it isn't. You act like only employees can write valuable code with sound quality, when in fact the opposite could be true and often is. > Clouds are and should be exactly as simple or complex as their customers request. Clouds should not be built around opinionated forum posts. The concept of a cloud provider is not anything unique. Hence why there are so many of them implementing their own API. Cloud providers budgets are heavily skewed to marketing, because if everyone could get the rates available through OVH, Hetzner, etc while still getting the same features through their own 3+node zero-config cluster then no one would pay the outrageous prices of AWS, Azure, etc.
terrathrowaway 5y ago
I commented, committed, worked with contributors and Hashicorp employees early on in Terraform. It was so refreshing to see a project that did what I wanted, but better than the tooling I'd already written -- and that it was so actively working with the community. Phinze was a large part of feeling like my contributions were welcomed, even just as a community member. A few months back I encountered something of Paul's, noticed his GitHub work had tapered off and found the blog posts entitled "diagnosis" and "treatment" on his blog. I know nothing more about current events pertaining to Hashicorp or Terraform, but seeing some negativity here coupled with the experience of encountering that sad news led me to comment. That team has made world class tooling, with the community. The people I know who work there are some of the sharpest and kindest people I've ever met. I'm willing to give them the benefit of the doubt and I hope you yourself will as well before presuming anything that leads to more negativity. It's obviously not ideal, but perhaps there are legitimate reasons and at least they were transparent.
mdaniel 5y ago
I recall once upon a time there was a GitHub project wherein the owner would just immediately merge any PRs that were opened -- I believe it was a social experiment, and I don't recall the exact nature of the repo in order to know if that kind of thing is ludicrous here. But I do think it'd be good fun to take this lull and find out the outcome of a hypothetical github.com/open-terraform/open-terraform which just ran a github action that merged PRs that had more than a 5(10?) differential of :+1: to :-1: reactions on it. Build failures would instantly close the PR, and test failures would be exempt from auto-merge If that worked sufficiently well, I'd initially mirror every repo from https://github.com/terraform-providers https://github.com/terraform-providers into that same GitHub organization and continue that exercise. IMHO the providers suffer from bitrot a lot more than formal terraform does I think of that hypothesis as a "open source optimist" versus "open source pessimist:" are people who go out of their way to open PRs trying to improve the common good, or trying to drain the life out of maintainers?
- bpodgursky 5y ago
  https://github.com/illacceptanything/illacceptanything https://github.com/illacceptanything/illacceptanything
  - mdaniel 5y ago
    Heh, thanks for tracking that down, and it's as marvelous as one would expect from 6 years of exposure to the Internet :-)
- kubanczyk 5y ago
  Canonical article about that: Pieter Hintjens "Why Optimistic Merging Works Better" (2015) http://hintjens.com/blog:106 http://hintjens.com/blog:106
Aeolun 5y ago
Didn’t they just release v1? How can their staffing be low right now?
purpleidea 5y ago
It's cool that they're being open about this, but I'd be curious to know if the situation was their own doing or not? For example, how many non-hashicorp employees are maintainers? Do they allow this? (I'm actually asking-- I don't know the answer.) Open source is great, but a single SPOF in one company is becoming too much of an healthy norm. If you love your software, set it free. And if you're worried about people^W companies taking it proprietary and not giving back, then use copyleft.
nikolay 5y ago
This is pretty sad as it has too much of a lost potential with the slow updates of the core. I've been waiting for dynamic providers (i.e. created via for_each, count, or dynamic), but they are not coming to Terraform anytime soon and the copypasta must go on. Some of the providers, which HashiCorp maintains are pretty behind. It used to be where people used Terraform for AWS, because CloudFormation was so behind, but nowadays it's the opposite story.
awinter-py 5y ago
'help wanted or else'
deleted 5y ago
[deleted]

249 points by jaxxstorm 5y ago

These things happen, props to Hashicorp for communicating about it. It would be easy to say that this is a failure of open-source in some way, but to do so would be unfair to the huge amount of work that companies put into tools like this, and the stewardship that they offer, both of which take a lot of time and money. If periods of low activity while teams change are the cost the community needs to pay for that, I think that's very fair.

leetrout 5y ago
I had a conversation with my coworkers when I worked there about being upfront with folks that we wouldn't review or accept their PR and stop leaving people hanging (this was on the TFE provider). I'm glad this was added. I stand by my statement then and now: there is nothing worse than contributing to something open source and then have your PR completely ignored.
- jdavis703 5y ago
  I’ve made PRs because I need the change for my day job. By the time I’ve traced an issue down to a third party library the cost of making a PR is minimal. But if they don’t want to take the changes, it doesn’t bother me. Of course I don’t work on open source out of passion, so I could imagine this is different for the true believers.
  - mrtweetyhack 5y ago
    If you spent the time and effort to hunt down the problem for them, the least they could do is look into it. repos that don't care enough are a waste of time.
    - makeitdouble 5y ago
      TBF, even if your PR solves an actual issue a number of users have, it still might not be a good fit for the project's future, and starting a discussion about it might not be worth anyone's time. More importantly, as you disclosed your solution, other affected by the issue can rely on your code in the meantime (= the rest of the project's life in some cases. I've been there). If it really is a critical issue that isn't solved, your fix can always be used in a different fork with better maintenance. All in all, the upstream repo not responding to a PR isn't the end of the world I think, and the openness of the system makes it an acceptable state in many ways IMHO.
  - mwaitjmp 5y ago
    Out of interest then how do you proceed? Do you fork the code and run your own patched version?
    - jdavis703 5y ago
      Yes, just use our own forked version.
      - arthurcolle 5y ago
        So then when your upstream repo diverges would you just rebase and manually add in anything you want from the forked development tree on the upstream side? Not sure what's best practice so... just curious how people have handled this - I usually leave my forks of stuff pretty stale and focus on my own little sub-pieces to achieve what I want but not too much else.
        rad_gruchalski 5y ago
        I usually have my own branch which can be used to get a diff and maintain the diff on any major code structure change. Most often there is not much hassle but can be annoying.
        dharmab 5y ago
        Yes, we would branch from upstream again and apply the patch set against the new branch. Normally it is trivial but once in a while there are manual changes necessary.
        acidbaseextract 5y ago
        Maintaining the fork as a stack of patches using something like Stacked Git can make it a little easier than rebasing: https://stacked-git.github.io/ https://stacked-git.github.io/
      - don-code 5y ago
        I do this. I wish I didn't. First of all, building a provider isn't straightforward. The best way I've found to do this is to wrap `terraform init`, and have it `docker run` a build process for a plugin version that never existed - then dumping the built provider into the `.terraform` directory for the project. It's prone to failure; new users of the Terraform project complain that the build eats 8GB of RAM and takes many minutes. Second, providers are constantly changing, and it's not always possible to cleanly rebase a set of community changes on top of master. Part of the trouble with letting PRs wither on the vine is that they themselves become stale - in one case, the code still compiles, but the end result is completely wrong. For what it's worth: my use case was needing to use Terraform with some more "unusual" features of CloudFront and ALBs. The 80% use case for support is great. There's a remaining 15% that's well implemented by unmerged PRs, and another 5% entirely that's completely unsuppored. I've kept it IaC by using the `null_resource` provisioner to shell out to the AWS CLI where absolutely necessary.
- qbasic_forever 5y ago
  I don't understand why people think that just doing unsolicitied work and pressing a button means the people on the other side are obligated to spend time reviewing and integrating it. GitHub has weirdly unbalanced open source contributions with a heavier burden and burnout on maintainers. Contributing to a project doesn't mean just slinging code and calling it good. Communicate--talk to the people maintaining the code. Ask them, hey I have an idea and want to add this feature/fix this bug/etc.--do you have bandwidth for that change? Mailing lists, discussion forums, chat rooms, e-mails, etc. are the place to sort this out, not snarky or even angrier and angrier replies to an unsolicited pull request that goes unreviewed.
  - leetrout 5y ago
    To be explicit: I agree. But I still think it's basic decency to close the PR and tell them No. Or if you plan to leave it around make a comment to that effect.
    - geerlingguy 5y ago
      There are many projects where just doing that would be the equivalent of a full time position. Even saying no requires a skilled review.
      - amyjess 5y ago
        Is there no way to just have a script add a comment on every new PR saying "Due to insufficient staff, your PR may not be reviewed for a considerable amount of time."?
        tmp538394722 5y ago
        Your code is important to us. Please stay on the line, and the next available reviewer will answer your code. Due to unusually high code volumes, this response may take longer than usual. Something like that?
- sergiotapia 5y ago
  I disagree, when someone opens a PR on my project it's an imposition of my time. I appreciate the help but I will review when I have time and feel like it's a good time to do it.
- junon 5y ago
  > I stand by my statement then and now: there is nothing worse than contributing to something open source and then have your PR completely ignored Really? Nothing worse?
  - raffraffraff 5y ago
    Jock itch is pretty bad. Might even be worse.
benatkin 5y ago
They don't happen to community projects like Node.js, Rust, Vue, or Debian. Perhaps this is apples and oranges, but I'm most interested in open source projects that are governed by a foundation.
- wmf 5y ago
  Community projects fall behind on PRs all the time. Many of them are permanently behind. Also, foundation != community.
- void_mint 5y ago
  Comparing programming languages to individual open source projects is silly. Lots of Node and Rust projects are behind on PRs. Also Vue just doesn't allow community development, which makes everything much easier.
mpd 5y ago
I think it's the likely reality of far more projects than just terraform, and I don't fault the project completely for reaching this state - attrition is a brutal thing. It's distressing personally though, that it's an issue unknown to me before I read this post, and affects a product I use and champion.
throwaway09223 5y ago
"It would be easy to say that this is a failure of open-source in some way" I have seen far more commercial, closed source products go through similar staffing crunches. The difference is that the problems are hidden away behind misdirecting sales teams and so on. I can't tell you how many times I've reached out to someone on the inside of a company to get a straight answer as to whether a product is being properly staffed and supported. Or, conversely, how many times I myself have had to decide to orphan some commercial, customer facing work to meet a goal with a higher priority. In my experience, useful open source products are less likely to suffer from inadequate staffing than closed.

lloydatkinson 5y ago

Wow. Glad I started using ARM/Bicep for my recent learning about devops.

dagss 5y ago
ARM is a complete joke, and Bicep compiles down to ARM. (More details: ARM is simply a dumb script that says "do this, do that", it doesn't interact with your cloud resources in any smart way. As one example: You can download a template for an Azure SQL instance from the Azure portal. That template will then randomly fail to execute, because it contains two "configuration" child resources of Azure SQL, which ARM will try to deploy in paralell, but oh, they touch the same parent sources, so they crash with an error... Then try to add depends_on in order to have it, perhaps, not fail. Or how about the Azure SQL feature where, if you block Azure SQL to only AD logins, you'd have to declare an administrator password on first ARM run (resource creation), then remove it on subsequent ARM runs (resource update) because having it is then prevented... These aren't minor issue or glitches; it's symptomatic of a system that's just built in the wrong way from ground up. ARM isn't a stateless description of your resources, it's an awkward scripting language (although since the APIs it interacts with are idempotent the difference isn't obvious at first). How Microsoft could decide to have Bicep compile to ARM is beyond me -- Azure has some good features (refer to resources by name rather than allocated ID) that could greatly simplify the approach that Terraform/Pulumi takes if they only targeted Azure -- why didn't Microsoft do that instead, give us Terraform/Pulumi for Azure without having to keep a statefile (especially a statefile with secrets in it).
- jen20 5y ago
  This is only partly accurate, because it conflates ARM the CRUD API with ARM Templates. ARM’s CRUD API is used by Terraform’s AzureRM provider (I bootstrapped that provider in 2016). I’m not sure exactly what you’re looking for with regards to referencing by name rather than ID - names must be qualified with a resource group and so forth, and to actually unambiguously identify things, you’d need all components of the ID. Bicep looks like a solution in search of a problem to me, though fortunately I haven’t had to spent more than about 5 minutes looking at it.

shadycuz 5y ago

Maybe they are desperate enough to hire me

mrtweetyhack 5y ago
nobody's THAT desperate haha jk

braddeicide 5y ago

Surely community patches are such a good return on investment they should be pulling devs off other areas to keep someone on reviewing public prs. I've always been confused by companies that aren't over the moon to spend minimal review time to get the benefit of hours of work by free employees.

devoutsalsa 5y ago
Getting people to submit PRs that stand up to your requirements can be a nontrivial exercise.
- YorickPeterse 5y ago
  In addition, many people will only contribute once or twice. The result is that you as a maintainer may need to invest a lot of time, while the results are minimal.
Etheryte 5y ago
There are no free lunches, pull requests are no exception. For starters, before merging every pull request needs to be reviewed at a minimum. That by itself can oftentimes be a very time-consuming activity, especially if the changes are from someone outside the circle of regular contributors. Outside of fixes for typos and other trivialities, pull requests generally require a lot of back and forth to get to a good state — does this change make sense architecturally, does it cover edge cases, does it come with tests? Additionally, oftentimes pull requests expand the scope of what you need to maintain, whether you want to take on that permanent burden is a critical question in and of itself. The list goes on. There are many projects that do make it work, but make no mistake that this takes a considerable amount of effort.
OJFord 5y ago
Isn't a good review at least as hard as a good PR?
- Notanothertoo 5y ago
  No, reading code is far easier than writing it. Either way both have go be done regardless of the author.

bawolff 5y ago

I don't see anything wrong with that. The beauty of open source is you can run your project however you want: bazzar, cathedral, or something else. If people think its a bad choice, they can fork.

electroly 5y ago

Props to them for being honest, but it's still not a good look for Hashicorp. Their stewardship of Terraform leaves a lot to be desired. For years now I've watched Terraform PRs just wither on the vine. You get the impression that nobody is working on the AWS provider at all. Every PR to the AWS provider that I've ever cared about has taken years to be reviewed and merged despite lots of thumbs-ups and comments from users begging for it to be merged. This has been ongoing for years. There is clearly a priority problem here.

OJFord 5y ago
(Minor contributor, perhaps major issue-opener-commenter speaking) I have a lot of respect for the team personally (as in myself), though I do recognise what you say completely too. I just think there's a lot of focus on trying to do things right, and long-term-maintainably, that it often presents with a poor outlook or like there's a lack of interest. @apparentlymart from OP in particular - I have no idea who he is in Hashicorp hierarchy, I just recognise him from GitHub - in particular is really excellent in responding to good issues and adding information along the lines of 'yes we want this but unfortunately XYZ so we're hoping PQR is going to make this easier but first we need to ABC so yes but sorry not a priority right now' sort of thing.
- nomoreplease 5y ago
  Ya, but I’ve seen those same responses 4+ years ago. I had to give up on using it for a project years ago as they didn’t support some things and closed the PRs
jen20 5y ago
A few years ago, there used to be community reviewers of pull requests to the AWS provider - indeed I reviewed dozens after leaving HashiCorp. My own access for this got removed in (IIRC) late 2018, and I’d assume this was across the board. I don’t think it’s conceivable that any reasonable number of employees could satisfy the demand of maintaining the first-party provider set in the current form, without leveraging the community. However, I for one will not sign a CLA that allows proprietary relicensing, and I’d guess most people who could give meaningful reviews are in a similar boat, or already work on the provider teams. However, I’m also not sure that there is a “priority problem” as such - most providers don’t make HashiCorp money from consumers or contributors, and employee time is better spent on products which contribute to a positive bottom line. The Terraform Provider Registry has made it much more palatable to run a fork of any given provider than it was previously - I’d recommend doing so if you have functionality you need that hasn’t been integrated.

topspin 5y ago

One of these days Hashicorp is going to cash in and one of the major cloud operators will own the lingua franca of cloud provisioning. Oracle, for instance, directs the cloud users to Terraform as its de facto first class provisioning tool. It works rather well BTW.

zxcvbn4038 5y ago
I've been very happy with Terraform. I also have a huge investment in Cloudformation and it doesn't take long to discover that Cloudformation isn't implemented consistently across services, it's full of bugs, and unless your bug happens to impact Amazon internally you can wait years for a fix. New functionality for existing resources can also take years to materialize. Back when I had an eight figure AWS budget they would offer to let me fund a Cloudformation fix (seriously! They wanted me to pay them to fix their errors) but since moving to a smaller shop they don't even offer that. With Terraform (and its AWS provider) I've found things are generally implemented consistently, I've not encountered any crashing bugs yet, and all of the manual steps I've had to do with the AWS CLI have Terraform equivalents.
- jamesfinlayson 5y ago
  Looking at what CloudFormation can and can't do it seems like completely different teams work on a service and its CloudFormation support. I guess Amazon use the AWS API internally?
  - devcpp 5y ago
    I work at AWS. The service teams do own and write their CloudFormation providers, although if you've written a custom provider you'd see it is somewhat clunky, so it's sometimes considered more of an operational burden, and you can tell. We dogfood both the SDK and CloudFormation internally, we just deal with its numerous gripes much the same way you would externally (although we can also contact service teams directly if needed).
    - 5y ago

encryptluks2 5y ago

So Terraform manages to become the recommended DevOps tool for most cloud providers and now they won't even accept PRs to improve and add features from the community? I think there are greater issues here, first that we centralized around so few major providers instead of improving tooling to scale to any hosting provider and 2 that we let hosting providers create so much abstraction that each one requires an entirely different provider API. Clouds should be as simple as renting VMs or dedicated servers from hosting providers and having an image that automatically brings everything up for you.

void_mint 5y ago
> So Terraform manages to become the recommended DevOps tool for most cloud providers It's not the recommended devops tool for any cloud provider. > and now they won't even accept PRs to improve and add features from the community? Random people writing code doesn't make that code valuable or of any sound quality. They have employees whose jobs it is to improve and add features to their software. > Clouds should be as Clouds are and should be exactly as simple or complex as their customers request. Clouds should not be built around opinionated forum posts.
- encryptluks2 5y ago
  > It's not the recommended devops tool for any cloud provider. The Deployment manager templates under CFT for GCP specifically state they recommend the Terraform modules. It is also the first provider listed under Oracle cloud. Pretty much every cloud list it as the preferred DevOps tool. > Random people writing code doesn't make that code valuable or of any sound quality. They have employees whose jobs it is to improve and add features to their software. It doesn't mean that it isn't. You act like only employees can write valuable code with sound quality, when in fact the opposite could be true and often is. > Clouds are and should be exactly as simple or complex as their customers request. Clouds should not be built around opinionated forum posts. The concept of a cloud provider is not anything unique. Hence why there are so many of them implementing their own API. Cloud providers budgets are heavily skewed to marketing, because if everyone could get the rates available through OVH, Hetzner, etc while still getting the same features through their own 3+node zero-config cluster then no one would pay the outrageous prices of AWS, Azure, etc.

terrathrowaway 5y ago

I commented, committed, worked with contributors and Hashicorp employees early on in Terraform. It was so refreshing to see a project that did what I wanted, but better than the tooling I'd already written -- and that it was so actively working with the community. Phinze was a large part of feeling like my contributions were welcomed, even just as a community member. A few months back I encountered something of Paul's, noticed his GitHub work had tapered off and found the blog posts entitled "diagnosis" and "treatment" on his blog. I know nothing more about current events pertaining to Hashicorp or Terraform, but seeing some negativity here coupled with the experience of encountering that sad news led me to comment. That team has made world class tooling, with the community. The people I know who work there are some of the sharpest and kindest people I've ever met. I'm willing to give them the benefit of the doubt and I hope you yourself will as well before presuming anything that leads to more negativity. It's obviously not ideal, but perhaps there are legitimate reasons and at least they were transparent.

mdaniel 5y ago

I recall once upon a time there was a GitHub project wherein the owner would just immediately merge any PRs that were opened -- I believe it was a social experiment, and I don't recall the exact nature of the repo in order to know if that kind of thing is ludicrous here. But I do think it'd be good fun to take this lull and find out the outcome of a hypothetical github.com/open-terraform/open-terraform which just ran a github action that merged PRs that had more than a 5(10?) differential of :+1: to :-1: reactions on it. Build failures would instantly close the PR, and test failures would be exempt from auto-merge If that worked sufficiently well, I'd initially mirror every repo from https://github.com/terraform-providers https://github.com/terraform-providers into that same GitHub organization and continue that exercise. IMHO the providers suffer from bitrot a lot more than formal terraform does I think of that hypothesis as a "open source optimist" versus "open source pessimist:" are people who go out of their way to open PRs trying to improve the common good, or trying to drain the life out of maintainers?

bpodgursky 5y ago
https://github.com/illacceptanything/illacceptanything https://github.com/illacceptanything/illacceptanything
- mdaniel 5y ago
  Heh, thanks for tracking that down, and it's as marvelous as one would expect from 6 years of exposure to the Internet :-)
kubanczyk 5y ago
Canonical article about that: Pieter Hintjens "Why Optimistic Merging Works Better" (2015) http://hintjens.com/blog:106 http://hintjens.com/blog:106

Aeolun 5y ago

Didn’t they just release v1? How can their staffing be low right now?

purpleidea 5y ago

It's cool that they're being open about this, but I'd be curious to know if the situation was their own doing or not? For example, how many non-hashicorp employees are maintainers? Do they allow this? (I'm actually asking-- I don't know the answer.) Open source is great, but a single SPOF in one company is becoming too much of an healthy norm. If you love your software, set it free. And if you're worried about people^W companies taking it proprietary and not giving back, then use copyleft.

nikolay 5y ago

This is pretty sad as it has too much of a lost potential with the slow updates of the core. I've been waiting for dynamic providers (i.e. created via for_each, count, or dynamic), but they are not coming to Terraform anytime soon and the copypasta must go on. Some of the providers, which HashiCorp maintains are pretty behind. It used to be where people used Terraform for AWS, because CloudFormation was so behind, but nowadays it's the opposite story.

awinter-py 5y ago

'help wanted or else'

deleted 5y ago

[deleted]