5 ms·

It's very difficult to communicate what Terraform's strengths and weaknesses are to someone who's never used it or IAC in general. Spend enough time playing wi

It's very difficult to communicate what Terraform's strengths and weaknesses are to someone who's never used it or IAC in general. Spend enough time playing with it and understanding it, you'll end up like me thinking about all the shit you configure left and right such as hooking up Stripe's secret keys, Google Analytics and the webmaster console, and just about everything else we configure via web interfaces, and you'll think: Why can't we use Terraform for this as well? Manage these SaaS products the same way we manage the rest of our cloud, test and audit changes, automatically roll secrets and update anything that needs updating the moment you change a setting. Ah well. Not enough APIs out there. And it's difficult to write and maintain terraform plugins for these throwaway cases especially if they are going to use private APIs. Anyone know if Pulumi plugins are easier to write?

satya71 5y ago
Pulumi has dynamic providers. You define the crud operations and Pulumi manage the state.
- reilly3000 5y ago
  After a little grokking it’s a surprisingly easy way to manage arbitrary API resources. They are just functions in your language of choice and can accept parameters from any dependency. It’s also possible to roll your own provider, but dynamic providers cover all kinds of use cases.
deleted 5y ago
[deleted]
crabmusket 5y ago
I've used https://github.com/Mastercard/terraform-provider-restapi https://github.com/Mastercard/terraform-provider-restapi successfully with a cloud provider which provides a suitable HTTP API. There was a bit of fiddling with JSON formatting and their API docs, but it wasn't too hard all in all. But like you say - now I've done that, I want to do it for every UI that I'm forced to log in to!
- scrollaway 5y ago
  Wow, neat, thanks for the link. Maintained by Mastercard, eh? Anyone else on HN used this or worked on it?
- MuffinFlavored 5y ago
  what’s the backing database for this, the .tfstste file? do the resources/secrets you create end up getting “backed up” (committed) to git?
  - crabmusket 5y ago
    That's right, which is why they usually recommend not committing tfstate but using some remote backend. We've gotten away without that because the APIs we use don't contain any secrets. (The auth token for making the requests is just an environment variable.)
- weitzj 5y ago
  Yes. We use the Rest Api Provider extensively to provision ElasticSearch and Kibana.
mason55 5y ago
This is how I feel about any kind of configuration after moving all my personal systems to NixOS. “What do you mean run an installer and update these files…”
brightball 5y ago
IMO, this is an area where I think Terraform + Ansible pairs so well together. If there’s ever a gap in what Terraform offers you can pretty easily fill it with Ansible.
- weitzj 5y ago
  And this setup is also encouraged by HashiCorp (at least I saw a talk by them). Use ansible for your “smart” sequential executions and Terraform as a sane wrapper for state.
  - deleted 5y ago
    [deleted]
  - chucky_z 5y ago
    I’ve found running Terraform via Ansible to be a pretty good experience.
    - weitzj 5y ago
      So the other way around. Do you employ a GitOps approach this way? I find it hard to figure out how to use GitOps with Ansible? How do you make a PullRequest which indicates that something should get deleted? You still would have to keep around an ansible playbook for the stuff you want to delete.
  - Octabrain 5y ago
    I’ve seen (and fixed) so many ugly messes at this point made as a result of mixing and wrapping tools with different purposes together like Ansible + Terraform that it’s something I strongly discourage. I recommend to keep the boundaries and responsibilities of the tools clear. In this case, Terraform for the creation of resources and Ansible for the configuration of those resources. In my opinion, this gives as a result a much simpler and maintainable ecosystem in the long run.
- pram 5y ago
  Ansible is also amazing with Packer as the provisioner.
vasco 5y ago
I wish I could use it for everything as well. Every other tool the business depends on just scares me to shit that it's config isn't in code and we can't have a proper backup of it. At least Datadog supports terraform for most things and not only we manage all our infra through terraform we manage all our monitoring with it too. I doubt very much I'll ever go back to non-monitoring-as-code if that's even a term. Infrastructure, all the monitoring as well as all the on-call rotation configurations (and anything else that is in that loop) should all be in code, and all changes should be reviewed the same way as application code does. If it doesn't, you can't really trust you're gonna be alerted properly when things start breaking. I wish I could use it for personal things too, I'd rather have my bank account settings, my government tax information, yada yada in a personal terraform repository for example. Change of address? Commit a a change, check if the plan is good and apply to change it everywhere. Though having lots of experience with Terraform I can only imagine what the equivalent of trying to delete an S3 bucket that still has data in it is for a bank account.
- OJFord 5y ago
  I so agree; I tried/am trying to write an Android provider - currently just have app (un)installation working, and not very well, I expected settings management to be the hard part, but egh. Why can't everything have nice public APIs! And, while I'm at it, some sort of all-encompassing ticketing system, hell even if it were Jira. 'Pothole', assign local council. Blocked on '2021 roadworks funding increase', backlogged. Assigned to councillor. Won't fix. Ok - maybe I'm not making it sound great, but at least you could see some reasoning, and what the blockers are. Follow the chain to work out that 'communal lobby needs repainting' hasn't been acted kn by building management company because, ultimately, of global supply chain disruption and the contractor's supplier's supplier can't get any paint ingredients.
kall 5y ago
Your intuition is right on pulumi. Creating these kinds of extensions is minimal effort. You can start out by adding a "dynamic resource" class to your infra codebase and extract it into a plugin later, or not. These are not the same as "real" pulumi providers that run across all supported programming languages, but I think they are a good enough fit for the cases you mention.

satya71 5y ago

Pulumi has dynamic providers. You define the crud operations and Pulumi manage the state.

reilly3000 5y ago
After a little grokking it’s a surprisingly easy way to manage arbitrary API resources. They are just functions in your language of choice and can accept parameters from any dependency. It’s also possible to roll your own provider, but dynamic providers cover all kinds of use cases.

deleted 5y ago

[deleted]

crabmusket 5y ago

I've used https://github.com/Mastercard/terraform-provider-restapi https://github.com/Mastercard/terraform-provider-restapi successfully with a cloud provider which provides a suitable HTTP API. There was a bit of fiddling with JSON formatting and their API docs, but it wasn't too hard all in all. But like you say - now I've done that, I want to do it for every UI that I'm forced to log in to!

scrollaway 5y ago
Wow, neat, thanks for the link. Maintained by Mastercard, eh? Anyone else on HN used this or worked on it?
MuffinFlavored 5y ago
what’s the backing database for this, the .tfstste file? do the resources/secrets you create end up getting “backed up” (committed) to git?
- crabmusket 5y ago
  That's right, which is why they usually recommend not committing tfstate but using some remote backend. We've gotten away without that because the APIs we use don't contain any secrets. (The auth token for making the requests is just an environment variable.)
weitzj 5y ago
Yes. We use the Rest Api Provider extensively to provision ElasticSearch and Kibana.

mason55 5y ago

This is how I feel about any kind of configuration after moving all my personal systems to NixOS. “What do you mean run an installer and update these files…”

brightball 5y ago

IMO, this is an area where I think Terraform + Ansible pairs so well together. If there’s ever a gap in what Terraform offers you can pretty easily fill it with Ansible.

weitzj 5y ago
And this setup is also encouraged by HashiCorp (at least I saw a talk by them). Use ansible for your “smart” sequential executions and Terraform as a sane wrapper for state.
- deleted 5y ago
  [deleted]
- chucky_z 5y ago
  I’ve found running Terraform via Ansible to be a pretty good experience.
  - weitzj 5y ago
    So the other way around. Do you employ a GitOps approach this way? I find it hard to figure out how to use GitOps with Ansible? How do you make a PullRequest which indicates that something should get deleted? You still would have to keep around an ansible playbook for the stuff you want to delete.
- Octabrain 5y ago
  I’ve seen (and fixed) so many ugly messes at this point made as a result of mixing and wrapping tools with different purposes together like Ansible + Terraform that it’s something I strongly discourage. I recommend to keep the boundaries and responsibilities of the tools clear. In this case, Terraform for the creation of resources and Ansible for the configuration of those resources. In my opinion, this gives as a result a much simpler and maintainable ecosystem in the long run.
pram 5y ago
Ansible is also amazing with Packer as the provisioner.

vasco 5y ago

I wish I could use it for everything as well. Every other tool the business depends on just scares me to shit that it's config isn't in code and we can't have a proper backup of it. At least Datadog supports terraform for most things and not only we manage all our infra through terraform we manage all our monitoring with it too. I doubt very much I'll ever go back to non-monitoring-as-code if that's even a term. Infrastructure, all the monitoring as well as all the on-call rotation configurations (and anything else that is in that loop) should all be in code, and all changes should be reviewed the same way as application code does. If it doesn't, you can't really trust you're gonna be alerted properly when things start breaking. I wish I could use it for personal things too, I'd rather have my bank account settings, my government tax information, yada yada in a personal terraform repository for example. Change of address? Commit a a change, check if the plan is good and apply to change it everywhere. Though having lots of experience with Terraform I can only imagine what the equivalent of trying to delete an S3 bucket that still has data in it is for a bank account.

OJFord 5y ago
I so agree; I tried/am trying to write an Android provider - currently just have app (un)installation working, and not very well, I expected settings management to be the hard part, but egh. Why can't everything have nice public APIs! And, while I'm at it, some sort of all-encompassing ticketing system, hell even if it were Jira. 'Pothole', assign local council. Blocked on '2021 roadworks funding increase', backlogged. Assigned to councillor. Won't fix. Ok - maybe I'm not making it sound great, but at least you could see some reasoning, and what the blockers are. Follow the chain to work out that 'communal lobby needs repainting' hasn't been acted kn by building management company because, ultimately, of global supply chain disruption and the contractor's supplier's supplier can't get any paint ingredients.

kall 5y ago

Your intuition is right on pulumi. Creating these kinds of extensions is minimal effort. You can start out by adding a "dynamic resource" class to your infra codebase and extract it into a plugin later, or not. These are not the same as "real" pulumi providers that run across all supported programming languages, but I think they are a good enough fit for the cases you mention.