6 ms·

Terraform Module Registry

110 points by ApsOps 9y ago

atonse 9y ago
Very excited to see this (if it is what I think it is...). Still reading about it. I really like Terraform but haven't been a fan of the existing way of bringing external code into my terraform scripts, via some strange go-like "refer to this github" repo mechanism. Hopefully this makes them package able artifacts. Looking forward to watching the keynote and other videos from Hashiconf.
- jfroma 9y ago
  I'd really love to be able to build modules in some other language like python, ruby or even javascript just because I am not very fluent yet in Go and its tooling.
  - charlieegan3 9y ago
    The modules are written in HashiCorp configuration language.
    - lmickh 9y ago
      You can use json as well, but it is pretty unfriendly for the human users and you lose some of the interpolation.
      - meddlepal 9y ago
        However, very useful if you're code generating Terraform config because there are not a lot of HCL serializer/deserializer libs in existence.
    - jfroma 9y ago
      You are right, I should have say "providers" with new "resources" and yes, this new registry is for modules, not providers.
  - jen20 9y ago
    If you want to build custom Terraform providers _without_ using Go, this is actually on the cards now: as of `hashicorp/go-plugin#fa4a2a351`, plugins can use gRPC to communicate with the core of Terraform. I don't know if anyone has implemented the primitives for building providers in another language, but it's now possible without too much pain. [1]: https://github.com/hashicorp/go-plugin/commit/fa4a2a351f33b106d320ffadd69184504e040384 https://github.com/hashicorp/go-plugin/commit/fa4a2a351f33b1...
eropple 9y ago
We already see with other "use this as a building block" public registries like Docker's where people blithely pull this stuff in and don't pay attention to whether or not it actually works according to even some semblance of best practices. So what's the handler for the "hey, this is broken/insecure, this needs to be fixed or taken down" case? How does this alert users whose infrastructure pulls this in? HashiCorp, as some of the leaders of the Make Decisions On Five Minute Demos #squad, usually doesn't answer even half the questions for their Bright Ideas before people adopt them and find all the rakes on the lawn. This feels even worse than usual. Building strong foundations matters. This doesn't help somebody who knows what they're doing do it better/faster--because of the inevitable impedance mismatches, in part because folks do this stuff differently and in part because Terraform is not particularly expressive and workarounds will involve destructive manipulation of modules--and it harms people who don't.
- deleted 9y ago
  [deleted]
- DandyDev 9y ago
  How is this any different from the various packaging mechanisms that exist for all the different programming languages out there? You make it sound like this is a new kind of problem that started with Docker. When I add some dependency to the Maven pom of my legacy Java EE app, I'm exposed to exactly the same kind of risk. In short: research any dependency you pull in, whether it is a library for your favorite language or a module for your infrastructure automation tool of choice.
  - deleted 9y ago
    [deleted]
  - eropple 9y ago
    It's a difference-of-kind problem when you're working on infrastructure. You unwind a bad dependency in your app by removing the dependency. You unwind a bad dependency in your infrastructure by destroying large whacks of it.
  - 9y ago
crummy_bum 9y ago
This looks like a pretty frontend for the terraform-community-modules github org - https://github.com/terraform-community-modules https://github.com/terraform-community-modules The main problem with these off the shelf modules is the lack of standards around state management and reuse.
tjbiddle 9y ago
I'm getting an error stating `error reading user information` upon authentication with Github.
sandstrom 9y ago
hashicorp-people: do you have any plans to open-source sentinel? It seems like something that could have broader use, outside the hashicorp-suite. For example, many SaaS-applications (say, a system used by some bank) usually end up writing their home-grown DSL for auth or use a language library. IAM also has something similar, with JSON payloads (but that's not code). Would be interesting to know more about how sentinel is implemented under the hood.
Tehchops 9y ago
I find this pretty exciting. I imagine the most compelling use case for most orgs will be hosting their own private registry using Terraform Enterprise.
nvivo 9y ago
What is terraform? Read the page, but didn't get exactly what it does.
- ellius 9y ago
  Imagine you want to create some new machines. Maybe you need four servers: 2 running Ubuntu operating systems and 2 running Centos. Traditinally, you would get four physical servers together and then manually load an operating system onto each one. With the advent of cloud technology, you can simply go to Amazon or DigitalOcean or whoever and just "create" these resources by paying for space in their data centers. You get four "virtual servers" instead of physical boxes you have to configure yourself. Terraform essentially lets you "code" and automate the cloud provisioning. Instead of going to a dashboard and ordering four servers, you run a script that describes all of the servers you need as well as their operating system / networking / etc. needs. This scales very well (once you have a config you can replicate it to whatever quantity you require) and also provides consistency and reliability, as well as documentation, of your infrastructure.
- andy_ppp 9y ago
  Infrastructure as code. Terraform provisions servers and infrastructure based on yaml config files that you can do things like commit to source control and see a diff of changes to your cloud provider. It's really fun but very tied into each specific service providers' nomenclature and services, I wish it was higher level. It provides backends for all the major players but I think there is (was?) some large variation in quality. Wonderful talk here explaining with real examples: http://www.ybrikman.com/writing/2016/03/31/infrastructure-as-code-microservices-aws-docker-terraform-ecs/ http://www.ybrikman.com/writing/2016/03/31/infrastructure-as...
odammit 9y ago
This is absolutely useful. I keep a bunch of half-assed modules in a repo that I find common across projects. Love the idea of being able to find modules to kickstart common infrastructure. Also Terraform is a great tool and the maintainers and contributors have done a fantastic job putting in useful features in the last few versions. Thanks!
zimbatm 9y ago
Interesting, they are adopting the same pattern that I had discovered for AWS modules: * all modules take in a var.name and var.tags variables. * if the resource has a unique identifier, use var.name as a prefix (eg: "${var.name}-elb") * all resources are tagged with the Name merged into var.tags This allows to (1) avoid name clashes with the name prefix and (2) make sure everything is tagged and easier to filter by on AWS, while deferring most of the tagging decision to the user. See https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/master/main.tf#L10 https://github.com/terraform-aws-modules/terraform-aws-vpc/b... EDIT: there is still an issue with autoscaling groups who take a list of tags instead of a map. https://github.com/terraform-aws-modules/terraform-aws-autoscaling/blob/master/modules/autoscaling_group/main.tf#L30-L33 https://github.com/terraform-aws-modules/terraform-aws-autos...

110 points by ApsOps 9y ago

Very excited to see this (if it is what I think it is...). Still reading about it. I really like Terraform but haven't been a fan of the existing way of bringing external code into my terraform scripts, via some strange go-like "refer to this github" repo mechanism. Hopefully this makes them package able artifacts. Looking forward to watching the keynote and other videos from Hashiconf.

jfroma 9y ago
I'd really love to be able to build modules in some other language like python, ruby or even javascript just because I am not very fluent yet in Go and its tooling.
- charlieegan3 9y ago
  The modules are written in HashiCorp configuration language.
  - lmickh 9y ago
    You can use json as well, but it is pretty unfriendly for the human users and you lose some of the interpolation.
    - meddlepal 9y ago
      However, very useful if you're code generating Terraform config because there are not a lot of HCL serializer/deserializer libs in existence.
  - jfroma 9y ago
    You are right, I should have say "providers" with new "resources" and yes, this new registry is for modules, not providers.
- jen20 9y ago
  If you want to build custom Terraform providers _without_ using Go, this is actually on the cards now: as of `hashicorp/go-plugin#fa4a2a351`, plugins can use gRPC to communicate with the core of Terraform. I don't know if anyone has implemented the primitives for building providers in another language, but it's now possible without too much pain. [1]: https://github.com/hashicorp/go-plugin/commit/fa4a2a351f33b106d320ffadd69184504e040384 https://github.com/hashicorp/go-plugin/commit/fa4a2a351f33b1...

eropple 9y ago

We already see with other "use this as a building block" public registries like Docker's where people blithely pull this stuff in and don't pay attention to whether or not it actually works according to even some semblance of best practices. So what's the handler for the "hey, this is broken/insecure, this needs to be fixed or taken down" case? How does this alert users whose infrastructure pulls this in? HashiCorp, as some of the leaders of the Make Decisions On Five Minute Demos #squad, usually doesn't answer even half the questions for their Bright Ideas before people adopt them and find all the rakes on the lawn. This feels even worse than usual. Building strong foundations matters. This doesn't help somebody who knows what they're doing do it better/faster--because of the inevitable impedance mismatches, in part because folks do this stuff differently and in part because Terraform is not particularly expressive and workarounds will involve destructive manipulation of modules--and it harms people who don't.

deleted 9y ago
[deleted]
DandyDev 9y ago
How is this any different from the various packaging mechanisms that exist for all the different programming languages out there? You make it sound like this is a new kind of problem that started with Docker. When I add some dependency to the Maven pom of my legacy Java EE app, I'm exposed to exactly the same kind of risk. In short: research any dependency you pull in, whether it is a library for your favorite language or a module for your infrastructure automation tool of choice.
- deleted 9y ago
  [deleted]
- eropple 9y ago
  It's a difference-of-kind problem when you're working on infrastructure. You unwind a bad dependency in your app by removing the dependency. You unwind a bad dependency in your infrastructure by destroying large whacks of it.
- 9y ago

crummy_bum 9y ago

This looks like a pretty frontend for the terraform-community-modules github org - https://github.com/terraform-community-modules https://github.com/terraform-community-modules The main problem with these off the shelf modules is the lack of standards around state management and reuse.

tjbiddle 9y ago

I'm getting an error stating `error reading user information` upon authentication with Github.

sandstrom 9y ago

hashicorp-people: do you have any plans to open-source sentinel? It seems like something that could have broader use, outside the hashicorp-suite. For example, many SaaS-applications (say, a system used by some bank) usually end up writing their home-grown DSL for auth or use a language library. IAM also has something similar, with JSON payloads (but that's not code). Would be interesting to know more about how sentinel is implemented under the hood.

Tehchops 9y ago

I find this pretty exciting. I imagine the most compelling use case for most orgs will be hosting their own private registry using Terraform Enterprise.

nvivo 9y ago

What is terraform? Read the page, but didn't get exactly what it does.

ellius 9y ago
Imagine you want to create some new machines. Maybe you need four servers: 2 running Ubuntu operating systems and 2 running Centos. Traditinally, you would get four physical servers together and then manually load an operating system onto each one. With the advent of cloud technology, you can simply go to Amazon or DigitalOcean or whoever and just "create" these resources by paying for space in their data centers. You get four "virtual servers" instead of physical boxes you have to configure yourself. Terraform essentially lets you "code" and automate the cloud provisioning. Instead of going to a dashboard and ordering four servers, you run a script that describes all of the servers you need as well as their operating system / networking / etc. needs. This scales very well (once you have a config you can replicate it to whatever quantity you require) and also provides consistency and reliability, as well as documentation, of your infrastructure.
andy_ppp 9y ago
Infrastructure as code. Terraform provisions servers and infrastructure based on yaml config files that you can do things like commit to source control and see a diff of changes to your cloud provider. It's really fun but very tied into each specific service providers' nomenclature and services, I wish it was higher level. It provides backends for all the major players but I think there is (was?) some large variation in quality. Wonderful talk here explaining with real examples: http://www.ybrikman.com/writing/2016/03/31/infrastructure-as-code-microservices-aws-docker-terraform-ecs/ http://www.ybrikman.com/writing/2016/03/31/infrastructure-as...

odammit 9y ago

This is absolutely useful. I keep a bunch of half-assed modules in a repo that I find common across projects. Love the idea of being able to find modules to kickstart common infrastructure. Also Terraform is a great tool and the maintainers and contributors have done a fantastic job putting in useful features in the last few versions. Thanks!

zimbatm 9y ago

Interesting, they are adopting the same pattern that I had discovered for AWS modules: * all modules take in a var.name and var.tags variables. * if the resource has a unique identifier, use var.name as a prefix (eg: "${var.name}-elb") * all resources are tagged with the Name merged into var.tags This allows to (1) avoid name clashes with the name prefix and (2) make sure everything is tagged and easier to filter by on AWS, while deferring most of the tagging decision to the user. See https://github.com/terraform-aws-modules/terraform-aws-vpc/blob/master/main.tf#L10 https://github.com/terraform-aws-modules/terraform-aws-vpc/b... EDIT: there is still an issue with autoscaling groups who take a list of tags instead of a map. https://github.com/terraform-aws-modules/terraform-aws-autoscaling/blob/master/modules/autoscaling_group/main.tf#L30-L33 https://github.com/terraform-aws-modules/terraform-aws-autos...