5 ms·

Getting a lot of docker hub vibes from this one. HashiCorp is course within their rights. Can't be cheap to run the registry given the obscene size of some terr

Getting a lot of docker hub vibes from this one. HashiCorp is course within their rights. Can't be cheap to run the registry given the obscene size of some terraform providers. $ ls -lah terraform/providers/registry.terraform.io/hashicorp/aws/5.14.0/darwin_amd64/ total 368M Anyone have an idea of the reasons terraform needs a 370 MiB binary just to call REST APIs?

mschuster91 3y ago
> Anyone have an idea of the reasons terraform needs a 370 MiB binary just to call REST APIs? That's because Terraform fell for the Go trap. When space and bandwidth are cheap, why not go for an environment that only ships fully self contained binaries? Oh, and why not go for a language that attracts hipsters like fruits attract flies, but is a nightmare to develop in? Bloody ridiculous, it's a miracle Terraform got as far as it did. (Yes, I'm working with Terraform every day and it's pretty decent but I'd love to extend it for Atlassian Cloud stuff without having to add a sixth language to my already sized toolbelt. Why Atlassian doesn't offer Terraform integration on their own is beyond me in any case)
paulddraper 3y ago
https://github.com/hashicorp/terraform-provider-aws/issues/31345 https://github.com/hashicorp/terraform-provider-aws/issues/3... The size is what you get when you add every single AWS Go client into one binary. Each service client like 1-2MB. But when you have 200 services....
skywhopper 3y ago
The AWS Go SDK is the vast majority of that bulk. In general go binaries can get pretty big but AWS has hundreds of services with thousands of APIs and it’s all going to have to get included in the AWS provider.
- candiddevmike 3y ago
  The AWS Go SDK is actually a case study on making everything into pointers
  - jen20 3y ago
    It’s really a case in the abject inadequacy of the Go type system.
- viraptor 3y ago
  Also AWS has so many services that the SDKs are mainly generated from json descriptions + nice wrappers on top. That leads to a different and less abstracted type of code than you'd write yourself - which leads to bigger compiled objects. Ruby had this problem too and at some point split the SDK into multiple gems so you don't have to install everything.
  - Too 3y ago
    The Azure sdks are the same. Auto generated from some underlying description. Then for backwards compatibility, every previous version is its own complete copy, all included in one single bundle.
chippiewill 3y ago
I think the providers might actually come from GitHub because the registry insists you have them as release assets
- sigwinch28 3y ago
  That indeed seems to be the case. Seems that for community providers, HashiCorp is serving up JSON which refers to GitHub download links. Here’s a sample size of 1: https://registry.terraform.io/v1/providers/spacelift-io/spacelift/1.2.0/download/darwin/arm64 https://registry.terraform.io/v1/providers/spacelift-io/spac... According to the provider registry protocol, which I have previously implemented for internal hosting (in an afternoon of writing a single file of Python): https://developer.hashicorp.com/terraform/internals/provider-registry-protocol https://developer.hashicorp.com/terraform/internals/provider...
  - fishnchips 3y ago
    For their own managed providers they no longer provide binaries in GitHub releases, and serve them from their own servers instead. Which feels like a trap BTW.
    - mdaniel 3y ago
      I didn't realize that, and extra weird they publish the SHA256SUM file as a release artifact but it references 14 zip files and the manifest.json so, ... thanks? But, in their defense, installing an "unofficial" provider (or build!) into TF is some JFC so there's that. We'll just add that onto the damn near infinite pile of "I hope OpenTF fixes ..." things
jabiko 3y ago
> Can't be cheap to run the registry given the obscene size of some terraform providers. Some providers are also hosted externally. I guess if traffic is going to be a problem they might also just switch to hosting every provider that is build on GitHub to GitHub releases (and hope that GitHub won't change its policy) curl -s https://registry.terraform.io/v1/providers/carlpett/sops/0.7.2/download/linux/amd64 | jq .download_url "https://github.com/carlpett/terraform-provider-sops/releases/download/v0.7.2/terraform-provider-sops_0.7.2_linux_amd64.zip" curl -s https://registry.terraform.io/v1/providers/hashicorp/helm/2.10.1/download/linux/amd64 | jq .download_url "https://releases.hashicorp.com/terraform-provider-helm/2.10.1/terraform-provider-helm_2.10.1_linux_amd64.zip"

mschuster91 3y ago

> Anyone have an idea of the reasons terraform needs a 370 MiB binary just to call REST APIs? That's because Terraform fell for the Go trap. When space and bandwidth are cheap, why not go for an environment that only ships fully self contained binaries? Oh, and why not go for a language that attracts hipsters like fruits attract flies, but is a nightmare to develop in? Bloody ridiculous, it's a miracle Terraform got as far as it did. (Yes, I'm working with Terraform every day and it's pretty decent but I'd love to extend it for Atlassian Cloud stuff without having to add a sixth language to my already sized toolbelt. Why Atlassian doesn't offer Terraform integration on their own is beyond me in any case)

paulddraper 3y ago

https://github.com/hashicorp/terraform-provider-aws/issues/31345 https://github.com/hashicorp/terraform-provider-aws/issues/3... The size is what you get when you add every single AWS Go client into one binary. Each service client like 1-2MB. But when you have 200 services....

skywhopper 3y ago

The AWS Go SDK is the vast majority of that bulk. In general go binaries can get pretty big but AWS has hundreds of services with thousands of APIs and it’s all going to have to get included in the AWS provider.

candiddevmike 3y ago
The AWS Go SDK is actually a case study on making everything into pointers
- jen20 3y ago
  It’s really a case in the abject inadequacy of the Go type system.
viraptor 3y ago
Also AWS has so many services that the SDKs are mainly generated from json descriptions + nice wrappers on top. That leads to a different and less abstracted type of code than you'd write yourself - which leads to bigger compiled objects. Ruby had this problem too and at some point split the SDK into multiple gems so you don't have to install everything.
- Too 3y ago
  The Azure sdks are the same. Auto generated from some underlying description. Then for backwards compatibility, every previous version is its own complete copy, all included in one single bundle.

chippiewill 3y ago

I think the providers might actually come from GitHub because the registry insists you have them as release assets

sigwinch28 3y ago
That indeed seems to be the case. Seems that for community providers, HashiCorp is serving up JSON which refers to GitHub download links. Here’s a sample size of 1: https://registry.terraform.io/v1/providers/spacelift-io/spacelift/1.2.0/download/darwin/arm64 https://registry.terraform.io/v1/providers/spacelift-io/spac... According to the provider registry protocol, which I have previously implemented for internal hosting (in an afternoon of writing a single file of Python): https://developer.hashicorp.com/terraform/internals/provider-registry-protocol https://developer.hashicorp.com/terraform/internals/provider...
- fishnchips 3y ago
  For their own managed providers they no longer provide binaries in GitHub releases, and serve them from their own servers instead. Which feels like a trap BTW.
  - mdaniel 3y ago
    I didn't realize that, and extra weird they publish the SHA256SUM file as a release artifact but it references 14 zip files and the manifest.json so, ... thanks? But, in their defense, installing an "unofficial" provider (or build!) into TF is some JFC so there's that. We'll just add that onto the damn near infinite pile of "I hope OpenTF fixes ..." things

jabiko 3y ago

> Can't be cheap to run the registry given the obscene size of some terraform providers. Some providers are also hosted externally. I guess if traffic is going to be a problem they might also just switch to hosting every provider that is build on GitHub to GitHub releases (and hope that GitHub won't change its policy) curl -s https://registry.terraform.io/v1/providers/carlpett/sops/0.7.2/download/linux/amd64 | jq .download_url "https://github.com/carlpett/terraform-provider-sops/releases/download/v0.7.2/terraform-provider-sops_0.7.2_linux_amd64.zip" curl -s https://registry.terraform.io/v1/providers/hashicorp/helm/2.10.1/download/linux/amd64 | jq .download_url "https://releases.hashicorp.com/terraform-provider-helm/2.10.1/terraform-provider-helm_2.10.1_linux_amd64.zip"