4 ms·
I spent about 6 months using helm and made around 20+ charts for the services. In the end we got rid of it and replaced it with Terraform. If your infrastructu
by cbushko 7y ago
I spent about 6 months using helm and made around 20+ charts for the services.
In the end we got rid of it and replaced it with Terraform. If your infrastructure is 100% kubernetes then I think helm is great. Our infrastructure is not. We have databases, dns, buckets, service accounts and more so we were splitting our setup between terraform and helm. Passing data between the two tools was going to be a pain. We follow a layered approach of building up the infrastructure.
1) Networking: DNS
2) secrets, service accounts, buckets
3) DBs
4) Pre-application config (istio)
5) Services
Semi-related things are together and all of those cloud provider values we need are saved as secrets. We are on GCP so that means we need things like service accounts to access GCP resources (buckets, cloudsql) and all of those variables are available to our services to pick up.
And Terraform has STATE. This is unbelievably valuable when doing continuous delivery as you can tell what changed on every deploy and deploys are FAST. One thing that really bugged me about helm was that determining if a deploy failed was a post helm event. We were going to have to write monitoring for service health/uptime on deploy. This is not hard at all but you get it for free with terraform. If a service failed to start, terraform will throw an error...
I don't think people know that Terraform has a kubernetes provider. It does not support all the alpha objects but has decent support for 99% of the things you need. I wish someone made a provider for istio virtual services and service entries.
- MuffinFlavored 7y agoWhy don't more people use Terraform? I think Terraform is amazing. I wanted to make containerpen.io for docker-compose + dockerfiles + terraform files to compete with codepen.io
- nojvek 7y agoMy experience with Terraform was suboptimal. At least with GCS, it would do weird things and mess up permissions. I found the documentation and it’s special syntax a pain. Just give me json with json schemas defines somewhere so vscode can give me great code completion and I can templatize as necessary
- MuffinFlavored 7y ago> Just give me json with json schemas defines somewhere so vscode can give me great code completion and I can templatize as necessary Is there something that does this in the wild?
- 616c 7y agoBecause the DSL is complicated and you encounter weird edge cases even for not so complicated things. Common, almost ubiquitous in my circles, and often reliable are things I would say about it, painless is not. I was a bigger fan until using it daily focused on a project for a month on it more than anything else. I feel this strongly enough I'm looking at AWS CDK and troposphere, likely to learn I suck at writing my own imperative style terraform and to be more thankful. Lol
- xref 7y agoCurious about what is deploying your services, are they still containerized or are you using VMs? Any ansible along with it?
- cbushko 7y agoAll of our services are containerized and running in kubernetes.
- alsoh 7y agoWe bootstrap tf with a short go script then use terraform for until the cluster runs. Then argocd takes over. I really don't like how TF holds it state and it is errorpron. Allone how you have tto keep tf version in sync feels wrong. The k8s support did not take us far. We had some kubectl script and it doesn't feel like a first class citizen. I wouldn't bet on tf for k8s. I do love argocd. Great product.
- cbushko 7y agoI do find that state management is a bit of a pain which is why I have the layered approach listed above. By keeping components/modules separate I am able to update things without worrying about breaking others. Each namespace has it's own state while the kube cluster has it's own state too. I find state to be invaluable when a deploy of anything fails.