6 ms·

I stopped arguing about which is better CF or TF. Some people go crazy every time this topic is discussed. Me, I just hate all this IAC thing. I spend weeks so

I stopped arguing about which is better CF or TF. Some people go crazy every time this topic is discussed. Me, I just hate all this IAC thing. I spend weeks sometimes to deliver a script that deploys an rds database with blah blah blah. Yeah it is nice to have all your infrastructure as code. It would be definitely useful if you are deploying the same structure again and again. or if you are running it frequently. But if you are deploying a structure once only, do you really need this? Do you have to spend 2 weeks on a script you will re-run once in 6 months? (and then you will find that many objects need to be updated to re-run without errors) It seems a lot of over-engineering for me (I know I will be hated so much for this). Sometimes when I run TF in production, I start praying that all goes well!

trhway 5y ago
>I spend weeks sometimes to deliver a script that deploys an rds database with blah blah blah and this is $500K+/year in SV. What other area pays that well for such little delivery? Probably only AI :)
cliffy 5y ago
So... what happens when the person who originally deployed everything eventually leaves or gets hit by a bus? Hey, it's six months later and you need to redeploy everything again. You need some automation or a really good how-to document, and hopefully both. It won't save you all the pain of learning how the deployment works but it will save you a lot of time and effort.
busterarm 5y ago
This is why companies like Gruntwork have pre-written production-ready Terraform modules for just about any service/provider you would need. I have my own set of modules that I've written and use in my consulting work. It's repeatable work that I can have tailored to my clients in under an hour. Terraform puts dollars in my pocket.
- raffraffraff 5y ago
  This. I haven't done that yet, but I'm thinking about going that route. My last job was with a start-up. I got to build everything from scratch, and I was 100% focused on infrastructure. I promised myself that everything new thing the team need would be deployed for them manually, then thoroughly researched and redeployed via IaaC asap. That meant that I was always delivering work that the team need. It also meant that I could take their request and build a really useful module from it. Example: nobody ever "just wanted" an SQS queue. They wanted a queue, various different sized workers in auto-scaling group with metrics to trigger scaling events. And logs. And probably an s3 bucket. And a deadletter queue. And all the security policies to allow those things to communicate with each other safely. Oh and custom metrics with alarms that notify you when your queries fill up. So I would build a single module that did all that stuff, using sensible and well-documented defaults. It was executed via for loop that iterated over a yaml structure. When the dev team wanted a new queue+workers, they edited the yaml (Hiera, in this case), added a few lines to the 'sqs-worker-groups:' block, and sent me a merge request. I would make sure they read the module readme, ask a few questions to make sure the foot-gun want cocked, and merge I know some people have said that the developers in their company write their own terraform. That's really neat, but do you get solutions like this, where a single person is focused on writing the DRYest possible code that can be re-used by everyone? When devs write infrastructure-as-code, there are also soooo many foot-guns that they all have to learn about over and over. Things that a seasoned infra engineer will have shot themselves with once and never again (s3 permissions, IOPS, egress costs, using us-east-1 etc). How many developers would bother studying the AWS infra-focused exam material?
  - nickjj 5y ago
    I am a developer who migrated more towards doing ops work. I still enjoy writing app code but most of the work I do now is ops related. That means setting up infrastructure and pipelines for companies to help them ship things quicker and safer. I think for most organizations having a dedicated ops person is worth it. Chances are developers are going to be developing app features. They won't have time to dedicate an entire day on why the nginx ingress for Kubernetes won't allocate an external IP address when you're using KinD locally with the nginx ingress' Helm chart default arguments. Basically, there's always something to work on. Either infrastructure related improvements or helping developers do things better and faster in development by writing custom scripts to improve their workflows.
not_kurt_godel 5y ago
> But if you are deploying a structure once only, do you really need this? Yes. Because "once" is never just once if you're doing something remotely useful. At a bare minimum you need to pull updates for your upstream dependencies and redeploy once in a while, even if you're doing zero extension or replication of actual functionality. > Do you have to spend 2 weeks on a script you will re-run once in 6 months? It does not take 2 weeks to write the vast majority of IAC. And even if does - at least you've saved the next person who needs to update it a world of complications of having to know how to manually curate the system.
- shukantpal 5y ago
  Non sequitur - you're putting words in the parent post's mouth. When they say "once", they're talking about "once". Arrogant to claim otherwise.
  - nostrebored 5y ago
    They're not though. Theu talk about deploying multiple times. They also talk about problems with drift which means the infrastructure is probably changing outside of IaC. The problem is the process at ops company not IaC.
nkotov 5y ago
> But if you are deploying a structure once only, do you really need this? Exact reason why I'm building my company. I worked on several teams now where developers would spend weeks figuring out how to get their apps to run in AWS instead of releasing features. At one company, we solved the issue by letting the devs use Terraform modules - but ended up with a different issue: devs would just copy/paste code without fully understanding why they are spinning the infra in that way. It's my honest opinion that developers should not be forced to become cloud experts.
viraptor 5y ago
> I spend weeks sometimes to deliver a script that deploys an rds database with blah blah blah It can take a while if you're doing it the first time. Once you get used to the CF ideas it's a couple hours max. But keep in mind how that compares to: setting it up by hand, documenting the process so others know it, tagging the relevant services according to project, redoing the process in a while when RDS deprecates your currently running version, redoing it again when you need to change the db instance sizes.
chousuke 5y ago
IaC is faster than doing things manually. If you're spending weeks on it, that sounds to me like you either haven't yet practiced enough or you're working in an environment that's already a mess. Once you have a bit of practice, using IaC tools is just better in every way compared to creating resources manually. It's faster, it's more consistent and robust, serves as emergency documentation and allows easy and usable change tracking and management because you can put the code in a git repository. If your Terraform manifests are going out of sync, that means your team is not actually using it and the manual, untrackable change is messing with you. I wrote the initial set of manifests for one setup (dozens of components and 3 identical environments) about 5 years ago. Since then, a different team has been expanding and managing the system and I really have no idea of everything that's there nowadays, but I do still occasionally need to go in and make changes and I can do that confidently because they kept using Terraform and I can just read the code to know what's new. I did make some mistakes (choosing Chef for CM, it wasn't a good fit in retrospect), but aggressively automating everything was not one of them. I also have one environment that I wrote which consists of a VPC, an EIP, a single EC2 instance and S3 buckets and policies for cross-account backups. I also automated that, and it definitely resulted in a better result faster than doing it manually. Turns out, even setting up a single server to run a single piece of well-packaged software (plus a local database) has a lot you need to take into account if you want to do everything properly, including backups and monitoring. On top of it, testing my backups was easy because I could just delete the whole environment and run my automation again to rebuild it, so I know that as long as there's a backup in a bucket somewhere, the system can be recovered in minutes.
lbriner 5y ago
One thing I can't see mentioned is disaster recovery. We expect the cloud to be largely bulletproof but that has been proven wrong, we could easily lose something for multiple hours, which for a SaaS business is very bad for business. When Azure London went down for 6 hours, or whatever, we should have been able to recreate our microservices cluster in another centre but we simply didn't have it setup like that and had to live with the outage. If we had to recreate any of our cloud system manually, it would take an enormous amount of time and would probably not work properly since we forgot those registry settings/frameworks/configurations that we did 2 years ago. Part of the point of IaC is the ability to recreate all of most of what we need at the click of a button. Even if you had everything documented, the physical time would be prohibitive.

trhway 5y ago

>I spend weeks sometimes to deliver a script that deploys an rds database with blah blah blah and this is $500K+/year in SV. What other area pays that well for such little delivery? Probably only AI :)

cliffy 5y ago

So... what happens when the person who originally deployed everything eventually leaves or gets hit by a bus? Hey, it's six months later and you need to redeploy everything again. You need some automation or a really good how-to document, and hopefully both. It won't save you all the pain of learning how the deployment works but it will save you a lot of time and effort.

busterarm 5y ago

This is why companies like Gruntwork have pre-written production-ready Terraform modules for just about any service/provider you would need. I have my own set of modules that I've written and use in my consulting work. It's repeatable work that I can have tailored to my clients in under an hour. Terraform puts dollars in my pocket.

raffraffraff 5y ago
This. I haven't done that yet, but I'm thinking about going that route. My last job was with a start-up. I got to build everything from scratch, and I was 100% focused on infrastructure. I promised myself that everything new thing the team need would be deployed for them manually, then thoroughly researched and redeployed via IaaC asap. That meant that I was always delivering work that the team need. It also meant that I could take their request and build a really useful module from it. Example: nobody ever "just wanted" an SQS queue. They wanted a queue, various different sized workers in auto-scaling group with metrics to trigger scaling events. And logs. And probably an s3 bucket. And a deadletter queue. And all the security policies to allow those things to communicate with each other safely. Oh and custom metrics with alarms that notify you when your queries fill up. So I would build a single module that did all that stuff, using sensible and well-documented defaults. It was executed via for loop that iterated over a yaml structure. When the dev team wanted a new queue+workers, they edited the yaml (Hiera, in this case), added a few lines to the 'sqs-worker-groups:' block, and sent me a merge request. I would make sure they read the module readme, ask a few questions to make sure the foot-gun want cocked, and merge I know some people have said that the developers in their company write their own terraform. That's really neat, but do you get solutions like this, where a single person is focused on writing the DRYest possible code that can be re-used by everyone? When devs write infrastructure-as-code, there are also soooo many foot-guns that they all have to learn about over and over. Things that a seasoned infra engineer will have shot themselves with once and never again (s3 permissions, IOPS, egress costs, using us-east-1 etc). How many developers would bother studying the AWS infra-focused exam material?
- nickjj 5y ago
  I am a developer who migrated more towards doing ops work. I still enjoy writing app code but most of the work I do now is ops related. That means setting up infrastructure and pipelines for companies to help them ship things quicker and safer. I think for most organizations having a dedicated ops person is worth it. Chances are developers are going to be developing app features. They won't have time to dedicate an entire day on why the nginx ingress for Kubernetes won't allocate an external IP address when you're using KinD locally with the nginx ingress' Helm chart default arguments. Basically, there's always something to work on. Either infrastructure related improvements or helping developers do things better and faster in development by writing custom scripts to improve their workflows.

not_kurt_godel 5y ago

> But if you are deploying a structure once only, do you really need this? Yes. Because "once" is never just once if you're doing something remotely useful. At a bare minimum you need to pull updates for your upstream dependencies and redeploy once in a while, even if you're doing zero extension or replication of actual functionality. > Do you have to spend 2 weeks on a script you will re-run once in 6 months? It does not take 2 weeks to write the vast majority of IAC. And even if does - at least you've saved the next person who needs to update it a world of complications of having to know how to manually curate the system.

shukantpal 5y ago
Non sequitur - you're putting words in the parent post's mouth. When they say "once", they're talking about "once". Arrogant to claim otherwise.
- nostrebored 5y ago
  They're not though. Theu talk about deploying multiple times. They also talk about problems with drift which means the infrastructure is probably changing outside of IaC. The problem is the process at ops company not IaC.

nkotov 5y ago

> But if you are deploying a structure once only, do you really need this? Exact reason why I'm building my company. I worked on several teams now where developers would spend weeks figuring out how to get their apps to run in AWS instead of releasing features. At one company, we solved the issue by letting the devs use Terraform modules - but ended up with a different issue: devs would just copy/paste code without fully understanding why they are spinning the infra in that way. It's my honest opinion that developers should not be forced to become cloud experts.

viraptor 5y ago

> I spend weeks sometimes to deliver a script that deploys an rds database with blah blah blah It can take a while if you're doing it the first time. Once you get used to the CF ideas it's a couple hours max. But keep in mind how that compares to: setting it up by hand, documenting the process so others know it, tagging the relevant services according to project, redoing the process in a while when RDS deprecates your currently running version, redoing it again when you need to change the db instance sizes.

chousuke 5y ago

IaC is faster than doing things manually. If you're spending weeks on it, that sounds to me like you either haven't yet practiced enough or you're working in an environment that's already a mess. Once you have a bit of practice, using IaC tools is just better in every way compared to creating resources manually. It's faster, it's more consistent and robust, serves as emergency documentation and allows easy and usable change tracking and management because you can put the code in a git repository. If your Terraform manifests are going out of sync, that means your team is not actually using it and the manual, untrackable change is messing with you. I wrote the initial set of manifests for one setup (dozens of components and 3 identical environments) about 5 years ago. Since then, a different team has been expanding and managing the system and I really have no idea of everything that's there nowadays, but I do still occasionally need to go in and make changes and I can do that confidently because they kept using Terraform and I can just read the code to know what's new. I did make some mistakes (choosing Chef for CM, it wasn't a good fit in retrospect), but aggressively automating everything was not one of them. I also have one environment that I wrote which consists of a VPC, an EIP, a single EC2 instance and S3 buckets and policies for cross-account backups. I also automated that, and it definitely resulted in a better result faster than doing it manually. Turns out, even setting up a single server to run a single piece of well-packaged software (plus a local database) has a lot you need to take into account if you want to do everything properly, including backups and monitoring. On top of it, testing my backups was easy because I could just delete the whole environment and run my automation again to rebuild it, so I know that as long as there's a backup in a bucket somewhere, the system can be recovered in minutes.

lbriner 5y ago

One thing I can't see mentioned is disaster recovery. We expect the cloud to be largely bulletproof but that has been proven wrong, we could easily lose something for multiple hours, which for a SaaS business is very bad for business. When Azure London went down for 6 hours, or whatever, we should have been able to recreate our microservices cluster in another centre but we simply didn't have it setup like that and had to live with the outage. If we had to recreate any of our cloud system manually, it would take an enormous amount of time and would probably not work properly since we forgot those registry settings/frameworks/configurations that we did 2 years ago. Part of the point of IaC is the ability to recreate all of most of what we need at the click of a button. Even if you had everything documented, the physical time would be prohibitive.