4 ms·

I've worked on 2 companies where I was in a team of 4/5 devs + 1 SRE in charge of the infra. In both cases, once the SRE guy created the appropriate terraform m

I've worked on 2 companies where I was in a team of 4/5 devs + 1 SRE in charge of the infra. In both cases, once the SRE guy created the appropriate terraform modules and CI template, he had to move on to a new team because there was literally no work left for him to do. Once things are infra as a code and the happy path is defined and implemented as a process, we rarely lost time thinking about infrastructure. Or anything. The system needs permission for AWS S3? Here is the component. Does it need to setup secrets? Here is the component. Does it needs to be in the same VPC as a Redshift cluster, or whatnot? There's a component for that. Do you need a SQS/RDS/Redis service? You get it. K8s, terraform and Atlantis have been such huge enablers I cannot overstate. Everything is so nimble and cheaply setup that our current infra bill is about 500USD/month to generate over 100k MRR for the company. And better yet, we rarely ever think of our infra unless we want to do something brand new (such as testing Sagemaker), or there's unexpected traffic above our current scaling limits (about once a year). Plus, if we need to fix anything infra related there's an internal cookbook for that. I also feel I have acquired useful skills by learning all of this.

steveBK123 4y ago
That sounds like a smaller shop maybe. The problem is big, established shops, with dozens of teams of dozens of people.. 300 developers, etc.. think they can get all the above with 2 CloudOps guys. Instead zero happy paths get defined in a timely manner, everything you are trying to do is hamstrung and waiting in the CloudOps queue, and you can't self-serve around them because you cannot do it yourself due to permissions. Imagine an org where "the happy path for RDS will be ready in 2-3 months, and only support one database type and 3 instance sizes" is somehow acceptable. Rinse & repeat for an alphabet soup of 25 other AWS services.
- dpkirchner 4y ago
  That's more a problem with bad organizations than cloud or kubernetes. Besides, a common complaint about ops (called IT back then) was how long it took to provision and deploy resources. Months was common, unless the org had the foresight and budget to have lots of cold hardware on standby.
  - steveBK123 4y ago
    Yeah, the challenge is that after decades many of these firms actually had their onprem solutions dialed in. VMs can exist in the datacenter as well, and by the year 2022 most places have this stuff unbooked & behind a ticketing system with a defined SLA. Unfortunately when these slow moving medium/big firms then move into the cloud, they re-invent all the opacity, inertia, and other issues that they had solved onprem already. Some think they can just replace 25 DC ops people with 2 cloudOps and somehow its going to all just work out.
hbrn 4y ago
> Once things are infra as a code and the happy path is defined and implemented as a process, we rarely lost time thinking about infrastructure. > we rarely ever think of our infra unless we want to do something brand new That just means your infra is stable. It would be exactly the same without terraform/k8s.

steveBK123 4y ago

That sounds like a smaller shop maybe. The problem is big, established shops, with dozens of teams of dozens of people.. 300 developers, etc.. think they can get all the above with 2 CloudOps guys. Instead zero happy paths get defined in a timely manner, everything you are trying to do is hamstrung and waiting in the CloudOps queue, and you can't self-serve around them because you cannot do it yourself due to permissions. Imagine an org where "the happy path for RDS will be ready in 2-3 months, and only support one database type and 3 instance sizes" is somehow acceptable. Rinse & repeat for an alphabet soup of 25 other AWS services.

dpkirchner 4y ago
That's more a problem with bad organizations than cloud or kubernetes. Besides, a common complaint about ops (called IT back then) was how long it took to provision and deploy resources. Months was common, unless the org had the foresight and budget to have lots of cold hardware on standby.
- steveBK123 4y ago
  Yeah, the challenge is that after decades many of these firms actually had their onprem solutions dialed in. VMs can exist in the datacenter as well, and by the year 2022 most places have this stuff unbooked & behind a ticketing system with a defined SLA. Unfortunately when these slow moving medium/big firms then move into the cloud, they re-invent all the opacity, inertia, and other issues that they had solved onprem already. Some think they can just replace 25 DC ops people with 2 cloudOps and somehow its going to all just work out.

hbrn 4y ago

> Once things are infra as a code and the happy path is defined and implemented as a process, we rarely lost time thinking about infrastructure. > we rarely ever think of our infra unless we want to do something brand new That just means your infra is stable. It would be exactly the same without terraform/k8s.