3 ms·

Getting something up and running and managing something long-term are two different things. I'm sure - even though I'm not particularly smart - I could get a s

by danjac 4y ago

Getting something up and running and managing something long-term are two different things. I'm sure - even though I'm not particularly smart - I could get a simple k8s solution up and running in a week with the help of a YouTube video. What happens six months in when I hit some critical k8s related issue and I lack the basic understanding of where to even begin to debug and fix it? For example I may introduce a serious security issue in a CI pipeline or whatever just due to inexperience and unfamiliarity. I'm not saying Kubernetes (or for that matter, microservices) is the wrong pick for a small 3-person team. You know your own business better than I or anyone else and I'm sure you made the right choice. But personally I would be cautious about adopting a core production technology where my experience consists of one week and a YouTube tutorial.

steveBK123 4y ago
Yes. As long as your k8s is some self contained island of code needing no R/W access to resources on k8s, maybe I'll believe it's easy to setup AND to support. Additionally if you are in some type of small shop environment where you have basically unfettered permissions in AWS console to do what you need, doubly so. Now try to get access to/from RDS, S3, EFS, Netapp NFS, some HTTP endpoint on-prem, some other HTTP endpoint on EC2, non-MSK Kafka, cross account, etc? Pain. Oh you need to submit a ticket to your Linux sysadmins, Infra team, CloudOps, DevOps, Networking or Infosec for each of the above permissions? And sometimes need 2-3 of those teams to coordinate to make it happen? Death.
- thegginthesky 4y ago
  I've worked on 2 companies where I was in a team of 4/5 devs + 1 SRE in charge of the infra. In both cases, once the SRE guy created the appropriate terraform modules and CI template, he had to move on to a new team because there was literally no work left for him to do. Once things are infra as a code and the happy path is defined and implemented as a process, we rarely lost time thinking about infrastructure. Or anything. The system needs permission for AWS S3? Here is the component. Does it need to setup secrets? Here is the component. Does it needs to be in the same VPC as a Redshift cluster, or whatnot? There's a component for that. Do you need a SQS/RDS/Redis service? You get it. K8s, terraform and Atlantis have been such huge enablers I cannot overstate. Everything is so nimble and cheaply setup that our current infra bill is about 500USD/month to generate over 100k MRR for the company. And better yet, we rarely ever think of our infra unless we want to do something brand new (such as testing Sagemaker), or there's unexpected traffic above our current scaling limits (about once a year). Plus, if we need to fix anything infra related there's an internal cookbook for that. I also feel I have acquired useful skills by learning all of this.
  - steveBK123 4y ago
    That sounds like a smaller shop maybe. The problem is big, established shops, with dozens of teams of dozens of people.. 300 developers, etc.. think they can get all the above with 2 CloudOps guys. Instead zero happy paths get defined in a timely manner, everything you are trying to do is hamstrung and waiting in the CloudOps queue, and you can't self-serve around them because you cannot do it yourself due to permissions. Imagine an org where "the happy path for RDS will be ready in 2-3 months, and only support one database type and 3 instance sizes" is somehow acceptable. Rinse & repeat for an alphabet soup of 25 other AWS services.
    - dpkirchner 4y ago
      That's more a problem with bad organizations than cloud or kubernetes. Besides, a common complaint about ops (called IT back then) was how long it took to provision and deploy resources. Months was common, unless the org had the foresight and budget to have lots of cold hardware on standby.
      - steveBK123 4y ago
        Yeah, the challenge is that after decades many of these firms actually had their onprem solutions dialed in. VMs can exist in the datacenter as well, and by the year 2022 most places have this stuff unbooked & behind a ticketing system with a defined SLA. Unfortunately when these slow moving medium/big firms then move into the cloud, they re-invent all the opacity, inertia, and other issues that they had solved onprem already. Some think they can just replace 25 DC ops people with 2 cloudOps and somehow its going to all just work out.
  - hbrn 4y ago
    > Once things are infra as a code and the happy path is defined and implemented as a process, we rarely lost time thinking about infrastructure. > we rarely ever think of our infra unless we want to do something brand new That just means your infra is stable. It would be exactly the same without terraform/k8s.
- gloosx 4y ago
  As long as it's all in the same VPC everything goes through HTTP and other protocols like a knife thru butter, you don't even have to think about it knowing it is all in the same private network
deleted 4y ago
[deleted]
gloosx 4y ago
The point is – k8s essentially simplifies the solution, so what you'd get up and running will serve you for the time being without any maintenance. You're talking about it like it's a whole different framework of thinking which you will need to adopt from scratch, but it's not. Containerisation and virtualisation concepts are here since the beginning of time, just this thing makes it real easy to grasp, apart from that it is all same technology. It literally allows anyone to spin-up amazing production clusters quickly with zero overhead – assuming you know what you're spinning up, something that just took months of configuration works – now abstracted to few funny .yaml files.
- danjac 4y ago
  There's literally a web site of Kubernetes failure stories: https://k8s.af/ https://k8s.af/ where people experienced in Kubernetes have shot themselves in the foot. To be fair, it's over a year old, and probably the state of the art and best practices have advanced to the point where many of these are unlikely to happen. But in general, Kubernetes is widely considered a very big complex thing full of footguns for the unwary for a reason. So while I agree with you that sure, you can get up and running by watching some videos and copy-pasting a few YAML files, the problem is when you are deep into critical production and you run into issues (the classic "I get woken up at 3am by an alert or my boss and we need to fix this in 10 minutes") and you lack the deeper understanding of the framework to even know where to start. Or even worse, you insert an insidious security issue through a misconfiguration and you are not even aware you have been owned. Security is notoriously difficult to get right, even for people with experience in frameworks and systems, let alone a newbie. Kubernetes is not "secure by default" and again, there are footguns for the unwary: https://snyk.io/learn/kubernetes-security/ https://snyk.io/learn/kubernetes-security/ None of these things make Kubernetes bad, per se, and I'm sure the benefits outweigh the costs. But personally I would be cautious about adopting it for serious production without having some solid real-world experience first. There's a bigger question here as well, regarding small teams (as in startups, not just pizza teams in a tech giant). Where do you want your very limited time and focus to be? If I'm building some SAAS application I want to put my coding focus close to 100% towards the business model. If that model requires me learning new tech then fine, but otherwise it's a case of "use what you know", because new tech that isn't focused on that is going to waste my precious time and resources down the line.

by danjac 4y ago

steveBK123 4y ago

Yes. As long as your k8s is some self contained island of code needing no R/W access to resources on k8s, maybe I'll believe it's easy to setup AND to support. Additionally if you are in some type of small shop environment where you have basically unfettered permissions in AWS console to do what you need, doubly so. Now try to get access to/from RDS, S3, EFS, Netapp NFS, some HTTP endpoint on-prem, some other HTTP endpoint on EC2, non-MSK Kafka, cross account, etc? Pain. Oh you need to submit a ticket to your Linux sysadmins, Infra team, CloudOps, DevOps, Networking or Infosec for each of the above permissions? And sometimes need 2-3 of those teams to coordinate to make it happen? Death.

thegginthesky 4y ago
I've worked on 2 companies where I was in a team of 4/5 devs + 1 SRE in charge of the infra. In both cases, once the SRE guy created the appropriate terraform modules and CI template, he had to move on to a new team because there was literally no work left for him to do. Once things are infra as a code and the happy path is defined and implemented as a process, we rarely lost time thinking about infrastructure. Or anything. The system needs permission for AWS S3? Here is the component. Does it need to setup secrets? Here is the component. Does it needs to be in the same VPC as a Redshift cluster, or whatnot? There's a component for that. Do you need a SQS/RDS/Redis service? You get it. K8s, terraform and Atlantis have been such huge enablers I cannot overstate. Everything is so nimble and cheaply setup that our current infra bill is about 500USD/month to generate over 100k MRR for the company. And better yet, we rarely ever think of our infra unless we want to do something brand new (such as testing Sagemaker), or there's unexpected traffic above our current scaling limits (about once a year). Plus, if we need to fix anything infra related there's an internal cookbook for that. I also feel I have acquired useful skills by learning all of this.
- steveBK123 4y ago
  That sounds like a smaller shop maybe. The problem is big, established shops, with dozens of teams of dozens of people.. 300 developers, etc.. think they can get all the above with 2 CloudOps guys. Instead zero happy paths get defined in a timely manner, everything you are trying to do is hamstrung and waiting in the CloudOps queue, and you can't self-serve around them because you cannot do it yourself due to permissions. Imagine an org where "the happy path for RDS will be ready in 2-3 months, and only support one database type and 3 instance sizes" is somehow acceptable. Rinse & repeat for an alphabet soup of 25 other AWS services.
  - dpkirchner 4y ago
    That's more a problem with bad organizations than cloud or kubernetes. Besides, a common complaint about ops (called IT back then) was how long it took to provision and deploy resources. Months was common, unless the org had the foresight and budget to have lots of cold hardware on standby.
    - steveBK123 4y ago
      Yeah, the challenge is that after decades many of these firms actually had their onprem solutions dialed in. VMs can exist in the datacenter as well, and by the year 2022 most places have this stuff unbooked & behind a ticketing system with a defined SLA. Unfortunately when these slow moving medium/big firms then move into the cloud, they re-invent all the opacity, inertia, and other issues that they had solved onprem already. Some think they can just replace 25 DC ops people with 2 cloudOps and somehow its going to all just work out.
- hbrn 4y ago
  > Once things are infra as a code and the happy path is defined and implemented as a process, we rarely lost time thinking about infrastructure. > we rarely ever think of our infra unless we want to do something brand new That just means your infra is stable. It would be exactly the same without terraform/k8s.
gloosx 4y ago
As long as it's all in the same VPC everything goes through HTTP and other protocols like a knife thru butter, you don't even have to think about it knowing it is all in the same private network

deleted 4y ago

[deleted]

gloosx 4y ago

The point is – k8s essentially simplifies the solution, so what you'd get up and running will serve you for the time being without any maintenance. You're talking about it like it's a whole different framework of thinking which you will need to adopt from scratch, but it's not. Containerisation and virtualisation concepts are here since the beginning of time, just this thing makes it real easy to grasp, apart from that it is all same technology. It literally allows anyone to spin-up amazing production clusters quickly with zero overhead – assuming you know what you're spinning up, something that just took months of configuration works – now abstracted to few funny .yaml files.

danjac 4y ago
There's literally a web site of Kubernetes failure stories: https://k8s.af/ https://k8s.af/ where people experienced in Kubernetes have shot themselves in the foot. To be fair, it's over a year old, and probably the state of the art and best practices have advanced to the point where many of these are unlikely to happen. But in general, Kubernetes is widely considered a very big complex thing full of footguns for the unwary for a reason. So while I agree with you that sure, you can get up and running by watching some videos and copy-pasting a few YAML files, the problem is when you are deep into critical production and you run into issues (the classic "I get woken up at 3am by an alert or my boss and we need to fix this in 10 minutes") and you lack the deeper understanding of the framework to even know where to start. Or even worse, you insert an insidious security issue through a misconfiguration and you are not even aware you have been owned. Security is notoriously difficult to get right, even for people with experience in frameworks and systems, let alone a newbie. Kubernetes is not "secure by default" and again, there are footguns for the unwary: https://snyk.io/learn/kubernetes-security/ https://snyk.io/learn/kubernetes-security/ None of these things make Kubernetes bad, per se, and I'm sure the benefits outweigh the costs. But personally I would be cautious about adopting it for serious production without having some solid real-world experience first. There's a bigger question here as well, regarding small teams (as in startups, not just pizza teams in a tech giant). Where do you want your very limited time and focus to be? If I'm building some SAAS application I want to put my coding focus close to 100% towards the business model. If that model requires me learning new tech then fine, but otherwise it's a case of "use what you know", because new tech that isn't focused on that is going to waste my precious time and resources down the line.