4 ms·

I don’t want Kubernetes period. Best decision we’ve made at work is to migrate away from k8s and onto AWS ECS. I just want to deploy containers! DevOps went fro

I don’t want Kubernetes period. Best decision we’ve made at work is to migrate away from k8s and onto AWS ECS. I just want to deploy containers! DevOps went from something you did when standing up or deploying an application, to an industry-wide jobs program. It’s the TSA of the software world.

mugsie 2y ago
Thats great if that works for you, and for a lot people and teams. You have just shifted the complexity of networking, storage, firewalling, IP management, L7 proxying to AWS, but hey, you do have click ops there. > DevOps went from something you did when standing up or deploying an application, to an industry-wide jobs program. It’s the TSA of the software world. DevOps was never a job title, or process, it was a way of working, that went beyond yeeting to prod, and ignoring it. From that one line, you never did devops - you did dev, with some deployment tools (that someone else wrote?)
- ninjha 2y ago
  You can have Click-Ops on Kubernetes too! Everything has a schema so it's possible to build a nice UI on top of it (with some effort). My current project is basically this, except it edits your git-ops config repository, so you can click-ops while you git-ops.
  - k8sToGo 2y ago
    You mean ArgoCD and Rancher? Both ready to do click ops!
    - ninjha 2y ago
      I mean you can edit a big YAML file inside ArgoCD, but what I'm building is an actual web form (e.x. `spec.rules[].http.paths[].pathType` is a dropdown of `Prefix`, `ImplementationSpecific`, `Exact`), and all your documentation inline as you're editing. People have tried this before but usually the UI version is not fully complete so you have to drop to YAML. Now that the spec is good enough it's possible to build a complete UI for this.
      - liveoneggs 2y ago
        ArgoCD has a "New App" button that opens an actual web form you fill out.
      - mugsie 2y ago
        Yup, and it has the advantage of having a easily backed up state store to represent the actions of the GUI. I always liked the octant UI autogeneration for CRDs and the way it just parsed things correctly from the beginning, if they had an edit mode that would be perfect
        ninjha 2y ago
        Is there anything in particular you like about what Octant does? I don't see anything that actually looks at the object spec, just the status fields / etc.
      - k8sToGo 2y ago
        Sounds great. An interactive Spec builder, if I understand correctly.
  - potamic 2y ago
    Anywhere we can see your project?
frazbin 2y ago
If I may ask, just to educate myself where do you keep the ECS service/task specs and how do you mutate them across your stacks? How long does it take to stand up/decomm a new instance of your software stack? How do you handle application lifecycle concerns like database backup/restore, migrations/upgrades? How have you supported developer stories like "I want to test a commit against our infrastructure without interfering with other development"? I recognize these can all be solved for ECS but I'm curious about the details and how it's going. I have found Kubernetes most useful when maintaining lots of isolated tenants within limited (cheap) infrastructure, esp when velocity of software and deployments is high and has many stakeholders (customer needs their demo!)
- liveoneggs 2y ago
  https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... https://docs.aws.amazon.com/AmazonECS/latest/developerguide/example_task_definitions.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... etc
  - mugsie 2y ago
    Yeah, that doesn't really answer the question at all... Do you just have a pile of cloudformation on your desktop? point and click? tf? And then none of the actual questions like > How do you handle application lifecycle concerns like database backup/restore, migrations/upgrades? were even touched.
    - liveoneggs 2y ago
      There is no difference between cloudformation, clicking, terraform, boto, awscli, pulumi, or whatever else. The platform at the other end of those tools is still ECS. Backing up databases isn't the job of the container-running platform (ECS), especially not in AWS-world where databases are managed with RDS. The rest of the questions were "how do I run containers on ecs?" in various forms. The answers to all of them is "by asking ecs to run containers in various forms."
k8sToGo 2y ago
It is always this holier than thou attitude of Software engineers towards DevOps that is annoying. Especially if it comes from ignorance. These days often DevOps is done by former Software Engineers rather than "old fashioned" Sys admins. Just because you are ignorant on how to use AKS efficiently, doesn't mean your alternative is better.
- mugsie 2y ago
  Yeah, DevOps was a culture not a job title, and then we let us software engineers in who just want to throw something into prod and go home on friday night, so they decided it was a task, and the lowest importance thing possible, but simultaniously, the devops/sre/prod eng teams needed to be perfect, because its prod. it is a wierd dichotomy I have seem, and it is getting worse. We let teams have access to argo manifiests, and helm charts, and even let them do custom in repo charts. not one team in the last year has actually gone and looked at k8s docs to figure out how to do basic shit, they just dump questions into channels, and soak up time from people explaining the basics of the system their software runs on.
- sgarland 2y ago
  > These days often DevOps is done by former Software Engineers rather than "old fashioned" Sys admins. Yes, and the world is a poorer place for it. Google’s SRE model works in part because they have _both_ Ops and SWE backgrounds. The thing about traditional Ops is, while it may not scale to Google levels, it does scale quite well to the level most companies need, _and_ along the way, it forces people to learn how computers and systems work to a modicum of depth. If you’re having to ssh into a box to see why a process is dying, you’re going to learn something about that process, systemd, etc. If you drag the dev along with you to fix it, now two people have learned cross-areas. If everything is in a container, and there’s an orchestrator silently replacing dying pods, that no longer needs to exist. To be clear, I _love_ K8s. I run it at home, and have used it professionally at multiple jobs. What I don’t like is how it (and every other abstraction) have made it such that “infra” people haven’t the slightest clue how infra actually operates, and if you sat them down in front of an empty, physical server, they’d have no idea how to bootstrap Linux on it.
blazing234 2y ago
Why don't you just deploy to cloud run on gcp and call it a day
Spivak 2y ago
I'm so confused about the jobs program thing. I'm an infra engineer who has had the title devops for parts of my career. I feel like I've always been desperately needed by teams of software devs that don't want to concern themselves with the gritty reality of actually running software in production. The job kinda sucks but for some reason jives with my brain. I take a huge amount of work and responsibility off the plates of my devs and my work scales well to multiple teams and multiple products. I've never seen an infra/devops/platform team not swamped with work and just spinning their tires on random unnecessary projects. We're more expensive on average than devs, harder to hire, and two degrees separated from revenue. We're not a typically overstaffed role.
nijave 2y ago
ECS is very very similar to Kubernetes and duplicates pretty much all of the functionality except AWS names and manages each piece as a separate service/offering. ECS+Route53+ALB/ELB+EFS+Parameter Store+Secrets Manager+CloudWatch (Metrics, Logs, Events)+VPC+IAM/STS and you're pretty close in functionality.

mugsie 2y ago

Thats great if that works for you, and for a lot people and teams. You have just shifted the complexity of networking, storage, firewalling, IP management, L7 proxying to AWS, but hey, you do have click ops there. > DevOps went from something you did when standing up or deploying an application, to an industry-wide jobs program. It’s the TSA of the software world. DevOps was never a job title, or process, it was a way of working, that went beyond yeeting to prod, and ignoring it. From that one line, you never did devops - you did dev, with some deployment tools (that someone else wrote?)

ninjha 2y ago
You can have Click-Ops on Kubernetes too! Everything has a schema so it's possible to build a nice UI on top of it (with some effort). My current project is basically this, except it edits your git-ops config repository, so you can click-ops while you git-ops.
- k8sToGo 2y ago
  You mean ArgoCD and Rancher? Both ready to do click ops!
  - ninjha 2y ago
    I mean you can edit a big YAML file inside ArgoCD, but what I'm building is an actual web form (e.x. `spec.rules[].http.paths[].pathType` is a dropdown of `Prefix`, `ImplementationSpecific`, `Exact`), and all your documentation inline as you're editing. People have tried this before but usually the UI version is not fully complete so you have to drop to YAML. Now that the spec is good enough it's possible to build a complete UI for this.
    - liveoneggs 2y ago
      ArgoCD has a "New App" button that opens an actual web form you fill out.
    - mugsie 2y ago
      Yup, and it has the advantage of having a easily backed up state store to represent the actions of the GUI. I always liked the octant UI autogeneration for CRDs and the way it just parsed things correctly from the beginning, if they had an edit mode that would be perfect
      - ninjha 2y ago
        Is there anything in particular you like about what Octant does? I don't see anything that actually looks at the object spec, just the status fields / etc.
    - k8sToGo 2y ago
      Sounds great. An interactive Spec builder, if I understand correctly.
- potamic 2y ago
  Anywhere we can see your project?

frazbin 2y ago

If I may ask, just to educate myself where do you keep the ECS service/task specs and how do you mutate them across your stacks? How long does it take to stand up/decomm a new instance of your software stack? How do you handle application lifecycle concerns like database backup/restore, migrations/upgrades? How have you supported developer stories like "I want to test a commit against our infrastructure without interfering with other development"? I recognize these can all be solved for ECS but I'm curious about the details and how it's going. I have found Kubernetes most useful when maintaining lots of isolated tenants within limited (cheap) infrastructure, esp when velocity of software and deployments is high and has many stakeholders (customer needs their demo!)

liveoneggs 2y ago
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... https://docs.aws.amazon.com/AmazonECS/latest/developerguide/example_task_definitions.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... etc
- mugsie 2y ago
  Yeah, that doesn't really answer the question at all... Do you just have a pile of cloudformation on your desktop? point and click? tf? And then none of the actual questions like > How do you handle application lifecycle concerns like database backup/restore, migrations/upgrades? were even touched.
  - liveoneggs 2y ago
    There is no difference between cloudformation, clicking, terraform, boto, awscli, pulumi, or whatever else. The platform at the other end of those tools is still ECS. Backing up databases isn't the job of the container-running platform (ECS), especially not in AWS-world where databases are managed with RDS. The rest of the questions were "how do I run containers on ecs?" in various forms. The answers to all of them is "by asking ecs to run containers in various forms."

k8sToGo 2y ago

It is always this holier than thou attitude of Software engineers towards DevOps that is annoying. Especially if it comes from ignorance. These days often DevOps is done by former Software Engineers rather than "old fashioned" Sys admins. Just because you are ignorant on how to use AKS efficiently, doesn't mean your alternative is better.

mugsie 2y ago
Yeah, DevOps was a culture not a job title, and then we let us software engineers in who just want to throw something into prod and go home on friday night, so they decided it was a task, and the lowest importance thing possible, but simultaniously, the devops/sre/prod eng teams needed to be perfect, because its prod. it is a wierd dichotomy I have seem, and it is getting worse. We let teams have access to argo manifiests, and helm charts, and even let them do custom in repo charts. not one team in the last year has actually gone and looked at k8s docs to figure out how to do basic shit, they just dump questions into channels, and soak up time from people explaining the basics of the system their software runs on.
sgarland 2y ago
> These days often DevOps is done by former Software Engineers rather than "old fashioned" Sys admins. Yes, and the world is a poorer place for it. Google’s SRE model works in part because they have _both_ Ops and SWE backgrounds. The thing about traditional Ops is, while it may not scale to Google levels, it does scale quite well to the level most companies need, _and_ along the way, it forces people to learn how computers and systems work to a modicum of depth. If you’re having to ssh into a box to see why a process is dying, you’re going to learn something about that process, systemd, etc. If you drag the dev along with you to fix it, now two people have learned cross-areas. If everything is in a container, and there’s an orchestrator silently replacing dying pods, that no longer needs to exist. To be clear, I _love_ K8s. I run it at home, and have used it professionally at multiple jobs. What I don’t like is how it (and every other abstraction) have made it such that “infra” people haven’t the slightest clue how infra actually operates, and if you sat them down in front of an empty, physical server, they’d have no idea how to bootstrap Linux on it.

blazing234 2y ago

Why don't you just deploy to cloud run on gcp and call it a day

Spivak 2y ago

I'm so confused about the jobs program thing. I'm an infra engineer who has had the title devops for parts of my career. I feel like I've always been desperately needed by teams of software devs that don't want to concern themselves with the gritty reality of actually running software in production. The job kinda sucks but for some reason jives with my brain. I take a huge amount of work and responsibility off the plates of my devs and my work scales well to multiple teams and multiple products. I've never seen an infra/devops/platform team not swamped with work and just spinning their tires on random unnecessary projects. We're more expensive on average than devs, harder to hire, and two degrees separated from revenue. We're not a typically overstaffed role.

nijave 2y ago

ECS is very very similar to Kubernetes and duplicates pretty much all of the functionality except AWS names and manages each piece as a separate service/offering. ECS+Route53+ALB/ELB+EFS+Parameter Store+Secrets Manager+CloudWatch (Metrics, Logs, Events)+VPC+IAM/STS and you're pretty close in functionality.