6 ms·

If your application doesn't need and likely won't need to scale to large clusters, or multiple clusters, then there's nothing wrong per se. with your solution.

by YZF 2y ago

If your application doesn't need and likely won't need to scale to large clusters, or multiple clusters, then there's nothing wrong per se. with your solution. I don't think k8s is that hard but there are a lot of moving pieces and there's a bit to learn. Finding someone with experience to help you can make a ton of difference. Questions worth asking: - Do you need a load balancer? - TLS certs and rotation? - Horizontal scalability. - HA/DR - dev/stage/production + being able to test/stage your complete stack on demand. - CI/CD integrations, tools like ArgoCD or Spinnaker - Monitoring and/or alerting with Prometheus and Grafana - Would you benefit from being able to deploy a lot of off the shelf software (lessay Elastic Search, or some random database, or a monitoring stack) via helm quickly/easily. - "Ingress"/proxy. - DNS integrations. If you answer yes to many of those questions there's really no better alternative than k8s. If you're building large enough scale web applications the almost to most of these will end up being yes at some point.

hkkwritesgmail 2y ago
If I need "some" of the above, Kubernetes forces me to grapple with "all" of the above. I think that is the issue.
probablybetter 2y ago
Containerization and orchestration of containers vs learning how to configure HaProxy, how to use Certbot, hmmmm The questions you pose are legit skills web developers need to have. Nothing you mentioned is obviated by K8s or containerization. "oh but you can get someone elses pre-configured image" uh huh... sure, you can also install malware. You will also need to one day maintain or configure the software running in them. You may even need to address issues your running software causes. You can't do that without mastering the software you are running!
zug_zug 2y ago
> If you answer yes to many of those questions there's really no better alternative than k8s. Nah, most of that list is basically free for any company that uses an amazon loadbalancer and an autoscale group. In terms of likeliness of incidents, time, and cost, those will each be an order of magnitude higher with a team of kubernetes engineers than less complex setup.
xorcist 2y ago
Every item on that list is "boring" tech. Approximately everyone have used load balancers, test environments and monitoring since the 90s just fine. What is it that you think make Kubernetes especially suited for this compared to every other solution during the past three decades? There are good reasons to use Kubernetes, mainly if you are using public clouds and want to avoid lock-in. I may be partial, since managing it pays my bills. But it is complex, mostly unnecessarily so, and no one should be able to say with a straight face that it achieves better uptime or requires less personnel than any alternative. That's just sales talk, and should be a big warning sign.
- YZF 2y ago
  It's the way things work together. If you want to add a new service you just annotate that service and DNS gets updated, your ingress gets the route added, cert-manager gets you the certs from let's encrypt. You want Prometheus to monitor your pod you just add the right annotation. When your server goes down k8s will move your pod around. k8s storage will take care of having the storage follow your pod. Your entire configuration is highly available and replicated in etcd. It's just very different than your legacy "standard" technology.
  - gr3ml1n 2y ago
    None of this is difficult to do or automate, and we've done it for years. Kubernetes simply makes it more complex by adding additional abstractions in the pursuit of pretending hardware doesn't exist. There are, maybe, a dozen companies in the world with a large enough physical footprint where Kubernetes might make sense. Everyone else is either engaged in resume-driven development, or has gone down some profoundly wrong path with their application architecture to where it is somehow the lesser evil.
    - sampullman 2y ago
      I used to feel the same way, but have come around. I think it's great for small companies for a few reasons. I can spin up effectively identical dev/ci/stg/prod clusters for a new project in an hour for a medium sized project, with CD in addition to everything GP mentioned. I basically don't have to think about ops anymore until something exotic comes up, it's nice. I agree that it feels clunky, and it was annoying to learn, but once you have something working it's a huge time saver. The ability to scale without drastically changing the system is a bonus.
signal11 2y ago
> If you answer yes to many of those questions there's really no better alternative than k8s. This is not even close to true with even a small number of resources. The notion that k8s somehow is the only choice is right along the lines of “Java Enterprise Edition is the only choice” — ie a real failure of the imagination. For startups and teams with limited resources, DO, fly.io and render are doing lots of interesting work. But what if you can’t use them? Is k8s your only choice? Let’s say you’re a large orgs with good engineering leadership, and you have high-revenue systems where downtime isn’t okay. Also for compliance reasons public cloud isn’t okay. DNS in a tightly controlled large enterprise internal network can be handled with relatively simple microservices. Your org will likely have something already though. Dev/Stage/Production: if you can spin up instances on demand this is trivial. Also financial services and other regulated biz have been doing this for eons before k8s. Load Balancers: lots of non-k8s options exist (software and hardware appliances). Prometheus / Grafana (and things like Netdata) work very well even without k8s. Load Balancing and Ingress is definitely the most interesting piece of the puzzle. Some choose nginx or Envoy, but there’s also teams that use their own ingress solution (sometimes open-sourced!) But why would a team do this? Or more appropriately, why would their management spend on this? Answer: many don’t! But for those that do — the driver is usually cost*, availability and accountability, along with engineering capability as a secondary driver. (*cost because it’s easy to set up a mixed ability team with experienced, mid-career and new engineers for this. You don’t need a team full of kernel hackers.) It costs less than you think, it creates real accountability throughout the stack and most importantly you’ve now got a team of engineers who can rise to any reasonable challenge, and who can be cross pollinated throughout the org. In brief the goal is to have engineers not “k8s implementers” or “OpenShift implementers” or “Cloud Foundry implementers”.
- lmm 2y ago
  > DNS in a tightly controlled large enterprise internal network can be handled with relatively simple microservices. Your org will likely have something already though. And it will likely be buggy with all sorts of edge cases. > Dev/Stage/Production: if you can spin up instances on demand this is trivial. Also financial services and other regulated biz have been doing this for eons before k8s. In my experience financial services have been notably not doing it. > Load Balancers: lots of non-k8s options exist (software and hardware appliances). The problem isn't running a load balancer with a given configuration at a given point in time. It's how you manage the required changes to load balancers and configuration as time goes on. It's very common for that to be a pile of perl scripts that add up to an ad-hoc informally specified bug-ridden implementation of half of kubernetes.
otabdeveloper4 2y ago
Kubernetes is great example of the "second-system effect". Kubernetes only works if you have a webapp written in a slow interpreted language. For anything else it is a huge impedance mismatch with what you're actually trying to do. P.S. In the real world, Kubernetes isn't used to solve technical problems. It's used as a buffer between the dev team and the ops team, who usually have different schedules/budgets, and might even be different corporate entities. I'm sure there might be an easier way to solve that problem without dragging in Google's ridiculous and broken tech stack.
- YZF 2y ago
  I think I live in the real world and your statement is not true for any of the projects I've been involved in. Kubernetes is absolutely used to solve real technical problems that would otherwise require a lot of work to solve. I would say as a rule it's not a webapp in a slow interpreted language that's hosted in k8s. It truly is about decoupling from the need to manage machines and operating systems at a lower level and being able to scale seamlessly. I'm really not following on the impedance mismatch from what you're actually trying to do. Where is that impedance mismatch? Let's take a simple example, Elastic Search and the k8s operator. You can edit a single line in your yaml and grow your cluster. That takes care of resources, storage, network etc. Can you do this manually with Elastic running on bare metal or in containers or in VM? Absolutely, it's a nightmare, non-replicable, process that will take you days. You don't need elastic search, or you never need to scale it, fine. You can run it on a single machine and lose all your data if that machine dies - fine.
  - tacitusarc 2y ago
    I’m curious if you’ve ever built and maintained a k8s cluster capable of reliably hosting an ES cluster? Because I have, and it was painful enough that we swapped to provisioning real HW with ansible. It is much easier to manage. I should note, we still manage a K8s cluster, but not for anything using persistent storage.
- maxdo 2y ago
  kubernetes is an API for your cluster, that is portable between providers, more or less. there are other abstractions, but they are not portable, e.g. fly.io, DO etc. so unless you want a vendor lock-in, you need it. for one of my products, I had to migrate due to business reasons 4 times into different kube flavors, from self-manged ( 2 times ) to GKE and EKS.
psychoslave 2y ago
Oz Nova nailed it nicely in "You Are Not Google" https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
- YZF 2y ago
  If you were Google k8s wouldn't cut it. I have experience of both options in multiple projects. Managing containers yourself and the surrounding infrastructure vs. using k8s. k8s just works. It's a mature ecosystem. It's really not as hard as people make of it. Replicating all the functionality of k8s and the ecosystem yourself is a ton more work. There are definitely wide swaths of applications that don't need container, don't need high availability, don't need load balancing, don't need monitoring, don't need any of this stuff or need some simpler subset. Then by all means don't k8s, don't use containers etc.

by YZF 2y ago

hkkwritesgmail 2y ago

If I need "some" of the above, Kubernetes forces me to grapple with "all" of the above. I think that is the issue.

probablybetter 2y ago

Containerization and orchestration of containers vs learning how to configure HaProxy, how to use Certbot, hmmmm The questions you pose are legit skills web developers need to have. Nothing you mentioned is obviated by K8s or containerization. "oh but you can get someone elses pre-configured image" uh huh... sure, you can also install malware. You will also need to one day maintain or configure the software running in them. You may even need to address issues your running software causes. You can't do that without mastering the software you are running!

zug_zug 2y ago

> If you answer yes to many of those questions there's really no better alternative than k8s. Nah, most of that list is basically free for any company that uses an amazon loadbalancer and an autoscale group. In terms of likeliness of incidents, time, and cost, those will each be an order of magnitude higher with a team of kubernetes engineers than less complex setup.

xorcist 2y ago

Every item on that list is "boring" tech. Approximately everyone have used load balancers, test environments and monitoring since the 90s just fine. What is it that you think make Kubernetes especially suited for this compared to every other solution during the past three decades? There are good reasons to use Kubernetes, mainly if you are using public clouds and want to avoid lock-in. I may be partial, since managing it pays my bills. But it is complex, mostly unnecessarily so, and no one should be able to say with a straight face that it achieves better uptime or requires less personnel than any alternative. That's just sales talk, and should be a big warning sign.

YZF 2y ago
It's the way things work together. If you want to add a new service you just annotate that service and DNS gets updated, your ingress gets the route added, cert-manager gets you the certs from let's encrypt. You want Prometheus to monitor your pod you just add the right annotation. When your server goes down k8s will move your pod around. k8s storage will take care of having the storage follow your pod. Your entire configuration is highly available and replicated in etcd. It's just very different than your legacy "standard" technology.
- gr3ml1n 2y ago
  None of this is difficult to do or automate, and we've done it for years. Kubernetes simply makes it more complex by adding additional abstractions in the pursuit of pretending hardware doesn't exist. There are, maybe, a dozen companies in the world with a large enough physical footprint where Kubernetes might make sense. Everyone else is either engaged in resume-driven development, or has gone down some profoundly wrong path with their application architecture to where it is somehow the lesser evil.
  - sampullman 2y ago
    I used to feel the same way, but have come around. I think it's great for small companies for a few reasons. I can spin up effectively identical dev/ci/stg/prod clusters for a new project in an hour for a medium sized project, with CD in addition to everything GP mentioned. I basically don't have to think about ops anymore until something exotic comes up, it's nice. I agree that it feels clunky, and it was annoying to learn, but once you have something working it's a huge time saver. The ability to scale without drastically changing the system is a bonus.

signal11 2y ago

> If you answer yes to many of those questions there's really no better alternative than k8s. This is not even close to true with even a small number of resources. The notion that k8s somehow is the only choice is right along the lines of “Java Enterprise Edition is the only choice” — ie a real failure of the imagination. For startups and teams with limited resources, DO, fly.io and render are doing lots of interesting work. But what if you can’t use them? Is k8s your only choice? Let’s say you’re a large orgs with good engineering leadership, and you have high-revenue systems where downtime isn’t okay. Also for compliance reasons public cloud isn’t okay. DNS in a tightly controlled large enterprise internal network can be handled with relatively simple microservices. Your org will likely have something already though. Dev/Stage/Production: if you can spin up instances on demand this is trivial. Also financial services and other regulated biz have been doing this for eons before k8s. Load Balancers: lots of non-k8s options exist (software and hardware appliances). Prometheus / Grafana (and things like Netdata) work very well even without k8s. Load Balancing and Ingress is definitely the most interesting piece of the puzzle. Some choose nginx or Envoy, but there’s also teams that use their own ingress solution (sometimes open-sourced!) But why would a team do this? Or more appropriately, why would their management spend on this? Answer: many don’t! But for those that do — the driver is usually cost*, availability and accountability, along with engineering capability as a secondary driver. (*cost because it’s easy to set up a mixed ability team with experienced, mid-career and new engineers for this. You don’t need a team full of kernel hackers.) It costs less than you think, it creates real accountability throughout the stack and most importantly you’ve now got a team of engineers who can rise to any reasonable challenge, and who can be cross pollinated throughout the org. In brief the goal is to have engineers not “k8s implementers” or “OpenShift implementers” or “Cloud Foundry implementers”.

lmm 2y ago
> DNS in a tightly controlled large enterprise internal network can be handled with relatively simple microservices. Your org will likely have something already though. And it will likely be buggy with all sorts of edge cases. > Dev/Stage/Production: if you can spin up instances on demand this is trivial. Also financial services and other regulated biz have been doing this for eons before k8s. In my experience financial services have been notably not doing it. > Load Balancers: lots of non-k8s options exist (software and hardware appliances). The problem isn't running a load balancer with a given configuration at a given point in time. It's how you manage the required changes to load balancers and configuration as time goes on. It's very common for that to be a pile of perl scripts that add up to an ad-hoc informally specified bug-ridden implementation of half of kubernetes.

otabdeveloper4 2y ago

Kubernetes is great example of the "second-system effect". Kubernetes only works if you have a webapp written in a slow interpreted language. For anything else it is a huge impedance mismatch with what you're actually trying to do. P.S. In the real world, Kubernetes isn't used to solve technical problems. It's used as a buffer between the dev team and the ops team, who usually have different schedules/budgets, and might even be different corporate entities. I'm sure there might be an easier way to solve that problem without dragging in Google's ridiculous and broken tech stack.

YZF 2y ago
I think I live in the real world and your statement is not true for any of the projects I've been involved in. Kubernetes is absolutely used to solve real technical problems that would otherwise require a lot of work to solve. I would say as a rule it's not a webapp in a slow interpreted language that's hosted in k8s. It truly is about decoupling from the need to manage machines and operating systems at a lower level and being able to scale seamlessly. I'm really not following on the impedance mismatch from what you're actually trying to do. Where is that impedance mismatch? Let's take a simple example, Elastic Search and the k8s operator. You can edit a single line in your yaml and grow your cluster. That takes care of resources, storage, network etc. Can you do this manually with Elastic running on bare metal or in containers or in VM? Absolutely, it's a nightmare, non-replicable, process that will take you days. You don't need elastic search, or you never need to scale it, fine. You can run it on a single machine and lose all your data if that machine dies - fine.
- tacitusarc 2y ago
  I’m curious if you’ve ever built and maintained a k8s cluster capable of reliably hosting an ES cluster? Because I have, and it was painful enough that we swapped to provisioning real HW with ansible. It is much easier to manage. I should note, we still manage a K8s cluster, but not for anything using persistent storage.
maxdo 2y ago
kubernetes is an API for your cluster, that is portable between providers, more or less. there are other abstractions, but they are not portable, e.g. fly.io, DO etc. so unless you want a vendor lock-in, you need it. for one of my products, I had to migrate due to business reasons 4 times into different kube flavors, from self-manged ( 2 times ) to GKE and EKS.

psychoslave 2y ago

Oz Nova nailed it nicely in "You Are Not Google" https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb

YZF 2y ago
If you were Google k8s wouldn't cut it. I have experience of both options in multiple projects. Managing containers yourself and the surrounding infrastructure vs. using k8s. k8s just works. It's a mature ecosystem. It's really not as hard as people make of it. Replicating all the functionality of k8s and the ecosystem yourself is a ton more work. There are definitely wide swaths of applications that don't need container, don't need high availability, don't need load balancing, don't need monitoring, don't need any of this stuff or need some simpler subset. Then by all means don't k8s, don't use containers etc.