12 ms·

Kubernetes is a Google scale solution. Lots of teams said “hey if Google does it then it must be good!”…but forgot that they didn’t have the scale. It caught

Kubernetes is a Google scale solution. Lots of teams said “hey if Google does it then it must be good!”…but forgot that they didn’t have the scale. It caught on so much that for whatever reason it’s now the horrendous default. I’ve worked on at least 3 consulting projects that incorporated K8s and it slowed everything down and took way too much time, and we got nothing in return - because those projects only needed several instances, and not dozens or hundreds. If you need less than 8 instances to do host your product, run far away anytime anyone mentions k8s

jmillikin 4y ago
Kubernetes can't (currently) scale to Google sizes. It's designed for small- or medium-sized businesses, which might have 50,000 VMs or fewer. There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter.
- hpkuarg 4y ago
  > There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter. Forget a whole datacenter, even just one rack is an unimaginable amount of computing power, these days!
  - dublin 4y ago
    Fact: An well-equipped Raspberry Pi 4 has more memory, more compute power, more storage and vastly faster networking than the Cray supercomputer I worked with at a major oil company in the 1990's! With the exception of the craziness around click-tracking (web ads and marketing have warped compute use even more than crypto), the data required to run even enterprise-scale businesses today is not really all that large. For almost all purposes, we don't really need thousands of containers running on unimaginably fast computers, coordinated by AI-driven automation systems. What we need is software that is not morbidly obese.
    - gh02t 4y ago
      I mean a single V100 GPU (~= 100 TFLOPS) has FLOPS throughputs similar to top end mid-00's supercomputers like Blue Gene, at least superficially. And you can squeeze like 4+ in a 1U if you have enough cooling and power. https://en.wikipedia.org/wiki/History_of_supercomputing https://en.wikipedia.org/wiki/History_of_supercomputing
- blue_cookeh 4y ago
  > small- or medium-sized businesses, which might have 50,000 VMs or fewer Holy shit, is this considered small to medium enterprise now?
  - dijit 4y ago
    a single cluster supports approx 5,000 nodes, 110 pods per node~ The estimated maximum single cluster is 300,000 containers. That's pretty medium, I've ran more than a million processes before, and nomad has 1million containers as its challenge https://www.hashicorp.com/c1m https://www.hashicorp.com/c1m borg can handle this easily.
    - dlp211 4y ago
      I don't know anyone running 5k node K8S clusters. That said, borg (as an end user) appears to just keep scaling, but it uses a different model and makes different assumptions than K8S.
      - dijit 4y ago
        Sure. Most people have more clusters before they hit 5k nodes on a single cluster. But I’ve been in situations where it would have been worthwhile. I’ve been in situations with 30,000 machines that needed to be controlled. Splitting them out into very many clusters would be a lot of wasted overhead in configuration, administration and because you lose nodes to masters.
- ec109685 4y ago
  That’s a bit out of date. K8s can do 5,000 nodes and 300k VMs within its performance envelope: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg...
  - jmillikin 4y ago
    A VM would be a node. A pod isn't a VM, it's a process tree.
- drdaeman 4y ago
  I'm afraid this is why people pick Kubernetes. They believe small business needs those tens of thousands of VMs distributed across thousand of nodes and so on. With some exceptions, I believe that's a few orders of magnitude above what a small business can run on. Nowadays people just start their day by drinking some K2l-aid and spinning up a "basic" 6-node cluster for a development prototype. Maybe I'm wrong, of course.
- bg24 4y ago
  Your scale (50,000 VMs) is way too high for small and medium sized business :-) Is my observation correct that unicorns start to see that scale?
dusanh 4y ago
Asking as someone who has only tipped his toes into devops lately and is looking to learn K8, what is considered a reasonable "lightweight" alternative to Kubernetes these days?
- jstream67 4y ago
  If you are on AWS you should check out ECS Fargate (serverless). It is really really good. Probably one of their more polished products. If you want to stay on the Kubernetes route check out k3s. Super easy to setup and usable for small production workloads
  - jeffwask 4y ago
    I second this. There are a few limitations in Fargate that are annoying but overall it's solid and easy to use.
  - Sohcahtoa82 4y ago
    As a security engineer, I always cringe when anything involving containers is referred to as "serverless". I always thought that one of the advantages of going serverless was that you didn't have to worry about keeping the underlying operating system up-to-date. No needing to do a monthly "sudo apt update && apt upgrade" or whatever. But containers throw that all away when container images enter the world. Instead of updating your operating system, you're updating your images...and it's basically the same thing.
    - OJFord 4y ago
      Is anyone's goal of 'serverless' that they no longer have to deal with updating the OS? Most would say even a server-ful system (k8s, or whatever) should be considered 'cattle not pets' with immutable nodes replaced as needed anyway. No update, just replace. Just like building a new image and having new pods (or serverless whatevers) pull it.
      - Sohcahtoa82 4y ago
throwaway7865 4y ago
We’ve moved a small-scale business to Kubernetes and it made our lives much easier. Anywhere I’ve worked business always prioritizes high availability and close to zero downtime. No one sees a random delivered feature. But if a node fails at night - everybody knows it. Clients first of all. We’ve achieved it all almost out of the box with EKS. Setup with Fargate nodes was literally a one-liner of eksctl. Multiple environments are separated with namespaces. Leader elections between replicas are also easy. Lens is a very simple to use k8s IDE. If you know what you’re doing with Kubernetes (don’t use EC2 for nodes, they fail randomly), it’s a breeze.
- AaronM 4y ago
  We don't have an issue with that last point, lots of EC2 EKS nodes and they don't fail randomly. Were you using resources and limits correctly? EKS nodes can fall over randomly if you don't reserve resources on the nodes for system processes, and your workloads eat up all the resources. That's probably not well documented either.
  - Spivak 4y ago
    EC2 instances are inherently unreliable and that's not a knock on them, that's exactly the contract that you get using them and you're supposed to plan your architecture around the fact that at any moment an EC2 instance could die. We lose about 2-3 EC2 nodes per day (not like our app stops, like Amazon's own instance health goes red) and we couldn't care less.
    - jamesfinlayson 4y ago
      What percentage of EC2 nodes is that?
      - theK 4y ago
        Empirically around 0.1%
  - raffraffraff 4y ago
nokya 4y ago
Exactly. I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our architecture meetings, although he used to be a banking software architect in his previous assignments. Every question I ask ends up in a long 15 minutes monologue by the guy who architected all of it, even the most simple questions. They are soon launching a mobile app (only a mobile app and its corresponding API, not even a website) and they already have more than 60 containers running and talking to each other across three k8s clusters and half of them interact directly with third-parties outside. Even as I am being paid by the hour, I really feel sad for both the CTO and the developers attending the meeting. k8s is definitely not for everyone. Google has thousands hardware systems running the same hypervisor, same OS, same container engine and highly specialized stacks of micro-services that need to run by the thousands. And even, I am not sure that k8s would satisfy Google's actual needs tbh. Ironically, there are some companies that highly benefit from this and they are not necessarily "large" companies. In my case, k8s and devops in general made my life infinitely easier for on-site trainings: those who come with a poorly configured or decade-old laptop can actually enjoy the labs at the same pace than every other attendee.
- rahkiin 4y ago
  60 containers sounds like an architecture problem, not a Kubernetes problem. Kubernetes does not stop you from running 1 container in 1 pod receiving ingress and talking to a database.
  - deathanatos 4y ago
    Presuming too (it's hard to tell) that they mean 60 different types of containers. One of my clusters currently has ~311 containers, but that's mostly due to replication. If I count actually different containers (like, unique PodSpecs, or so), that count drops to ≈30. Even that is "high", and from an architectural standpoint, it isn't really a number I'd use. E.g., we have a simple daemon, but it also has a cronjob associated with it. So it has "2" PodSpecs by that count. But architecturally I'd call it a single thing. How it implements itself, that's up to it. A lot of our "unique PodSpec" count, too, comes from utility type things that do one thing, and do it well. Logging (which comes from our vendor) is 3 PodSpecs. Metrics is another 3. We have a network latency measurement (literally ping shoved into a container…): PodSpec. A thing that checks certs to ensure they don't expire: PodSpec. HTTP proxy (for SSRF avoidance): PodSpec. A tool that rotates out nodes so that their OSes can be patched: PodSpec. Let's Encrypt automation (a third party too): 3 PodSpecs … but hey, it does its job, and it's a third party tool, so what do I care, so long as it works the API between me and it suffices (and honestly, its logs are pretty good. When it has had problems, I've usually been able to discern why from the logs). DB backup. But most of these don't really add much conceptual overhead; any one is maybe tied (conceptually) to our applications, but not really to all the other utilities. (E.g., there isn't really coupling between, say, the cert renewer and the logging tooling.) A confused/new dev might need to have it explained to them what any of those given tools do, ofc., but … many of them you can just Google. … in previous jobs where we didn't use Kubernetes, we mostly just ignored a lot of the tasks that these tools handle. E.g., reboot a VM for patches? It was a custom procedure, depending on VM, and what is running on that VM. You needed to understand that, determine what the implications were … etc. And the end result was that reboots just didn't happen. K8s abstracts that (in the form of PDBs, and readiness checks) and can thus automate it. (And ensure that new loads don't need TLC that … an app dev realistically isn't going to be given the time to give.) If we needed a common thing on every node? That would be rough. We did finally get to having a common base VM image, but even them, all of the per-app VM images would need to be rebased on the newer one, and then all rolled out, and who/how would one even track that? And … in practice, it didn't happen.
dosethree 4y ago
You don't even end up spending time on Kubernetes, because k8s is just part of the solution, a container scheduler. You have to bring logs, monitoring, a container registry ,as well as a CI system with custom jobs and do integration of everything.
- deathanatos 4y ago
  This is true. But I had to bring all those things anyways, when I didn't run on k8s. I still needed some form of all of that. (Though "container registry" might be "package store", or something, depending on specifics of the implementation. Some form of artifact store.) And with-k8s and without-k8s to me is pretty similar: we vendor or FOSS most of it. The major cloud vendors all have container registries (of … varying quality…); similarly, at a previous company we used S3+a small shim as a Python package store. (We later moved to a vendored solution.) ELK for logs meant having a daemon set up per VM. Easier in k8s where I can push a DaemonSet to the entire cluster. With VMs … it's a per-app nightmare, really. Even then, that's really not perfect. In practice, in both situations, I feel like you end up having to integrate the apps with the metrics/logs providers. There's just not a common format. Sometimes, there are some libraries, e.g., there's some stuff for Prom's HTTP metrics APIs. Logs … eugh. Nothing amazing; getting structured logging requires per-app changes regardless of what you do. Sure, in either VM or k8s, you can just "suck up syslog/journald / docker logs", but what format are those in? They're not, is the answer, and I find most places do a "one text log per line" assumption (and then have stuff with multiline logs that just gets destroyed/corrupted/lost by the logging daemon) and it misses out on any sort of structured logs. jsonlines through those channels is a slight step up, but usually requires app changes.
  - dosethree 4y ago
    For sure, I had all this stuff before k8s as well
908B64B197 4y ago
> Kubernetes is a Google scale solution. Lots of teams said “hey if Google does it then it must be good!”…but forgot that they didn’t have the scale. It's also a Google engineer caliber solution. Lots of teams said “hey if Google engineers do it then it must be good!”…but forgot that they didn’t have the same in-house talent as Google.
KaiserPro 4y ago
> Kubernetes is a Google scale solution The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing. for example: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg... no more than 5k nodes. Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that the whole thing is fragile at that size. Having run a 36k node cluster in 2014, I know that K8s is just not designed for high scale high turnover vaguely complicated job graphs. I get the allure, but in practice K8s is designed for a specific usecase, and most people don't have that usecase. for most people you will want either ECS(its good enough, so long as you work around its fucking stupid service scheme) or something similar.
- russellendicott 4y ago
  Yeah, I don't know if it's because ECS was my first container orchestration experience but every time I look at teams trying to do k8s on AWS I think how much easier ECS would be.
  - 015a 4y ago
    The complexity difference between bog-standard ECS+Fargate and EKS+Fargate deployments rounds down pretty small. Biggest I've seen: ALB integration, IAM integration, and maybe certificate management. Most of that stuff is out-of-box on ECS, but on EKS you need some extra containers or configuration to watch the K8s API and provision stuff for you (if you want to use it; you can also just go pure-k8s) (edit: just to be clear, they provide all this for you; its not out-of-box, but its easy-to-add-box e.g. [1]). An argument could be made for something like CodeDeploy being better integrated on ECS, but that's more of a "k8s doesn't need CodeDeploy but ECS might" kind of thing. And even then, I wouldn't touch it. An argument could also be made that upgrading ECS clusters is a bit easier, as the cluster itself, uh, doesn't have a "version". But on Fargate, its pretty painless on EKS, and Fargate ECS tasks do have a "platform version" that generally doesn't have to be worried about (version: LATEST), but is nonzero nonetheless. Which is really to say that both ECS and EKS puke complexity, because its AWS, but the volume is pretty similar. [1] https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html https://docs.aws.amazon.com/eks/latest/userguide/aws-load-ba...
datavirtue 4y ago
We just moved our single instance web app with 50 users to K8. Not kidding. I totally bailed on that one. It started with moving it to the cloud (meh) and ended up with K8 in the cloud. A few of the guys wanted to pad the resume so we just turned our heads.
systemvoltage 4y ago
Scaling is oversold and under criticized. Folks, listen, if StackOverflow can run on this: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... So can your doctor's appointment website, your little ML app or Notion clone. "But...". No. You ain't gonna need it. Do some load testing, prove it to yourself. Now, multiply the load by 100x, reserve AWS resources and you're good to go.

jmillikin 4y ago

Kubernetes can't (currently) scale to Google sizes. It's designed for small- or medium-sized businesses, which might have 50,000 VMs or fewer. There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter.

hpkuarg 4y ago
> There are entire SaaS industries that could fit into a single Google/Facebook/Amazon datacenter. Forget a whole datacenter, even just one rack is an unimaginable amount of computing power, these days!
- dublin 4y ago
  Fact: An well-equipped Raspberry Pi 4 has more memory, more compute power, more storage and vastly faster networking than the Cray supercomputer I worked with at a major oil company in the 1990's! With the exception of the craziness around click-tracking (web ads and marketing have warped compute use even more than crypto), the data required to run even enterprise-scale businesses today is not really all that large. For almost all purposes, we don't really need thousands of containers running on unimaginably fast computers, coordinated by AI-driven automation systems. What we need is software that is not morbidly obese.
  - gh02t 4y ago
    I mean a single V100 GPU (~= 100 TFLOPS) has FLOPS throughputs similar to top end mid-00's supercomputers like Blue Gene, at least superficially. And you can squeeze like 4+ in a 1U if you have enough cooling and power. https://en.wikipedia.org/wiki/History_of_supercomputing https://en.wikipedia.org/wiki/History_of_supercomputing
blue_cookeh 4y ago
> small- or medium-sized businesses, which might have 50,000 VMs or fewer Holy shit, is this considered small to medium enterprise now?
- dijit 4y ago
  a single cluster supports approx 5,000 nodes, 110 pods per node~ The estimated maximum single cluster is 300,000 containers. That's pretty medium, I've ran more than a million processes before, and nomad has 1million containers as its challenge https://www.hashicorp.com/c1m https://www.hashicorp.com/c1m borg can handle this easily.
  - dlp211 4y ago
    I don't know anyone running 5k node K8S clusters. That said, borg (as an end user) appears to just keep scaling, but it uses a different model and makes different assumptions than K8S.
    - dijit 4y ago
      Sure. Most people have more clusters before they hit 5k nodes on a single cluster. But I’ve been in situations where it would have been worthwhile. I’ve been in situations with 30,000 machines that needed to be controlled. Splitting them out into very many clusters would be a lot of wasted overhead in configuration, administration and because you lose nodes to masters.
ec109685 4y ago
That’s a bit out of date. K8s can do 5,000 nodes and 300k VMs within its performance envelope: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg...
- jmillikin 4y ago
  A VM would be a node. A pod isn't a VM, it's a process tree.
drdaeman 4y ago
I'm afraid this is why people pick Kubernetes. They believe small business needs those tens of thousands of VMs distributed across thousand of nodes and so on. With some exceptions, I believe that's a few orders of magnitude above what a small business can run on. Nowadays people just start their day by drinking some K2l-aid and spinning up a "basic" 6-node cluster for a development prototype. Maybe I'm wrong, of course.
bg24 4y ago
Your scale (50,000 VMs) is way too high for small and medium sized business :-) Is my observation correct that unicorns start to see that scale?

dusanh 4y ago

Asking as someone who has only tipped his toes into devops lately and is looking to learn K8, what is considered a reasonable "lightweight" alternative to Kubernetes these days?

jstream67 4y ago
If you are on AWS you should check out ECS Fargate (serverless). It is really really good. Probably one of their more polished products. If you want to stay on the Kubernetes route check out k3s. Super easy to setup and usable for small production workloads
- jeffwask 4y ago
  I second this. There are a few limitations in Fargate that are annoying but overall it's solid and easy to use.
- Sohcahtoa82 4y ago
  As a security engineer, I always cringe when anything involving containers is referred to as "serverless". I always thought that one of the advantages of going serverless was that you didn't have to worry about keeping the underlying operating system up-to-date. No needing to do a monthly "sudo apt update && apt upgrade" or whatever. But containers throw that all away when container images enter the world. Instead of updating your operating system, you're updating your images...and it's basically the same thing.
  - OJFord 4y ago
    Is anyone's goal of 'serverless' that they no longer have to deal with updating the OS? Most would say even a server-ful system (k8s, or whatever) should be considered 'cattle not pets' with immutable nodes replaced as needed anyway. No update, just replace. Just like building a new image and having new pods (or serverless whatevers) pull it.
    - Sohcahtoa82 4y ago

throwaway7865 4y ago

We’ve moved a small-scale business to Kubernetes and it made our lives much easier. Anywhere I’ve worked business always prioritizes high availability and close to zero downtime. No one sees a random delivered feature. But if a node fails at night - everybody knows it. Clients first of all. We’ve achieved it all almost out of the box with EKS. Setup with Fargate nodes was literally a one-liner of eksctl. Multiple environments are separated with namespaces. Leader elections between replicas are also easy. Lens is a very simple to use k8s IDE. If you know what you’re doing with Kubernetes (don’t use EC2 for nodes, they fail randomly), it’s a breeze.

AaronM 4y ago
We don't have an issue with that last point, lots of EC2 EKS nodes and they don't fail randomly. Were you using resources and limits correctly? EKS nodes can fall over randomly if you don't reserve resources on the nodes for system processes, and your workloads eat up all the resources. That's probably not well documented either.
- Spivak 4y ago
  EC2 instances are inherently unreliable and that's not a knock on them, that's exactly the contract that you get using them and you're supposed to plan your architecture around the fact that at any moment an EC2 instance could die. We lose about 2-3 EC2 nodes per day (not like our app stops, like Amazon's own instance health goes red) and we couldn't care less.
  - jamesfinlayson 4y ago
    What percentage of EC2 nodes is that?
    - theK 4y ago
      Empirically around 0.1%
- raffraffraff 4y ago

nokya 4y ago

Exactly. I am consulting with a startup right now that chose to go everything docker/k8s. The CTO is half-shocked/half-depressed by the complexity of our architecture meetings, although he used to be a banking software architect in his previous assignments. Every question I ask ends up in a long 15 minutes monologue by the guy who architected all of it, even the most simple questions. They are soon launching a mobile app (only a mobile app and its corresponding API, not even a website) and they already have more than 60 containers running and talking to each other across three k8s clusters and half of them interact directly with third-parties outside. Even as I am being paid by the hour, I really feel sad for both the CTO and the developers attending the meeting. k8s is definitely not for everyone. Google has thousands hardware systems running the same hypervisor, same OS, same container engine and highly specialized stacks of micro-services that need to run by the thousands. And even, I am not sure that k8s would satisfy Google's actual needs tbh. Ironically, there are some companies that highly benefit from this and they are not necessarily "large" companies. In my case, k8s and devops in general made my life infinitely easier for on-site trainings: those who come with a poorly configured or decade-old laptop can actually enjoy the labs at the same pace than every other attendee.

rahkiin 4y ago
60 containers sounds like an architecture problem, not a Kubernetes problem. Kubernetes does not stop you from running 1 container in 1 pod receiving ingress and talking to a database.
- deathanatos 4y ago
  Presuming too (it's hard to tell) that they mean 60 different types of containers. One of my clusters currently has ~311 containers, but that's mostly due to replication. If I count actually different containers (like, unique PodSpecs, or so), that count drops to ≈30. Even that is "high", and from an architectural standpoint, it isn't really a number I'd use. E.g., we have a simple daemon, but it also has a cronjob associated with it. So it has "2" PodSpecs by that count. But architecturally I'd call it a single thing. How it implements itself, that's up to it. A lot of our "unique PodSpec" count, too, comes from utility type things that do one thing, and do it well. Logging (which comes from our vendor) is 3 PodSpecs. Metrics is another 3. We have a network latency measurement (literally ping shoved into a container…): PodSpec. A thing that checks certs to ensure they don't expire: PodSpec. HTTP proxy (for SSRF avoidance): PodSpec. A tool that rotates out nodes so that their OSes can be patched: PodSpec. Let's Encrypt automation (a third party too): 3 PodSpecs … but hey, it does its job, and it's a third party tool, so what do I care, so long as it works the API between me and it suffices (and honestly, its logs are pretty good. When it has had problems, I've usually been able to discern why from the logs). DB backup. But most of these don't really add much conceptual overhead; any one is maybe tied (conceptually) to our applications, but not really to all the other utilities. (E.g., there isn't really coupling between, say, the cert renewer and the logging tooling.) A confused/new dev might need to have it explained to them what any of those given tools do, ofc., but … many of them you can just Google. … in previous jobs where we didn't use Kubernetes, we mostly just ignored a lot of the tasks that these tools handle. E.g., reboot a VM for patches? It was a custom procedure, depending on VM, and what is running on that VM. You needed to understand that, determine what the implications were … etc. And the end result was that reboots just didn't happen. K8s abstracts that (in the form of PDBs, and readiness checks) and can thus automate it. (And ensure that new loads don't need TLC that … an app dev realistically isn't going to be given the time to give.) If we needed a common thing on every node? That would be rough. We did finally get to having a common base VM image, but even them, all of the per-app VM images would need to be rebased on the newer one, and then all rolled out, and who/how would one even track that? And … in practice, it didn't happen.

dosethree 4y ago

You don't even end up spending time on Kubernetes, because k8s is just part of the solution, a container scheduler. You have to bring logs, monitoring, a container registry ,as well as a CI system with custom jobs and do integration of everything.

deathanatos 4y ago
This is true. But I had to bring all those things anyways, when I didn't run on k8s. I still needed some form of all of that. (Though "container registry" might be "package store", or something, depending on specifics of the implementation. Some form of artifact store.) And with-k8s and without-k8s to me is pretty similar: we vendor or FOSS most of it. The major cloud vendors all have container registries (of … varying quality…); similarly, at a previous company we used S3+a small shim as a Python package store. (We later moved to a vendored solution.) ELK for logs meant having a daemon set up per VM. Easier in k8s where I can push a DaemonSet to the entire cluster. With VMs … it's a per-app nightmare, really. Even then, that's really not perfect. In practice, in both situations, I feel like you end up having to integrate the apps with the metrics/logs providers. There's just not a common format. Sometimes, there are some libraries, e.g., there's some stuff for Prom's HTTP metrics APIs. Logs … eugh. Nothing amazing; getting structured logging requires per-app changes regardless of what you do. Sure, in either VM or k8s, you can just "suck up syslog/journald / docker logs", but what format are those in? They're not, is the answer, and I find most places do a "one text log per line" assumption (and then have stuff with multiline logs that just gets destroyed/corrupted/lost by the logging daemon) and it misses out on any sort of structured logs. jsonlines through those channels is a slight step up, but usually requires app changes.
- dosethree 4y ago
  For sure, I had all this stuff before k8s as well

908B64B197 4y ago

> Kubernetes is a Google scale solution. Lots of teams said “hey if Google does it then it must be good!”…but forgot that they didn’t have the scale. It's also a Google engineer caliber solution. Lots of teams said “hey if Google engineers do it then it must be good!”…but forgot that they didn’t have the same in-house talent as Google.

KaiserPro 4y ago

> Kubernetes is a Google scale solution The problem is that its _not_ a google scale solution. Its something that _looks_ like a google scale solution, but is like a movie set compared to the real thing. for example: https://kubernetes.io/docs/setup/best-practices/cluster-large/ https://kubernetes.io/docs/setup/best-practices/cluster-larg... no more than 5k nodes. Its extra ordinarily chatty at that scale, which means it;ll cost on inter-vpc traffic. I also strongly suspect that the whole thing is fragile at that size. Having run a 36k node cluster in 2014, I know that K8s is just not designed for high scale high turnover vaguely complicated job graphs. I get the allure, but in practice K8s is designed for a specific usecase, and most people don't have that usecase. for most people you will want either ECS(its good enough, so long as you work around its fucking stupid service scheme) or something similar.

russellendicott 4y ago
Yeah, I don't know if it's because ECS was my first container orchestration experience but every time I look at teams trying to do k8s on AWS I think how much easier ECS would be.
- 015a 4y ago
  The complexity difference between bog-standard ECS+Fargate and EKS+Fargate deployments rounds down pretty small. Biggest I've seen: ALB integration, IAM integration, and maybe certificate management. Most of that stuff is out-of-box on ECS, but on EKS you need some extra containers or configuration to watch the K8s API and provision stuff for you (if you want to use it; you can also just go pure-k8s) (edit: just to be clear, they provide all this for you; its not out-of-box, but its easy-to-add-box e.g. [1]). An argument could be made for something like CodeDeploy being better integrated on ECS, but that's more of a "k8s doesn't need CodeDeploy but ECS might" kind of thing. And even then, I wouldn't touch it. An argument could also be made that upgrading ECS clusters is a bit easier, as the cluster itself, uh, doesn't have a "version". But on Fargate, its pretty painless on EKS, and Fargate ECS tasks do have a "platform version" that generally doesn't have to be worried about (version: LATEST), but is nonzero nonetheless. Which is really to say that both ECS and EKS puke complexity, because its AWS, but the volume is pretty similar. [1] https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html https://docs.aws.amazon.com/eks/latest/userguide/aws-load-ba...

datavirtue 4y ago

We just moved our single instance web app with 50 users to K8. Not kidding. I totally bailed on that one. It started with moving it to the cloud (meh) and ended up with K8 in the cloud. A few of the guys wanted to pad the resume so we just turned our heads.

systemvoltage 4y ago

Scaling is oversold and under criticized. Folks, listen, if StackOverflow can run on this: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... So can your doctor's appointment website, your little ML app or Notion clone. "But...". No. You ain't gonna need it. Do some load testing, prove it to yourself. Now, multiply the load by 100x, reserve AWS resources and you're good to go.