9 ms·

I'm always kind of blown away by experiences like this. Admittedly, I've been using Kubernetes since the early days and I manage an Infra team that operates a c

I'm always kind of blown away by experiences like this. Admittedly, I've been using Kubernetes since the early days and I manage an Infra team that operates a couple thousand self-managed Kubernetes clusters so... expert blindness at work. Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy. Maybe it's because I adopted early and have grown with the technology it all just makes sense? It's not that complicated if you limit yourself to the core stuff. Maybe I need to write a book like "Kubernetes for Greybeards" or something like that. What does fucking kill me in the Kubernetes ecosystem is the amount of add-on crap that is pitched as "necessary". Sidecars... so many sidecars. Please stop. There's way too much vendor garbage surrounding the ecosystem and dev's rarely stop to think about whether they should deploy something when it's easy as dropping in some YAML and letting the cluster magically run it.

theptrk 2y ago
I would pay for the outline of this book.
t-writescode 2y ago
> Admittedly, I've been using Kubernetes since the early days and I manage an Infra team I think this is where the big difference is. If you're leading a team and introduced all good practices from the start, then the k8s and Terraform or whatever config files can never get so very complicated that a Gordian knot isn't created. Perhaps k8s is nice and easy to use - many of the commands certainly are, in my experience. Developers have, over years and decades, learned how to navigate code and hop from definition to definition, climbing the tree and learning the language they're operating in, and most of the languages follow similar-enough patterns that they can crawl around. Configuring a k8s cluster has absolutely none of that knowledge built up; and, reading something that has rough practices is not a good place to learn what it should look like.
- paddy_m 2y ago
  Thank you. I can always xargs grep for a function name, at worst. Dir() in python at a debugger for other things. With YAML, kubernetes and other devops hotness, I frequently can’t even find the relevant scripts/YAML that are executed nor their codebases. This also happens with configuration based packaging setups. Python hatch in particular, but sometimes node/webpack/rollup/vite.
adastra22 2y ago
I would buy that book.
figassis 2y ago
This, all the sidecars. Use kubernetes to run your app like you would without it, take advantage of the flexibility, avoid the extra complexity. Service discovery sidecars? Why not just use the out of the box dns features?
- tommica 2y ago
  Because new people don't know better - I've never used k8s, but have seen sidecars being promoted as a good thing, so I might have used them
  - namaria 2y ago
    Maybe the "I've heard about" approach to tooling is the problem here?
stiray 2y ago
I would buy the book. Just translate all "new language" concepts into well known concepts from networking and system administration. It would be best seller. If I would only have a penny for each time I wasted hours trying to figure out what something in "modern IT" is, just to figure out that I already knew what it is, but it was well hidden under layers of newspeak...
- radicalbyte 2y ago
  The book I read on K8S written by a core maintainer made is very clear.
  - schnirz 2y ago
    Which book would that be, out of interest?
    - radicalbyte 2y ago
      Kubernetes in Action
      - xorcist 2y ago
        A book from 2017? Is that still relevant to understand a modern Kubernetes cluster? The CNCF ecosystem looked a lot different back then.
        radicalbyte 2y ago
        Yup it's a great introduction as you can fly through it (3-4 hours to read most of it), at least if you already have a grasp of networking, linux, processes, threads, containers etc. Then you can hit other resources (in my case working with a team who've been using K8S for a few years). If you (or anyone else) has suggestions for something newer and covering more than just the core (like various different components you can use, Helm, Argo, ISTIO etc etc) then I'd appreciate it :-)
        2y ago
AtlasBarfed 2y ago
So you don't run any databases in those thousands of clusters? To your point, and I have not used k8s I just started to research it when my former company was thinking about shoehorning cassandra into k8s... But there was dogma around not allowing access to VM command exec via kubectl, while I basically needed it in the basic mode for certain one-off diagnosis needs and nodetool stuff... And yes, some of the floated stuff was "use sidecars" which also seemed to architect complexity for dogma's sake.
- voidfunc 2y ago
  > So you don't run any databases in those thousands of clusters? We do, but not of the SQL variety (that I am aware of). We have persistent key-value and document store databases hosted in these clusters. SQL databases are off-loaded to managed offering's in the cloud. Admittedly, this does simplify a lot of problems for us.
  - tayo42 2y ago
    How much data? I keep hearing k8s isn't usable becasue sometimes there is to much data and it can't be moved around.
    - pletnes 2y ago
      The simplest approach I’m aware of is to create the k8s cluster and databases in the same datacenter / availability zone.
    - darkstar_16 2y ago
      In the managed k8s space, the data is on a PVC in the same availability zone as the node it is being mounted on. If the node dies, the volume is just mounted on to a new node in the same zone. There is no data movement required.
    - eek04_ 2y ago
      While I've not played with k8, I did run stuff in Google's Borg for a very long while, and that has a similar architecture. My team was petabyte scale and we were far from the team with the largest footprint. So it is clearly possible to handle large scale data in this type of architecture.
mitjam 2y ago
Absolutely, a reasonably sophisticated scalable app platform looks like a half-baked and undocumented reimagination of Kubernetes. Admittedly: The ecosystem is huge and less is more in most cases, but the foundation is cohesive and sane. Would love to read the k8s for greybeards book.
- coldtea 2y ago
  >Absolutely, a reasonably sophisticated scalable app platform looks like a half-baked and undocumented reimagination of Kubernetes. Or maybe Kubernetes looks like a committee designed, everything and the kitchen sink, over-engineered, second system effect, second system effect suffering, YAGNI P.O.S., that only the kind of "enterprise" mindset that really enjoyed J2EE in 2004 and XML/SOAP vs JSON/REST would love...
  - p_l 2y ago
    You misspelled DCOS /s
BiteCode_dev 2y ago
Would buy. But you probably should teach a few live courses before writing it because of expert blindness. Otherwise you will miss thr mark. Would pay for a decent remote live course intro.
awestroke 2y ago
I would buy that book in a heartbeat. All documentation and guides on kubernetes seem to assume you already know why things are done a certain way
DanielHB 2y ago
> What does fucking kill me in the Kubernetes ecosystem is the amount of add-on crap that is pitched as "necessary". Sidecars... so many sidecars. Yeah it is the same with terraform modules, I was trying to argument at a previous job that we should stick to a single module (the cloud provider module) but people just love adding crap if it saves them 5 lines of configuration. Said crap of course adding tons of unnecessary resources in your cloud that no one understands.
karmarepellent 2y ago
Agreed. The best thing we did back when we ran k8s clusters, was moving a few stateful services to dedicated VMs and keep the clusters for stateless services (the bulk) only. Running k8s for stateless services was an absolute bliss. At that time stateful services were somewhat harder to operate on k8s because statefulness (and all that it encapsulates) was kinda full of bugs. That may certainly have changed over the last few years. Maybe we just did it wrong. In any case if you focused on the core parts of k8s that were mature back then, k8s was (and is) a good thing.
jq-r 2y ago
Those "necessary" add-ons and sidecars are out of control, but its the people problem. I'm part of the infra team and we manage just couple of k8s clusters, but those are quite big and have very high traffic load. The k8s + terraform code is simple, with no hacks, reliable and easy to upgrade. Our devs love it, we love it too and all of this makes my job pleasant and as little stressful as possible. But we recently hired a staff engineer to the team (now the most senior) and the guy just cannot rest still. "Oh we need a service mesh because we need visibility! I've been using it on my previous job and its the best thing ever." Even though we have all the visibility/metrics that we need and never needed more than that. Then its "we need a different ingress controller, X is crap Y surely is much better!" etc. So its not inexperienced engineers wanting newest hotness because they have no idea how to solve stuff with the tools they have, its sometimes senior engineers trying to justify their salary, "seniority" by buying into complexity as they try to make themselves irreplaceable.
- alienchow 2y ago
  How do you scale mTLS ops when the CISO comes knocking?
- jppittma 2y ago
  Service mesh is complicated, but the reason you use it integrate services across clusters. That and it has a bunch of useful reverse proxy features. On the other hand, It took me and 2 other guys two evenings of blood, sweat, and tears to understand what the fuck a virtual service actually does. It’s not strictly necessary, but if you’ve had to put in the work elsewhere, I’d use it.
- cyberpunk 2y ago
  To be fair, istio and cilium are extremely useful tools to have under your belt. There’s always a period of “omgwhat” when new senior engineers join and they want to improve things. There’s a short window between joining and getting bogged into a million projects where this is possible. Embrace it I recon.
  - p_l 2y ago
carlmr 2y ago
>expert blindness at work. >It's not that complicated if you limit yourself to the core stuff. Isn't this the core problem with a lot of technologies. There's a right way to use it, but most ways are wrong. An expert will not look left and right anymore, but to anyone entering the technology with fresh eyes it's a field with abundance of landmines to navigate around. It's simply bad UX and documentation. It could probably be better. But now it's too late to change everything because you'd annoy all the experts. >There's way too much vendor garbage surrounding the ecosystem Azure has been especially bad in this regard. Poorly documented in all respects, too many confusing UI menus that have similar or same names and do different things. If you use Azure Kubernetes the wrapper makes it much harder to learn the "core essentials". It's better to run minkube and get to know k8s first. Even then a lot of the Azure stuff remains confusing.
- wruza 2y ago
  This and a terminology rug pull. You wanted to upload a script and install some deps? Here’s your provisioning genuination frobnicator tutorial, at the end of which you’ll learn how to maintain the coalescing encabulation for your appliance unit schema, which is needed for automatic upload. It always feels like thousands times bigger complexity (just in this part!) than your whole project.
  - jterrys 2y ago
    You nailed it. Genuinely the most frustrating part about learning kubernetes... is just realizing that whatever the fuck they're talking about is a fancy wrapper for a concept that's existed since 90s.
- rbanffy 2y ago
  > There's a right way to use it, but most ways are wrong. This is my biggest complaint. There is no simple obvious way to set it up. There is no "sane default" config. > It's better to run minkube and get to know k8s first. Indeed. It should be trivial to set up a cluster from bare metal - nothing more than a `dnf install` and some other command to configure core functionality and to join machines into that cluster. Even when you go the easy way (with, say, Docker Desktop) you need to do a lot of steps just to have an ingress router.
pas 2y ago
My problem is the brittleness. Sure, I am playing with fire (k3s, bare metal, cilium, direct assigned IP to Ingresses), but a few weeks ago on one cluster suddenly something stopped working in the external IP -> internal cluster IP network path. (And after a complete restart things got worse. Oops. Well okay time to test backups.)
cryptonym 2y ago
K8S is a beast and the ecosystem is wild. Newcomer won't know how to proceed to keep things simple, while still understanding everything that is being used.
quietbritishjim 2y ago
I hardly know the first thing about Kubernetes or cloud, so maybe you can help explain something to me: There's another Kubernetes post on the front page of HN at the moment, where they complain it's too complex and they had to stop using it. The comments are really laying into the article author because they used almost 50 clusters. Of course they were having trouble, the comments say, if you introduce that much complexity. They should only need one single cluster (maybe also a backup and a dev one at most). That's the whole point. But then here you are saying your team "operates a couple thousand" clusters. If 50 is far too many, and bound to be unmanageable, how is it reasonable to have more than a thousand?
- jalk 2y ago
  Sounds like it's their primary job is to manage clusters for others, which ofc is different from trying to manage your primary service, that you deployed as 50 microservices in individual clusters (didn't read the other article)
- voidfunc 2y ago
  > But then here you are saying your team "operates a couple thousand" clusters. If 50 is far too many, and bound to be unmanageable, how is it reasonable to have more than a thousand? It's not unmanageable to have a couple thousand Kube clusters but you need to have the resources to build a staff and tool chain to support that, which most companies cannot do. Clusters are how we shard our customer workloads (a workload being say a dozen services and a database, a customer may have many workloads spread across the entire fleet). We put between 100 and 150 workloads per cluster. What this gives us is a relatively small impact area if a single cluster becomes problematic as it only impacts the workloads on it.
coldtea 2y ago
>Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy. Sounds like a great KISS solution. Why did it regress into Kubernetes?
- fhke 2y ago
  > Why did it regress into Kubernetes The “KISS solution” didn’t scale to the requirements of modern business. I remember running chef - essentially a complicated ruby script - on 100ks of servers, each of which with their own local daemon & a central management plane orchestrating it. The problem was that if a server failed… it failed, alongside everything on it. Compared to that setup, k8s is a godsend - auto healing, immutable deployments, scaling, etc - and ultimately, you were already running a node agent, API, and state store, so the complexity lift wasn’t noticeable. The problem came about when companies who need to run 5 containers ended up deploying a k8s cluster :-)
khafra 2y ago
Container orchestration is second nature to us SRE's, so it's easy to forget that the average dev probably only knows the syntax for deployments and one or two service manifests. And pods, of course
- pdimitar 2y ago
  A factor for sure, but as a programmer I find that the discoverability of stuff in code is much higher than with k8s. Give me access to a repo full of YAML files and I'm truly and completely lost and wouldn't even know where to begin. YAML is simply not the right tool for this job. Sure you got used to it but that's exactly the point: you had to get used to it. It was not intuitive and it did not come naturally to you.
tombert 2y ago
If I had set up the infrastructure myself I'd probably have a different opinion on all of this stuff, but I came into this job where everything was set up for me. I don't know if it was don't "incorrectly", and I do fear that stating as such might get into territory adjacent to the "no true Scotsman" fallacy. I mostly just think that k8s integration with GCP is a huge pain in the ass, every time I have to touch it it's the worst part of me day.
- fragmede 2y ago
  What about your integration makes it a huge pita?
  - tombert 2y ago
    It's just a lot of stuff; we have a couple hundred services, and when I've had to add shit, it ends up with me updating like two hundred files. Infrastructure as code is great, but lets be honest, most people are not thoroughly reading through a PR with 200+ files. There's of course tpl files to help reduce duplication, and I'm grateful for that stuff when I can get it, but for one reason or another, I can't always do that. It's also not always clear to me which YAML corresponds to which service, though I think that might be more of an issue with our individual setup.
    - fragmede 2y ago
      Yeah ew sounds like a mess. I'm sorry, thanks for sharing. Death by a thousand paper cuts is never fun. The solution in looking at for my problem in this area is to add another layer of indirection and make a config generator so that there's only one config file for users to define their service in, and the program goes off and makes all the necessary changes to the pile of yaml files.
thiht 2y ago
> It's not that complicated if you limit yourself to the core stuff Completely agree. I use Kubernetes (basically just Deployments and CronJobs) because it makes deployments simple, reliable and standard, for a relatively low cost (assuming that I use a managed Kubernetes like GKE where I don’t need to care at all about the Kubernetes engine). Using Kubernetes as a developer is really not that hard, and it gives you no vendor lock-in in practice (every cloud provider has a Kubernetes offer) and easy replication. It’s not the only solution, not the simplest either, but it’s a good one. And if you already know Kubernetes, it doesn’t cost much to just use it.
mmcnl 2y ago
I didn't grow up in the trenches like you did, but nonetheless I think Kubernetes has a very user-friendly API. Imo Kubernetes is only complex because IT infrastructure is complex. It just makes the complexity transparant and manageable. Kubernetes is a friendly guide helping you navigate the waters.
kjs3 2y ago
Maybe it's because I adopted early and have grown with the technology it all just makes sense? Survivorship bias?

theptrk 2y ago

I would pay for the outline of this book.

t-writescode 2y ago

> Admittedly, I've been using Kubernetes since the early days and I manage an Infra team I think this is where the big difference is. If you're leading a team and introduced all good practices from the start, then the k8s and Terraform or whatever config files can never get so very complicated that a Gordian knot isn't created. Perhaps k8s is nice and easy to use - many of the commands certainly are, in my experience. Developers have, over years and decades, learned how to navigate code and hop from definition to definition, climbing the tree and learning the language they're operating in, and most of the languages follow similar-enough patterns that they can crawl around. Configuring a k8s cluster has absolutely none of that knowledge built up; and, reading something that has rough practices is not a good place to learn what it should look like.

paddy_m 2y ago
Thank you. I can always xargs grep for a function name, at worst. Dir() in python at a debugger for other things. With YAML, kubernetes and other devops hotness, I frequently can’t even find the relevant scripts/YAML that are executed nor their codebases. This also happens with configuration based packaging setups. Python hatch in particular, but sometimes node/webpack/rollup/vite.

adastra22 2y ago

I would buy that book.

figassis 2y ago

This, all the sidecars. Use kubernetes to run your app like you would without it, take advantage of the flexibility, avoid the extra complexity. Service discovery sidecars? Why not just use the out of the box dns features?

tommica 2y ago
Because new people don't know better - I've never used k8s, but have seen sidecars being promoted as a good thing, so I might have used them
- namaria 2y ago
  Maybe the "I've heard about" approach to tooling is the problem here?

stiray 2y ago

I would buy the book. Just translate all "new language" concepts into well known concepts from networking and system administration. It would be best seller. If I would only have a penny for each time I wasted hours trying to figure out what something in "modern IT" is, just to figure out that I already knew what it is, but it was well hidden under layers of newspeak...

radicalbyte 2y ago
The book I read on K8S written by a core maintainer made is very clear.
- schnirz 2y ago
  Which book would that be, out of interest?
  - radicalbyte 2y ago
    Kubernetes in Action
    - xorcist 2y ago
      A book from 2017? Is that still relevant to understand a modern Kubernetes cluster? The CNCF ecosystem looked a lot different back then.
      - radicalbyte 2y ago
        Yup it's a great introduction as you can fly through it (3-4 hours to read most of it), at least if you already have a grasp of networking, linux, processes, threads, containers etc. Then you can hit other resources (in my case working with a team who've been using K8S for a few years). If you (or anyone else) has suggestions for something newer and covering more than just the core (like various different components you can use, Helm, Argo, ISTIO etc etc) then I'd appreciate it :-)
      - 2y ago

AtlasBarfed 2y ago

So you don't run any databases in those thousands of clusters? To your point, and I have not used k8s I just started to research it when my former company was thinking about shoehorning cassandra into k8s... But there was dogma around not allowing access to VM command exec via kubectl, while I basically needed it in the basic mode for certain one-off diagnosis needs and nodetool stuff... And yes, some of the floated stuff was "use sidecars" which also seemed to architect complexity for dogma's sake.

voidfunc 2y ago
> So you don't run any databases in those thousands of clusters? We do, but not of the SQL variety (that I am aware of). We have persistent key-value and document store databases hosted in these clusters. SQL databases are off-loaded to managed offering's in the cloud. Admittedly, this does simplify a lot of problems for us.
- tayo42 2y ago
  How much data? I keep hearing k8s isn't usable becasue sometimes there is to much data and it can't be moved around.
  - pletnes 2y ago
    The simplest approach I’m aware of is to create the k8s cluster and databases in the same datacenter / availability zone.
  - darkstar_16 2y ago
    In the managed k8s space, the data is on a PVC in the same availability zone as the node it is being mounted on. If the node dies, the volume is just mounted on to a new node in the same zone. There is no data movement required.
  - eek04_ 2y ago
    While I've not played with k8, I did run stuff in Google's Borg for a very long while, and that has a similar architecture. My team was petabyte scale and we were far from the team with the largest footprint. So it is clearly possible to handle large scale data in this type of architecture.

mitjam 2y ago

Absolutely, a reasonably sophisticated scalable app platform looks like a half-baked and undocumented reimagination of Kubernetes. Admittedly: The ecosystem is huge and less is more in most cases, but the foundation is cohesive and sane. Would love to read the k8s for greybeards book.

coldtea 2y ago
>Absolutely, a reasonably sophisticated scalable app platform looks like a half-baked and undocumented reimagination of Kubernetes. Or maybe Kubernetes looks like a committee designed, everything and the kitchen sink, over-engineered, second system effect, second system effect suffering, YAGNI P.O.S., that only the kind of "enterprise" mindset that really enjoyed J2EE in 2004 and XML/SOAP vs JSON/REST would love...
- p_l 2y ago
  You misspelled DCOS /s

BiteCode_dev 2y ago

Would buy. But you probably should teach a few live courses before writing it because of expert blindness. Otherwise you will miss thr mark. Would pay for a decent remote live course intro.

awestroke 2y ago

I would buy that book in a heartbeat. All documentation and guides on kubernetes seem to assume you already know why things are done a certain way

DanielHB 2y ago

> What does fucking kill me in the Kubernetes ecosystem is the amount of add-on crap that is pitched as "necessary". Sidecars... so many sidecars. Yeah it is the same with terraform modules, I was trying to argument at a previous job that we should stick to a single module (the cloud provider module) but people just love adding crap if it saves them 5 lines of configuration. Said crap of course adding tons of unnecessary resources in your cloud that no one understands.

karmarepellent 2y ago

Agreed. The best thing we did back when we ran k8s clusters, was moving a few stateful services to dedicated VMs and keep the clusters for stateless services (the bulk) only. Running k8s for stateless services was an absolute bliss. At that time stateful services were somewhat harder to operate on k8s because statefulness (and all that it encapsulates) was kinda full of bugs. That may certainly have changed over the last few years. Maybe we just did it wrong. In any case if you focused on the core parts of k8s that were mature back then, k8s was (and is) a good thing.

jq-r 2y ago

Those "necessary" add-ons and sidecars are out of control, but its the people problem. I'm part of the infra team and we manage just couple of k8s clusters, but those are quite big and have very high traffic load. The k8s + terraform code is simple, with no hacks, reliable and easy to upgrade. Our devs love it, we love it too and all of this makes my job pleasant and as little stressful as possible. But we recently hired a staff engineer to the team (now the most senior) and the guy just cannot rest still. "Oh we need a service mesh because we need visibility! I've been using it on my previous job and its the best thing ever." Even though we have all the visibility/metrics that we need and never needed more than that. Then its "we need a different ingress controller, X is crap Y surely is much better!" etc. So its not inexperienced engineers wanting newest hotness because they have no idea how to solve stuff with the tools they have, its sometimes senior engineers trying to justify their salary, "seniority" by buying into complexity as they try to make themselves irreplaceable.

alienchow 2y ago
How do you scale mTLS ops when the CISO comes knocking?
jppittma 2y ago
Service mesh is complicated, but the reason you use it integrate services across clusters. That and it has a bunch of useful reverse proxy features. On the other hand, It took me and 2 other guys two evenings of blood, sweat, and tears to understand what the fuck a virtual service actually does. It’s not strictly necessary, but if you’ve had to put in the work elsewhere, I’d use it.
cyberpunk 2y ago
To be fair, istio and cilium are extremely useful tools to have under your belt. There’s always a period of “omgwhat” when new senior engineers join and they want to improve things. There’s a short window between joining and getting bogged into a million projects where this is possible. Embrace it I recon.
- p_l 2y ago

carlmr 2y ago

>expert blindness at work. >It's not that complicated if you limit yourself to the core stuff. Isn't this the core problem with a lot of technologies. There's a right way to use it, but most ways are wrong. An expert will not look left and right anymore, but to anyone entering the technology with fresh eyes it's a field with abundance of landmines to navigate around. It's simply bad UX and documentation. It could probably be better. But now it's too late to change everything because you'd annoy all the experts. >There's way too much vendor garbage surrounding the ecosystem Azure has been especially bad in this regard. Poorly documented in all respects, too many confusing UI menus that have similar or same names and do different things. If you use Azure Kubernetes the wrapper makes it much harder to learn the "core essentials". It's better to run minkube and get to know k8s first. Even then a lot of the Azure stuff remains confusing.

wruza 2y ago
This and a terminology rug pull. You wanted to upload a script and install some deps? Here’s your provisioning genuination frobnicator tutorial, at the end of which you’ll learn how to maintain the coalescing encabulation for your appliance unit schema, which is needed for automatic upload. It always feels like thousands times bigger complexity (just in this part!) than your whole project.
- jterrys 2y ago
  You nailed it. Genuinely the most frustrating part about learning kubernetes... is just realizing that whatever the fuck they're talking about is a fancy wrapper for a concept that's existed since 90s.
rbanffy 2y ago
> There's a right way to use it, but most ways are wrong. This is my biggest complaint. There is no simple obvious way to set it up. There is no "sane default" config. > It's better to run minkube and get to know k8s first. Indeed. It should be trivial to set up a cluster from bare metal - nothing more than a `dnf install` and some other command to configure core functionality and to join machines into that cluster. Even when you go the easy way (with, say, Docker Desktop) you need to do a lot of steps just to have an ingress router.

pas 2y ago

My problem is the brittleness. Sure, I am playing with fire (k3s, bare metal, cilium, direct assigned IP to Ingresses), but a few weeks ago on one cluster suddenly something stopped working in the external IP -> internal cluster IP network path. (And after a complete restart things got worse. Oops. Well okay time to test backups.)

cryptonym 2y ago

K8S is a beast and the ecosystem is wild. Newcomer won't know how to proceed to keep things simple, while still understanding everything that is being used.

quietbritishjim 2y ago

I hardly know the first thing about Kubernetes or cloud, so maybe you can help explain something to me: There's another Kubernetes post on the front page of HN at the moment, where they complain it's too complex and they had to stop using it. The comments are really laying into the article author because they used almost 50 clusters. Of course they were having trouble, the comments say, if you introduce that much complexity. They should only need one single cluster (maybe also a backup and a dev one at most). That's the whole point. But then here you are saying your team "operates a couple thousand" clusters. If 50 is far too many, and bound to be unmanageable, how is it reasonable to have more than a thousand?

jalk 2y ago
Sounds like it's their primary job is to manage clusters for others, which ofc is different from trying to manage your primary service, that you deployed as 50 microservices in individual clusters (didn't read the other article)
voidfunc 2y ago
> But then here you are saying your team "operates a couple thousand" clusters. If 50 is far too many, and bound to be unmanageable, how is it reasonable to have more than a thousand? It's not unmanageable to have a couple thousand Kube clusters but you need to have the resources to build a staff and tool chain to support that, which most companies cannot do. Clusters are how we shard our customer workloads (a workload being say a dozen services and a database, a customer may have many workloads spread across the entire fleet). We put between 100 and 150 workloads per cluster. What this gives us is a relatively small impact area if a single cluster becomes problematic as it only impacts the workloads on it.

coldtea 2y ago

>Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy. Sounds like a great KISS solution. Why did it regress into Kubernetes?

fhke 2y ago
> Why did it regress into Kubernetes The “KISS solution” didn’t scale to the requirements of modern business. I remember running chef - essentially a complicated ruby script - on 100ks of servers, each of which with their own local daemon & a central management plane orchestrating it. The problem was that if a server failed… it failed, alongside everything on it. Compared to that setup, k8s is a godsend - auto healing, immutable deployments, scaling, etc - and ultimately, you were already running a node agent, API, and state store, so the complexity lift wasn’t noticeable. The problem came about when companies who need to run 5 containers ended up deploying a k8s cluster :-)

khafra 2y ago

Container orchestration is second nature to us SRE's, so it's easy to forget that the average dev probably only knows the syntax for deployments and one or two service manifests. And pods, of course

pdimitar 2y ago
A factor for sure, but as a programmer I find that the discoverability of stuff in code is much higher than with k8s. Give me access to a repo full of YAML files and I'm truly and completely lost and wouldn't even know where to begin. YAML is simply not the right tool for this job. Sure you got used to it but that's exactly the point: you had to get used to it. It was not intuitive and it did not come naturally to you.

tombert 2y ago

If I had set up the infrastructure myself I'd probably have a different opinion on all of this stuff, but I came into this job where everything was set up for me. I don't know if it was don't "incorrectly", and I do fear that stating as such might get into territory adjacent to the "no true Scotsman" fallacy. I mostly just think that k8s integration with GCP is a huge pain in the ass, every time I have to touch it it's the worst part of me day.

fragmede 2y ago
What about your integration makes it a huge pita?
- tombert 2y ago
  It's just a lot of stuff; we have a couple hundred services, and when I've had to add shit, it ends up with me updating like two hundred files. Infrastructure as code is great, but lets be honest, most people are not thoroughly reading through a PR with 200+ files. There's of course tpl files to help reduce duplication, and I'm grateful for that stuff when I can get it, but for one reason or another, I can't always do that. It's also not always clear to me which YAML corresponds to which service, though I think that might be more of an issue with our individual setup.
  - fragmede 2y ago
    Yeah ew sounds like a mess. I'm sorry, thanks for sharing. Death by a thousand paper cuts is never fun. The solution in looking at for my problem in this area is to add another layer of indirection and make a config generator so that there's only one config file for users to define their service in, and the program goes off and makes all the necessary changes to the pile of yaml files.

thiht 2y ago

> It's not that complicated if you limit yourself to the core stuff Completely agree. I use Kubernetes (basically just Deployments and CronJobs) because it makes deployments simple, reliable and standard, for a relatively low cost (assuming that I use a managed Kubernetes like GKE where I don’t need to care at all about the Kubernetes engine). Using Kubernetes as a developer is really not that hard, and it gives you no vendor lock-in in practice (every cloud provider has a Kubernetes offer) and easy replication. It’s not the only solution, not the simplest either, but it’s a good one. And if you already know Kubernetes, it doesn’t cost much to just use it.

mmcnl 2y ago

I didn't grow up in the trenches like you did, but nonetheless I think Kubernetes has a very user-friendly API. Imo Kubernetes is only complex because IT infrastructure is complex. It just makes the complexity transparant and manageable. Kubernetes is a friendly guide helping you navigate the waters.

kjs3 2y ago

Maybe it's because I adopted early and have grown with the technology it all just makes sense? Survivorship bias?