11 ms·

Did K8s reinvent VMs though? People keep forgetting that vm!=container and the security posture is completely different. Also, for the amounts of complexity K8s

by x86x87 4y ago

Did K8s reinvent VMs though? People keep forgetting that vm!=container and the security posture is completely different. Also, for the amounts of complexity K8s adds it's not worth the overhead in 95% of cases (i get it that it's cool and everyone and their dog want to claim they have experience with it - I am affraid that once the "movement" fizzles out we're going to be stuck with a lot of things that are over-overengineered)

deleted 4y ago
[deleted]
k8sToGo 4y ago
Do you know kubernetes or are you just jumping on the hate train? Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?
- jmspring 4y ago
  He has a point, containers and VMs are not the same.
  - k8sToGo 4y ago
    Yes that part is true. I was not referring to that part.
- de6u99er 4y ago
  > Do you know kubernetes or are you just jumping on the hate train? It seems rather that you don't know what kubernetes is. His comment is justified. You can put your pitchfork back. > Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines? It seems that you don't understand ehat he's trying to say.
  - k8sToGo 4y ago
    >It seems rather that you don't know what kubernetes is. The company, for which I helped migrating to k8s, pays me 6 figures to keep maintaining it, thinks otherwise (also look at my username ;) ). All of our developers love it because it replaced some weird hacked together deployment stuff. Which part do you think he is right or wrong? It just seems that in every thread people moan about k8s. But it seems that it's usually people who are overwhelmed by it, which is a legitimate reason not to use it. But just saying that it's generally bad is what starting to annoy me a bit.
    - packetized 4y ago
      I dare say that it’s annoying you as a result of cognitive dissonance about your employer paying six figures to migrate to it. If you’re being paid well, obviously you’ll be annoyed by concepts contrary to your work. e: edits are hated, but the downvotes prove out: Kubernetes is simply OpenStack for hipsters.
      - actually_a_dog 4y ago
        > It is difficult to get a man to understand something when his salary depends upon his not understanding it. - Upton Sinclair.
    - lovehashbrowns 4y ago
      That was pretty much my experience. From a distance, it looks over-engineered. Once I started using it in my personal projects and at work in production, it makes a lot of sense and it’s far easier than people make it out to be. I’m currently helping my employer switch everything to kubernetes and we’re replacing a lot of really, truly over-engineered trash. I will say I do hate yaml but there aren’t a ton of choices, tbh.
    - k8sToDie 4y ago
      Hi - since you're on credentials, I hold CKA, CKAD, and CKS. I have spent my entire Kubernetes career unfucking the people three degrees removed from you who hear advocacy like this and say wow, yeah, the people with gripes about Kubernetes are just moaning and overwhelmed. Meanwhile, a team at an unnamed government agency just allowed Accenture to con them into turning multiplayer Excel, a process which used to take two humans about a week, into an $18 million annual AWS system funded by the U.S. taxpayer. Because of choices-du-jour, including Spark and Kubernetes. It just seems that in every thread people moan about people moaning about Kubernetes and then project their interpretation of what that moaning is after starting from the presupposition that it's moaning. If you'd instead listen to the other side instead of the basically-crypto-advocacy-train-choo-choo of "if you have gripes, you just don't get it," you'd understand the cons a little better. Such as my entire decade of professional Kubernetes consulting being every single computing project, including in the Fortune 500, the Department of Defense, and at second-tier VC startups in the Bay Area, being a complete operational disaster that's bleeding capital like it's free. Capital is no longer free. Read the news. I now wield my certifications to advise against Kubernetes in any capacity because it is a strategic risk unless resourced exactly right and funded exactly right (which is impossible to justify on infra budgets), simply because of the experience I have watching people like you come in, explain to leadership that it's fine, and then ride that salary until it doesn't suit you anymore and leave a system that takes 5-6 FTEs to comfortably administer on whoever gets to pick up the ball. For every comment you see on Hacker News where someone like you is saying "yeah, Kubernetes runs great for us," they're (a) leaving out the operational toil, particularly in the break-fix department where the entire Unix paradigm changes and (b) not realizing that there are five other teams like them that are completely underwater while trying to put out their overengineered tire fire. And remember, nearly every project using Kubernetes is trying to do multiplayer Excel. That's it. And the more we invest in trying to make the world's most overcomplex computing system do that use case even harder, the less we think about "isn't there a simpler way to do all of this from first principles?"
- benreesman 4y ago
  I actually know k8 as well as another Borg-alike (Tupperware) that actually runs at Borg-like scale. GP is absolutely right that it is insane overkill with no conceivable way to pay back the complexity investment in all but a few niche cases. Working at a FAANG isn’t the only niche where full-Kube makes sense, there are others, but it’s a niche thing, with the niche being roughly: I rack and run physical gear on a limited number of SKUs at “cattle not pets” scale. If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust than any half-assed Borg-alike setup most people would come up with. They’re happy to take your money if you yo dawg like imaginary computers so I put a docker in your Xen VM so you can Kubernetes while you pay me big bucks, but that’s because they’re not running charities.
  - FridgeSeal 4y ago
    Ok, let’s agree that K8s is overkill. What commodity tool should I be replacing it with that gives me: * super for heterogenous workloads (web servers, big batch loads, streaming workloads) * autoscaling the workloads * autoscaling the cluster * auto-restarting workloads * basically automatic networking * the ability to have all that without the need to setup separate infrastructure, and to have the orchestrator move workloads around to optimise machine use * the ability to hire people with experience in it already * ability to run stateful workloads, and have disks managed and moved for you * the ability to do all that by just providing a config * the ability to do all that without having to let people have actual access to the machines/OS * per container/workload topology constraints and requirements, that are network and data centre aware, without having to write any code * have access to a wide range of tooling and docs * be able to drop in workloads that automatically handle log collection, etc. without needing to configure each deployment. * ability to swap out Network ingress/reverse proxy without having to redeploy or reconfigure your applications. * ability to drop in any number of tools to do traffic shadowing, mirroring, splits, etc without reconfiguring or redeploying your applications * ability to do all this without your devs needing to be networking experts. * not locked to a specific cloud provider I’m sure there’s tools out there that cover different subsets of those features, there’s probably tools that do all of them and more. K8s isn’t perfect, but it gives you an awful lot for some reasonable tradeoffs. Sure, if you’re deploying a single web-server/app, you shouldn’t bother with all this; and data centre level stuff probably has tools that make these features look like a cute picnic, but many devs I’ve worked with haven’t the capacity, ability or desire to figure that out. > If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust Sure, but you’ve still got to figure out how you want to setup, monitor, configure all of that, then stitch it together and maintain it, and also not get in the way of your main application development. To me, that’s what K8s gives me: I rely on the cloud provider to give me good machines, good networking, good storage primitives, and to heal/replace them when they fail, and K8s gives me a way to use all of that nicely, without needing to build and maintain it myself, and without my applications needing to know everything.
- x86x87 4y ago
  I know what it is, have worked with k8s, with borg and another half dozend similar systems. Saying that you should use it because people know does not hold water. Also, anecdotally very few people understand k8s (setup and running it). Most people just know how to deploy apps on it (and even there most impressionable are young devs that haven't done things in a preK8s world)
MBCook 4y ago
I thought the article went through the whole evolution from boxes to VMs to containers to K8 very well. It’s so much more than it’s headline.
- codsane 4y ago
  Yeah I’m not sure if it’s the Reddit mentality at play here or actual outage but it just always seems like these types of comments are reacting to a headline, it is unfortunate.
- x86x87 4y ago
  > Docker containers have been deliberately designed to contain just one application. What is a docker container? As a matter of fact: what is a container?
  - MBCook 4y ago
    The article explains that doesn’t it? I think I remember reading it in there at least.
    - x86x87 4y ago
      Here is the fundamental problem. You cannot explain (a lot of people worked with containers and kind of get what they are but nit really) what a container is and point to the article. I've read the article and it confidently talks about docker containers. I can tell you for a fact that just because you're using a docker file and build a image does not mean there is anything special about the container that docker spins up. In fact there is no such thing as a container. We made it up. It's just a bunch of Linux kerned features that are used together creatively. A container is not a replacement for properly securing/isolating your app. Any flaw in any of the mechanism the kernel uses to provide the "containerization" features can end up with all the containers on the node compromized. Also, unless you're really disciplined about how you're building your images you probably already have a bunch of zero days lurking in your deployments. It's all fun and games until we actually need accountability and the buck has to stop with someone.
orwin 4y ago
I am currently overengineering my previous work to run it inside kubernetes. It isn't my decision, i just asked for a VM and the access to ansible, but the chiefs wanted to use kubernetes for everything. I don't think this is that bad tbh. I runs my unit tests on container construction and while it was tedious to develop, mostly because i cannot run docker on my work PC, i understand that it makes it easier to inventory and check the advancement. The management will always love it, so i think kubernetes is here to stay. Not because it solve a lot of engineering issues (not for me anyway, i'd rather run a cron in my VM than declare a kube cronJob), but because it solve a lot of other fondamental issues for the management (inventory, visibility on existing/running projects. And everybody use the same system without using the _same_ system.)
- Already__Taken 4y ago
  > while it was tedious to develop, mostly because I cannot run docker on my work PC Have a look at tilt.dev You can get it to build in-cluster too so you only need the CLI. I've really enjoyed its workflow. Or try podman-remote. Should be easier to get those CLI tools on your work PC.
- CameronNemo 4y ago
  Our k8s based deployments go so much smoother than our VM based deployments (Ansible). We have centralized logging set up for k8s, but not VMs. We have post-update health checks for k8s pods, but not systemd services. Et cetera.
  - orwin 4y ago
    I understand that. But i already have the code to set up logging and health checks (i basically took the code i did at my previous work, it was open source so no issue here). Like i said, it is fine, i'm not mad about it, it took some time to understand but i get it now and while still less efficient, i'm way better at it, and when i finish this project the next one will be easier. Also, it is important that the whole team use the same product: it was a good reason to loose one man-month to get me up to date with the rest of the team and facilitate my next "app".
- 4y ago
pravus 4y ago
I've always said that it reinvented the mainframe by turning it inside-out. I did some work on an IBM 360/370 running VM/CMS and I can see a lot of parallels in how both systems operate. Most of the architecture is an interface that is implemented by some pluggable piece of software and there's also all of the resource scheduling and management which is a first-class component. This makes the system very flexible and reliable. And I think k8s is here to stay. If what you have is a bunch of containers you can easily push that into k8s. It's not any more complex than any other solution you would use to deploy them.
- x86x87 4y ago
  We'll see. Containers are definetly a thing and have their usecase. K8s as a whole? Imho the jury is still out.
  - vbezhenar 4y ago
    Let me tell the truth that I think. I'm k8s newbie and I'm delighted by it and terrified by it at the same time. I really like k8s architecture. It's just miracle architecture that I'd never come with myself. It's very extendable. I'm very inspired by it and I hope to apply it to my own projects some time. But UI is terrible. scopeSelector: matchExpressions: - scopeName: PriorityClass operator: In values: - middle Like really? I thought parsing problem was solved like 50 years ago. Why not `scopeSelector: "priorityClass in [middle]"`. It seems that we forced to write YAML AST instead of proper programming language. It really seems like some intermediate form that's supposed to be output from a proper frontend language but we've stuck with it. Also I'm not sure that I like all this modular nature of k8s. Sometimes it makes sense, for example for CSI. But why should I choose between CNI implementations? I just want best one and that's about it. I want it to be from Google. I don't need to choose between kube-apiserver implementations or kube-scheduler implementations, thanks god. Also the whole kubeadm story seems half-baked. It works, but far from ideal. I guess it'll get better with time. But those are small complains. Like I think that it has a good potential to improve. But there's nothing to replace it. Even if someone will build some lambda swarm which is better than k8s, nobody's going to migrate to it, because k8s is offered by so many cloud providers which made investments and are not going to move to other solutions in the foreseeable time.
jstx1 4y ago
I don't think it's fizzling out, it's kind of becoming a basic standard. And if your use case is simple enough to not need Kubernetes, then doing it with Kubernetes isn't that much more complicated either.
- jiggawatts 4y ago
  A standard in the Linux world. Windows doesn’t have “distributions”, so there’s less need for containers in that space. Microsoft uses Service Fabric internally for the Azure back-end, which covers most of the other use-cases of Kubernetes.
  - Jenk 4y ago
    Service Fabric needs to die. It is an abomination on software. Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in. No thanks. Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in. > Windows doesn’t have “distributions”, so there’s less need for containers in that space. I don't follow. What does distributions have to do with need for containers?
    - jiggawatts 4y ago
      > What does distributions have to do with need for containers? A lot (but not everything). It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc... In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters. Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself. Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable. [1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is. [2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc https://www.youtube.com/watch?v=Pzl1B7nB9Kc
gofreddygo 4y ago
Did JVM reinvent VM? Is it a VM or a container? No one (not enough people?) asked (cared?) for this about the JVM for like 25 years till Linux Cgroups came around and Docker capitalized on it. Its not VM v/s container question that we care about. (I am aware this is a very poor comparison but so is comparing a vm to a container, which is my point) K8S brings complexity yes, and in a consistent, predictable way. The biggest difference between a k8s based shop vs a DIY VM/Container shop is the former can (in theory) hire some people with intermediate K8S experience, and have them sort out the mess that the people without that experience (or more typically the ones who don't have clear requirements) may have created. With k8s it becomes more predictable to find the boundaries and start pulling things apart in production and make incremental changes. Without k8s, making changes to a production setup is a nightmare that folks who have done that sort of thing don't ever want to repeat.
- aprdm 4y ago
  That’s true if your shop didn’t create a bunch of custom controllers or apis inside the k8s world… or forked it.
  - koffiezet 4y ago
    Any decent k8s engineer worth his salt should find it pretty easy to navigate this, since how you deal with that is pretty standardised. I've written custom operators and submitted upstream patches to existing-ones, it's not that hard once you know what you're dealing with conceptually.
    - x86x87 4y ago
      So your point is that once you learn how to use the tool it's easier to use the tool? Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer?
      - koffiezet 4y ago
        > So your point is that once you learn how to use the tool it's easier to use the tool? Kubernetes is not just "a tool", it's a platform. So if you know the k8s fundamentals, being dropped in an unfamiliar environment should not be a problem. It's the same as knowing your way around a Linux system, you might not be familiar with that specific distribution you're thrown into, but you should be able to explore the system and figure stuff out. > Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer? Any engineer who claims to know a bit more than the basics of k8s? It being a software, "devops", SRE, "platform", ... engineer doesn't really matter, I've been labeled all of the above. I'll never claim k8s is perfect, I have plenty of issues with it, but very few of them are an architectural problem, and I've yet to encounter a platform that attempts to address all the issues k8s does. But you seem to be very critical of it, so what would you propose as a viable alternative?
pacarvalho 4y ago
Agree. As someone that has had to migrate systems from K8s to Elastic Beanstalks or equivalent, there are definitely situations where K8s is overkill and simpler systems can work better.
dijonman2 4y ago
I’m not too proficient with containers so maybe I am ignorant, but I view containers more like a package manager. A proper golden image pipeline can work well, and you can rightsize your workloads easily. All of the additional features such as ingress controllers and other complexities make debugging a bit obtuse, especially if you are not extremely proficient. I could be an old curmudgeon but simplicity is my priority.
- lamontcg 4y ago
  I am also an old curmudgeon. I think its vastly more useful to think of a container as an RPM with layered tarballs, a chroot/jail, and some cgroup isolation. Thinking of them as a virtual machine is the wrong metaphor. Kubernetes replaces the thing we were doing in 2003 blasting around tarballs of software with scp and some scripts to start and stop things, and is the end result of evolving from that point to a much more dynamic one and with cgroups providing the ability to have software teams not quite clobber each other's workloads while running on the same hardware. The problems it solves overlaps with many of the problems that VMs were addressing, but it is like looking at two different species that look quite similar but are in completely different animal families and are just similar because of the similar environment they evolved in.
- vbezhenar 4y ago
  How ingress controller makes debugging obtuse? You're using nginx in front of your services anyway, right? Ingress controller is just nginx with some tool autogenerating its config from kubernetes objects. You can view nginx logs easily.
aceBacker 4y ago
Kubernetes isn't that complicated, it's distributed computing that is complicated. Kubernetes makes it as simple as it can be without oversimplifying it.
- josh2600 4y ago
  Just wanted to chime in and hard agree on this. I remember the world where people were trying to build things like kubernetes before it existed. There was a period where enough people had left google post ipo to realize that they needed something like borg but nothing really existed like that outside of elgoog. Twitter had Aurora, then elsewhere mesos and mesosphere popped up and it seemed like there were going to be about 100 different frameworks until Kubernetes dropped and basically ate the industry alive. K8s feels pretty stable from where I’m sitting.
  - SuperQue 4y ago
    Not to mention all the various in-house systems in the 2010-era. I worked at a small place back in 2013 that had built their own container platform about a year before Docker was released. It was more like workqueue than borg. But it worked well enough. Eventually we got tired of maintaining our own snowflake scheduler and switched to Kubernetes. But I'm also old enough to remember pre-borg systems used in HPC. Maui, TORQUE, etc. I wasn't surprised batch scheduling finally got used for service deployment.
    - tjr225 4y ago
      Even in the post docker world people were trying to build container platforms on docker that resembled other things but had their own issues. I definitely had to maintain one of those for three years.
      - raffraffraff 4y ago
        Same. And my previous employer still runs the "thing" we built.
  - 4y ago
zozbot234 4y ago
"Containers" are just a pattern, the real technology behind it is kernel namespaces (plus ordinary process boundaries). (For that matter container security is still half-baked in many ways. The kernel namespacing has been added after-the-fact and it can't really be trusted to make an effective "sandbox" on its own.)
- vbezhenar 4y ago
  Everyone keep saying that. But containers are battle-tested for years at this point. I think that all low-hanging vulnerabilities have been fixed. Like you can expect that your container IS secure if your system is up to date. Every container escape is kernel bug which is fixed ASAP. And 0-days can be applied to everything, virtualization will not help you with that, there're 0-days in virtualization layers. Yes, surface of attack is significantly larger with containers and chance of 0-day is larger, but for most companies which don't really run code from attackers in their containers, this danger is over-stated. Container security is good enough for most workloads.
  - zozbot234 4y ago
    > Everyone keep saying that. Yes, and that "everyone" include kernel developers. That's my point. Now, does this mean containers are totally useless, of course not. Especially if you can prevent running attacker code via other means.
  - rfoo 4y ago
    > Every container escape is kernel bug Well, not so fast. Ubuntu apport (its coredump collector and crash reporter) had a series of bugs over multiple years which could be used to escape from containers. This is all because coredump filter is still a global thing and not a per namespace one. I agree with you it's actually more secure than people thought it is. A rule of thumb: just treat it as same as "running multiple applications with different user". Yeah, not a real sandbox. But nowadays also not many low hanging fruits, either.
Pakdef 4y ago
> Also, for the amounts of complexity K8s adds it's not worth the overhead in 95% of cases probably true for the cloud too
otabdeveloper4 4y ago
> I am affraid that once the "movement" fizzles out we're going to be stuck with a lot of things that are over-overengineered Have no fear, these k8s specialists are going to be extremely in demand for getting rid of legacy k8s hell. (Not even joking, I'd pay good money for someone to get rid of the k8s cluster previous consultants have cobbled together.)
seabrookmx 4y ago
I know most configurations don't use it, but k8s supports different runtimes and today you can use Kata or firecracker to run each pod in it's own "microvm." You then get the security benefits of a "real" VM while still distributing your software as an OCI (Docker) container, and can operate it like any other k8s cluster.

by x86x87 4y ago

deleted 4y ago

[deleted]

k8sToGo 4y ago

Do you know kubernetes or are you just jumping on the hate train? Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?

jmspring 4y ago
He has a point, containers and VMs are not the same.
- k8sToGo 4y ago
  Yes that part is true. I was not referring to that part.
de6u99er 4y ago
> Do you know kubernetes or are you just jumping on the hate train? It seems rather that you don't know what kubernetes is. His comment is justified. You can put your pitchfork back. > Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines? It seems that you don't understand ehat he's trying to say.
- k8sToGo 4y ago
  >It seems rather that you don't know what kubernetes is. The company, for which I helped migrating to k8s, pays me 6 figures to keep maintaining it, thinks otherwise (also look at my username ;) ). All of our developers love it because it replaced some weird hacked together deployment stuff. Which part do you think he is right or wrong? It just seems that in every thread people moan about k8s. But it seems that it's usually people who are overwhelmed by it, which is a legitimate reason not to use it. But just saying that it's generally bad is what starting to annoy me a bit.
  - packetized 4y ago
    I dare say that it’s annoying you as a result of cognitive dissonance about your employer paying six figures to migrate to it. If you’re being paid well, obviously you’ll be annoyed by concepts contrary to your work. e: edits are hated, but the downvotes prove out: Kubernetes is simply OpenStack for hipsters.
    - actually_a_dog 4y ago
      > It is difficult to get a man to understand something when his salary depends upon his not understanding it. - Upton Sinclair.
  - lovehashbrowns 4y ago
    That was pretty much my experience. From a distance, it looks over-engineered. Once I started using it in my personal projects and at work in production, it makes a lot of sense and it’s far easier than people make it out to be. I’m currently helping my employer switch everything to kubernetes and we’re replacing a lot of really, truly over-engineered trash. I will say I do hate yaml but there aren’t a ton of choices, tbh.
  - k8sToDie 4y ago
    Hi - since you're on credentials, I hold CKA, CKAD, and CKS. I have spent my entire Kubernetes career unfucking the people three degrees removed from you who hear advocacy like this and say wow, yeah, the people with gripes about Kubernetes are just moaning and overwhelmed. Meanwhile, a team at an unnamed government agency just allowed Accenture to con them into turning multiplayer Excel, a process which used to take two humans about a week, into an $18 million annual AWS system funded by the U.S. taxpayer. Because of choices-du-jour, including Spark and Kubernetes. It just seems that in every thread people moan about people moaning about Kubernetes and then project their interpretation of what that moaning is after starting from the presupposition that it's moaning. If you'd instead listen to the other side instead of the basically-crypto-advocacy-train-choo-choo of "if you have gripes, you just don't get it," you'd understand the cons a little better. Such as my entire decade of professional Kubernetes consulting being every single computing project, including in the Fortune 500, the Department of Defense, and at second-tier VC startups in the Bay Area, being a complete operational disaster that's bleeding capital like it's free. Capital is no longer free. Read the news. I now wield my certifications to advise against Kubernetes in any capacity because it is a strategic risk unless resourced exactly right and funded exactly right (which is impossible to justify on infra budgets), simply because of the experience I have watching people like you come in, explain to leadership that it's fine, and then ride that salary until it doesn't suit you anymore and leave a system that takes 5-6 FTEs to comfortably administer on whoever gets to pick up the ball. For every comment you see on Hacker News where someone like you is saying "yeah, Kubernetes runs great for us," they're (a) leaving out the operational toil, particularly in the break-fix department where the entire Unix paradigm changes and (b) not realizing that there are five other teams like them that are completely underwater while trying to put out their overengineered tire fire. And remember, nearly every project using Kubernetes is trying to do multiplayer Excel. That's it. And the more we invest in trying to make the world's most overcomplex computing system do that use case even harder, the less we think about "isn't there a simpler way to do all of this from first principles?"
benreesman 4y ago
I actually know k8 as well as another Borg-alike (Tupperware) that actually runs at Borg-like scale. GP is absolutely right that it is insane overkill with no conceivable way to pay back the complexity investment in all but a few niche cases. Working at a FAANG isn’t the only niche where full-Kube makes sense, there are others, but it’s a niche thing, with the niche being roughly: I rack and run physical gear on a limited number of SKUs at “cattle not pets” scale. If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust than any half-assed Borg-alike setup most people would come up with. They’re happy to take your money if you yo dawg like imaginary computers so I put a docker in your Xen VM so you can Kubernetes while you pay me big bucks, but that’s because they’re not running charities.
- FridgeSeal 4y ago
  Ok, let’s agree that K8s is overkill. What commodity tool should I be replacing it with that gives me: * super for heterogenous workloads (web servers, big batch loads, streaming workloads) * autoscaling the workloads * autoscaling the cluster * auto-restarting workloads * basically automatic networking * the ability to have all that without the need to setup separate infrastructure, and to have the orchestrator move workloads around to optimise machine use * the ability to hire people with experience in it already * ability to run stateful workloads, and have disks managed and moved for you * the ability to do all that by just providing a config * the ability to do all that without having to let people have actual access to the machines/OS * per container/workload topology constraints and requirements, that are network and data centre aware, without having to write any code * have access to a wide range of tooling and docs * be able to drop in workloads that automatically handle log collection, etc. without needing to configure each deployment. * ability to swap out Network ingress/reverse proxy without having to redeploy or reconfigure your applications. * ability to drop in any number of tools to do traffic shadowing, mirroring, splits, etc without reconfiguring or redeploying your applications * ability to do all this without your devs needing to be networking experts. * not locked to a specific cloud provider I’m sure there’s tools out there that cover different subsets of those features, there’s probably tools that do all of them and more. K8s isn’t perfect, but it gives you an awful lot for some reasonable tradeoffs. Sure, if you’re deploying a single web-server/app, you shouldn’t bother with all this; and data centre level stuff probably has tools that make these features look like a cute picnic, but many devs I’ve worked with haven’t the capacity, ability or desire to figure that out. > If you’re on a cloud provider? You’ve already got secure, restartable, restorable, on-demand-growable, discoverable, fail-over-able, imaginary machines a zillion times more robust Sure, but you’ve still got to figure out how you want to setup, monitor, configure all of that, then stitch it together and maintain it, and also not get in the way of your main application development. To me, that’s what K8s gives me: I rely on the cloud provider to give me good machines, good networking, good storage primitives, and to heal/replace them when they fail, and K8s gives me a way to use all of that nicely, without needing to build and maintain it myself, and without my applications needing to know everything.
x86x87 4y ago
I know what it is, have worked with k8s, with borg and another half dozend similar systems. Saying that you should use it because people know does not hold water. Also, anecdotally very few people understand k8s (setup and running it). Most people just know how to deploy apps on it (and even there most impressionable are young devs that haven't done things in a preK8s world)

MBCook 4y ago

I thought the article went through the whole evolution from boxes to VMs to containers to K8 very well. It’s so much more than it’s headline.

codsane 4y ago
Yeah I’m not sure if it’s the Reddit mentality at play here or actual outage but it just always seems like these types of comments are reacting to a headline, it is unfortunate.
x86x87 4y ago
> Docker containers have been deliberately designed to contain just one application. What is a docker container? As a matter of fact: what is a container?
- MBCook 4y ago
  The article explains that doesn’t it? I think I remember reading it in there at least.
  - x86x87 4y ago
    Here is the fundamental problem. You cannot explain (a lot of people worked with containers and kind of get what they are but nit really) what a container is and point to the article. I've read the article and it confidently talks about docker containers. I can tell you for a fact that just because you're using a docker file and build a image does not mean there is anything special about the container that docker spins up. In fact there is no such thing as a container. We made it up. It's just a bunch of Linux kerned features that are used together creatively. A container is not a replacement for properly securing/isolating your app. Any flaw in any of the mechanism the kernel uses to provide the "containerization" features can end up with all the containers on the node compromized. Also, unless you're really disciplined about how you're building your images you probably already have a bunch of zero days lurking in your deployments. It's all fun and games until we actually need accountability and the buck has to stop with someone.

orwin 4y ago

I am currently overengineering my previous work to run it inside kubernetes. It isn't my decision, i just asked for a VM and the access to ansible, but the chiefs wanted to use kubernetes for everything. I don't think this is that bad tbh. I runs my unit tests on container construction and while it was tedious to develop, mostly because i cannot run docker on my work PC, i understand that it makes it easier to inventory and check the advancement. The management will always love it, so i think kubernetes is here to stay. Not because it solve a lot of engineering issues (not for me anyway, i'd rather run a cron in my VM than declare a kube cronJob), but because it solve a lot of other fondamental issues for the management (inventory, visibility on existing/running projects. And everybody use the same system without using the _same_ system.)

Already__Taken 4y ago
> while it was tedious to develop, mostly because I cannot run docker on my work PC Have a look at tilt.dev You can get it to build in-cluster too so you only need the CLI. I've really enjoyed its workflow. Or try podman-remote. Should be easier to get those CLI tools on your work PC.
CameronNemo 4y ago
Our k8s based deployments go so much smoother than our VM based deployments (Ansible). We have centralized logging set up for k8s, but not VMs. We have post-update health checks for k8s pods, but not systemd services. Et cetera.
- orwin 4y ago
  I understand that. But i already have the code to set up logging and health checks (i basically took the code i did at my previous work, it was open source so no issue here). Like i said, it is fine, i'm not mad about it, it took some time to understand but i get it now and while still less efficient, i'm way better at it, and when i finish this project the next one will be easier. Also, it is important that the whole team use the same product: it was a good reason to loose one man-month to get me up to date with the rest of the team and facilitate my next "app".
4y ago

pravus 4y ago

I've always said that it reinvented the mainframe by turning it inside-out. I did some work on an IBM 360/370 running VM/CMS and I can see a lot of parallels in how both systems operate. Most of the architecture is an interface that is implemented by some pluggable piece of software and there's also all of the resource scheduling and management which is a first-class component. This makes the system very flexible and reliable. And I think k8s is here to stay. If what you have is a bunch of containers you can easily push that into k8s. It's not any more complex than any other solution you would use to deploy them.

x86x87 4y ago
We'll see. Containers are definetly a thing and have their usecase. K8s as a whole? Imho the jury is still out.
- vbezhenar 4y ago
  Let me tell the truth that I think. I'm k8s newbie and I'm delighted by it and terrified by it at the same time. I really like k8s architecture. It's just miracle architecture that I'd never come with myself. It's very extendable. I'm very inspired by it and I hope to apply it to my own projects some time. But UI is terrible. scopeSelector: matchExpressions: - scopeName: PriorityClass operator: In values: - middle Like really? I thought parsing problem was solved like 50 years ago. Why not `scopeSelector: "priorityClass in [middle]"`. It seems that we forced to write YAML AST instead of proper programming language. It really seems like some intermediate form that's supposed to be output from a proper frontend language but we've stuck with it. Also I'm not sure that I like all this modular nature of k8s. Sometimes it makes sense, for example for CSI. But why should I choose between CNI implementations? I just want best one and that's about it. I want it to be from Google. I don't need to choose between kube-apiserver implementations or kube-scheduler implementations, thanks god. Also the whole kubeadm story seems half-baked. It works, but far from ideal. I guess it'll get better with time. But those are small complains. Like I think that it has a good potential to improve. But there's nothing to replace it. Even if someone will build some lambda swarm which is better than k8s, nobody's going to migrate to it, because k8s is offered by so many cloud providers which made investments and are not going to move to other solutions in the foreseeable time.

jstx1 4y ago

I don't think it's fizzling out, it's kind of becoming a basic standard. And if your use case is simple enough to not need Kubernetes, then doing it with Kubernetes isn't that much more complicated either.

jiggawatts 4y ago
A standard in the Linux world. Windows doesn’t have “distributions”, so there’s less need for containers in that space. Microsoft uses Service Fabric internally for the Azure back-end, which covers most of the other use-cases of Kubernetes.
- Jenk 4y ago
  Service Fabric needs to die. It is an abomination on software. Why, why, *WHY* does each SF service need to have some terrible wrapper code that forces it to run in, and only in, SF? Oh ya.. vendor lock-in. No thanks. Yes, I'm aware that MS added support for docker containers in SF. That integration is dreadful. They did about as good a job of it as the WCF team did of shoe-horning "RESTFul Services" in. > Windows doesn’t have “distributions”, so there’s less need for containers in that space. I don't follow. What does distributions have to do with need for containers?
  - jiggawatts 4y ago
    > What does distributions have to do with need for containers? A lot (but not everything). It's important to recognise where the "stable API/ABI boundaries" are. In Linux, the stable boundary is the kernel. Userspace is unstable. There are many distributions, with many small and big differences. Different C runtimes, different folder structures, different configs, modules, etc... In Windows, the stable boundary is the Win32, COM, and .NET APIs, which are in user space. The kernel boundary is not stable[1] however, which also matters. Even Linus Torvalds has trouble distributing his hobby dive-computer software to Linux![2] He has no issues with MacOS and Windows, because they have stable APIs and distribution mechanisms. With Linux, the kernel is stable, but essentially nothing else is. If you want to distribute software to everyone, then this is a complex problem with many moving parts that you have to manage yourself. Containers sidestep this by packaging up the distribution along with the software being deployed. This works because the kernel ABI is stable. [1] In the Windows world you don't really need to package up the "Windows distribution" because there is only one: Microsoft Windows. Conversely, Windows containers aren't truly portable because the Windows kernel ABI isn't stable. However Microsoft changed this with Windows 11 and Server 2022, the kernel ABI is now "stable", or at least sufficiently stable for Server Core containers to be portable for 5+ years as-is. [2] He's complaining about "desktop applications", but the exact same rant would apply to server software also. This one talk made me understand containers and why they're so important in the Linux world: https://www.youtube.com/watch?v=Pzl1B7nB9Kc https://www.youtube.com/watch?v=Pzl1B7nB9Kc

gofreddygo 4y ago

Did JVM reinvent VM? Is it a VM or a container? No one (not enough people?) asked (cared?) for this about the JVM for like 25 years till Linux Cgroups came around and Docker capitalized on it. Its not VM v/s container question that we care about. (I am aware this is a very poor comparison but so is comparing a vm to a container, which is my point) K8S brings complexity yes, and in a consistent, predictable way. The biggest difference between a k8s based shop vs a DIY VM/Container shop is the former can (in theory) hire some people with intermediate K8S experience, and have them sort out the mess that the people without that experience (or more typically the ones who don't have clear requirements) may have created. With k8s it becomes more predictable to find the boundaries and start pulling things apart in production and make incremental changes. Without k8s, making changes to a production setup is a nightmare that folks who have done that sort of thing don't ever want to repeat.

aprdm 4y ago
That’s true if your shop didn’t create a bunch of custom controllers or apis inside the k8s world… or forked it.
- koffiezet 4y ago
  Any decent k8s engineer worth his salt should find it pretty easy to navigate this, since how you deal with that is pretty standardised. I've written custom operators and submitted upstream patches to existing-ones, it's not that hard once you know what you're dealing with conceptually.
  - x86x87 4y ago
    So your point is that once you learn how to use the tool it's easier to use the tool? Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer?
    - koffiezet 4y ago
      > So your point is that once you learn how to use the tool it's easier to use the tool? Kubernetes is not just "a tool", it's a platform. So if you know the k8s fundamentals, being dropped in an unfamiliar environment should not be a problem. It's the same as knowing your way around a Linux system, you might not be familiar with that specific distribution you're thrown into, but you should be able to explore the system and figure stuff out. > Also, what is a k8s engineer? What's the difference between this engineer and a "devops" engineer? This and software engineer? Any engineer who claims to know a bit more than the basics of k8s? It being a software, "devops", SRE, "platform", ... engineer doesn't really matter, I've been labeled all of the above. I'll never claim k8s is perfect, I have plenty of issues with it, but very few of them are an architectural problem, and I've yet to encounter a platform that attempts to address all the issues k8s does. But you seem to be very critical of it, so what would you propose as a viable alternative?

pacarvalho 4y ago

Agree. As someone that has had to migrate systems from K8s to Elastic Beanstalks or equivalent, there are definitely situations where K8s is overkill and simpler systems can work better.

dijonman2 4y ago

I’m not too proficient with containers so maybe I am ignorant, but I view containers more like a package manager. A proper golden image pipeline can work well, and you can rightsize your workloads easily. All of the additional features such as ingress controllers and other complexities make debugging a bit obtuse, especially if you are not extremely proficient. I could be an old curmudgeon but simplicity is my priority.

lamontcg 4y ago
I am also an old curmudgeon. I think its vastly more useful to think of a container as an RPM with layered tarballs, a chroot/jail, and some cgroup isolation. Thinking of them as a virtual machine is the wrong metaphor. Kubernetes replaces the thing we were doing in 2003 blasting around tarballs of software with scp and some scripts to start and stop things, and is the end result of evolving from that point to a much more dynamic one and with cgroups providing the ability to have software teams not quite clobber each other's workloads while running on the same hardware. The problems it solves overlaps with many of the problems that VMs were addressing, but it is like looking at two different species that look quite similar but are in completely different animal families and are just similar because of the similar environment they evolved in.
vbezhenar 4y ago
How ingress controller makes debugging obtuse? You're using nginx in front of your services anyway, right? Ingress controller is just nginx with some tool autogenerating its config from kubernetes objects. You can view nginx logs easily.

aceBacker 4y ago

Kubernetes isn't that complicated, it's distributed computing that is complicated. Kubernetes makes it as simple as it can be without oversimplifying it.

josh2600 4y ago
Just wanted to chime in and hard agree on this. I remember the world where people were trying to build things like kubernetes before it existed. There was a period where enough people had left google post ipo to realize that they needed something like borg but nothing really existed like that outside of elgoog. Twitter had Aurora, then elsewhere mesos and mesosphere popped up and it seemed like there were going to be about 100 different frameworks until Kubernetes dropped and basically ate the industry alive. K8s feels pretty stable from where I’m sitting.
- SuperQue 4y ago
  Not to mention all the various in-house systems in the 2010-era. I worked at a small place back in 2013 that had built their own container platform about a year before Docker was released. It was more like workqueue than borg. But it worked well enough. Eventually we got tired of maintaining our own snowflake scheduler and switched to Kubernetes. But I'm also old enough to remember pre-borg systems used in HPC. Maui, TORQUE, etc. I wasn't surprised batch scheduling finally got used for service deployment.
  - tjr225 4y ago
    Even in the post docker world people were trying to build container platforms on docker that resembled other things but had their own issues. I definitely had to maintain one of those for three years.
    - raffraffraff 4y ago
      Same. And my previous employer still runs the "thing" we built.
- 4y ago

zozbot234 4y ago

"Containers" are just a pattern, the real technology behind it is kernel namespaces (plus ordinary process boundaries). (For that matter container security is still half-baked in many ways. The kernel namespacing has been added after-the-fact and it can't really be trusted to make an effective "sandbox" on its own.)

vbezhenar 4y ago
Everyone keep saying that. But containers are battle-tested for years at this point. I think that all low-hanging vulnerabilities have been fixed. Like you can expect that your container IS secure if your system is up to date. Every container escape is kernel bug which is fixed ASAP. And 0-days can be applied to everything, virtualization will not help you with that, there're 0-days in virtualization layers. Yes, surface of attack is significantly larger with containers and chance of 0-day is larger, but for most companies which don't really run code from attackers in their containers, this danger is over-stated. Container security is good enough for most workloads.
- zozbot234 4y ago
  > Everyone keep saying that. Yes, and that "everyone" include kernel developers. That's my point. Now, does this mean containers are totally useless, of course not. Especially if you can prevent running attacker code via other means.
- rfoo 4y ago
  > Every container escape is kernel bug Well, not so fast. Ubuntu apport (its coredump collector and crash reporter) had a series of bugs over multiple years which could be used to escape from containers. This is all because coredump filter is still a global thing and not a per namespace one. I agree with you it's actually more secure than people thought it is. A rule of thumb: just treat it as same as "running multiple applications with different user". Yeah, not a real sandbox. But nowadays also not many low hanging fruits, either.

Pakdef 4y ago

> Also, for the amounts of complexity K8s adds it's not worth the overhead in 95% of cases probably true for the cloud too

otabdeveloper4 4y ago

> I am affraid that once the "movement" fizzles out we're going to be stuck with a lot of things that are over-overengineered Have no fear, these k8s specialists are going to be extremely in demand for getting rid of legacy k8s hell. (Not even joking, I'd pay good money for someone to get rid of the k8s cluster previous consultants have cobbled together.)

seabrookmx 4y ago

I know most configurations don't use it, but k8s supports different runtimes and today you can use Kata or firecracker to run each pod in it's own "microvm." You then get the security benefits of a "real" VM while still distributing your software as an OCI (Docker) container, and can operate it like any other k8s cluster.