11 ms·

The future of Kubernetes is virtual machines

220 points by MordodeMaru 8y ago

tybit 8y ago
As 2018 comes to a close its time to drag out the hubris and make a bold prediction. The future of Kubernetes is Virtual Machines, not Containers. I’d say that’s less of a prediction than a matter of fact since it’s already happened in 2018 for AWS and GCP. This post is kind of irrelevant for consumers imo, in that the future of Kubernetes is still the container interface, regardless of whether your vendor decides to run it in a container or a VM.
- halbritt 8y ago
  How has it already happened? It's true that EKS and GKE nodes are virtual machines, but that's entirely beside the point of this article which seems to make the point that the workloads running inside of Kubernetes will also be virtual machines. Interesting notion, but I don't see it. The reason for kubesprawl as it is today is a result of the fact that today Kubernetes is hard to tune for disparate workloads which leads most folks to just punt and stand up multiple clusters. That said, people are starting to figure it out and more tools like the vertical pod autoscaler are coming. Eventually the more efficient choice will be to run disparate workloads across the same set of hardware.
k__ 8y ago
I had the impression DevOps ppl would cling to containers as the final form of VMs? But I'm mostly a high level front-end guy doing back-ends with serverless tech only.
stevenacreman 8y ago
It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.
- int0x80 8y ago
  That is one big difference, with big implications.
  - stevenacreman 8y ago
    Just wanted to clarify that Kubernetes was still scheduling containers. Even if VM's are being used to isolate them. It's not all or nothing either. Containerd will support running a mix of containers and kata-containers across workers. For anyone interested in this topic I wrote about some other container runtimes here: https://kubedex.com/kubernetes-container-runtimes/ https://kubedex.com/kubernetes-container-runtimes/
    - amscanne 8y ago
      Note that this is not the case for virtual kubelet-based implementations, and your point here and above are specific to how Kata works (the article is talking more generally).
    - int0x80 8y ago
      Yeah.. I think I see what you say, I mean, the end user interface is the quite the same but this has big implications anyway for the systems design POV, so is not a small thing :)
- jacques_chester 8y ago
  > It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.
  - tptacek 8y ago
    The attack surface argument is debatable, depending on how the system is designers, since virtualization introduces the hypervisor surface.
amscanne 8y ago
An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (if useable at all). Personally, I think the value proposition is tenuous -- you can create VMs today, doing so via the Kubernetes API isn't suddenly revolutionary. Just like throwing something in a "hardware virtualized" instance doesn't suddenly make the whole system secure. Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs).
- serverascode 8y ago
  I agree, I'm not sure the virtual kubelet concept is a great idea overall. It sounds good on the surface, but most of the time these kind of abstractions do more harm then good, as far as I'm concerned. But, I could be wrong. :)
- jacques_chester 8y ago
  > Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs). I think the argument is that rather than the painful (and it will be very painful) and probably incomplete quest to retrofit multi-tenancy into a single-tenancy design, we can introduce multi-tenancy where it basically actually matters: at the worker node. At first glance it's confusing to go from "one master, many nodes" to "one node pool, many masters". But it actually works better on every front. Workload efficiency goes up. Security surface area between masters becomes close to nil. Very cheap VMs are the means to that end. Disclosure: I work for Pivotal and this argument fits our basic doctrine of how Kubernetes ought to be used.
orliesaurus 8y ago
This feels like an infinite russian doll problem, putting k8s inside a bunch of vms I mean.. but I am glad that while everyone is still trying to learn how to properly do k8s these folks are thinking of the next thing
- dehrmann 8y ago
  And running a Java services in the JVM. It's not real infrastructure unless you have at least four layers of VMs.
old-gregg 8y ago
Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".
- serverascode 8y ago
  I think they are quite different solutions, and each benefits from the other. Further, they also tend to run into the same concerns at the about the same time in their life time, eg. concerns around complexity.
- lykr0n 8y ago
  OpenStack's failure is that it's an extremely complex system that looks to be nearly impossible to deploy by yourself. You have to use a Distro that does everything for you- their way. Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you have to use their all-in-one system to have a deployment- and that requires a narrow set of variables which every environment might not have. Which is insane- and will hinder adoption. EDIT. Not 100% correct, see below comments.
  - bamboozled 8y ago
    The issue with Kubernetes isn't that it's hard to setup, but it's difficult to keep operate long term. Simple things like balancing workloads across availability-zones is still not trivial. I've heard many stories of people who tried to run Kubernetes themselves in production and didn't have a great experience for it.
    - halbritt 8y ago
      I've been running it in production for stateful workloads for over a year and it's been going swimmingly. I've never had better uptime or efficiency.
  - serverascode 8y ago
trjordan 8y ago
This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-infrastructure engineers) don't know or care about the precise definition of containers and VMs are. The point of "containers" is that I can define something that acts like an operating system from the ground up, and it builds quickly and runs quickly in production. Kubernetes doesn't win by forcing users to think about VMs. Kubernetes wins by adopting a VM standard that can be built by Dockerfiles. Infra engineers will love it. But besides them? Nobody will care, because Docker for Mac will look the same. [1] Maybe 1 cluster per region? There's a whole fascinating topic that starts with the question "when building a PaaS, do you expose region placement to devs?" The answer implies a ton of stuff about what exactly it's reasonable to expect from a PaaS and how much infrastructure your average dev has to know.
- deleted 8y ago
  [deleted]
- georgebarnett 8y ago
  > It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are operational reasons why having multiple clusters is a good idea. At the simplest level, making a config change and only risking a portion of the infrastructure is an example.
  - trjordan 8y ago
    Config blast radius, hardware sizing, networking partitions, disaster recovery, dev/prod separation -- lots of good reasons to have multiple clusters! As a gut check, most of these reasons apply to AWS AZs as well. If your Kubernetes strategy calls for more than one per region per AWS account, it means that you, organizationally, don't trust containers as much as you trust VMs on AWS. And right now, you're probably right to do so.
ec109685 8y ago
The nested kubernetes idea is interesting (linked to from the post). However, amazon and google aren’t using nested control planes for their infrastructure. Why is a single control plane good for them, but not for a Kubernetes deployment?
jchw 8y ago
"virtual machines" may be the wrong word. Containers is probably still the right word. What we probably want, is actually isolated containers. And hell, you can nearly get that today. Combining Docker with gVisor is a potential solution to the soft tenancy problem as far as I can tell, and Kubernetes supports using it. (And gVisor is by no stretch of the imagination a 'VM' - it is, at best, a tiny hypervisor, and maybe less than that.)
mcguire 8y ago
As someone who has been a sysadmin and system programmer for 25 years and a crusty rat bastard for almost that long, I have to wonder how long it is going to be until someone realizes that a piece of hardware doing a task is more efficient than 27 layers of virtual machines and a long pair of tongs.
- nimbius 8y ago
  amen. At first i figured containers were a good way to run code with difficult dependencies and weird operating environments, but that still doesnt make sense. - systemd and uwsgi for example play well when run as a single user per wsgi application under a single nginx/lb. - php-fpm already handles a ton of overhead from php apps. - ansible deployments called from gitlab-ci can roll out apps just as well as deploying from the registry. then i figured maybe they were on to something with the autoscaling thing...but that seems like a meaningless feature. Every good project already has metrics and forecasting...it would be absurd to think a final product like imgur.com or twitter does not know (down to the byte) how much storage theyll need in 4 months and the potential drivers. auto-scaling infrastructure just betrays the fact that most developers throw resources at load problems instead of waiting for ops to figure out the actual issue.
  - latchkey 8y ago
    I helped build an e-commerce marketplace, which is essentially a shopping cart system that was used by third parties. We had no idea how to predict traffic loads because we had no idea when one of our customers would run a successful campaign. Autoscaling on google appengine was a lifesaver for us. Our very small dev team focused on building features because we had zero devops and we didn't have to carry a pager.
  - rhinoceraptor 8y ago
    I don't think most developers or admins really have the toolset/skills to debug to root cause. Most people see debugging as the process of making problems go away, not as the science of understanding a problem. You need both developer and admin skills, as well as a pretty good understanding of the whole stack - the application, the framework/libraries, the application runtime/support libraries, the database, the kernel and the network.
elsonrodriguez 8y ago
The future of Kubernetes (for some interesting use cases), is virtual machines. For the rest of us the virtual kubelet project represents a bridge between Kubernetes and Serverless. I'd much rather pay my cloud provider to run my k8s workloads (billed by pod requests/limits) than pay for a control plane and three nodes just to run my workloads.
state_less 8y ago
While the use of namespaces seemed a bit discounted in the article, I think part of the problem is that it's an underutilized and poorly conveyed feature, but maybe that's just me? I'd like to see a better out-of-the-box vanilla K8S user management interface in the dashboard. Maybe the default UX on cluster creation is a user/service account that isn't cluster admin that is limited to 1 or more namespaces. There should be a better dashboard to configure and create users that is front and center when you first create a cluster. You have to work a bit to get something you can use to authenticate as cluster admin like role. This should help direct people towards creating more users and isolated namespaces.
- justicezyx 8y ago
  Also gVisor.
- halbritt 8y ago
  My org is using namespaces for tenant isolation. Works quite well.
  - geerlingguy 8y ago
    Namespaces++. Unless you’re just running one app in a Kubernetes cluster (where default == that app), namespaces solve a plethora of problems.
    - halbritt 8y ago
      Resource limits, network policy boundaries, etc.
      - smarterclayton 8y ago
        That’s what we designed them for :) I do think we’ve not explored enough of the per namespace policy stuff though - i’d like both podpreset and a reasonably simple scheduling policy (toleration + node selector control to replace the annotation based system) to make it in, as well as a simpler namespace initialization path so you can more easily lock down the contents of a namespace without having to proxy the create namespace API call.
lightedman 8y ago
Can't handle real machines, so let's virtualize everything despite the fact we don't have the physical machines fixed or sussed-out, yet. Bravo. Now I see why we've got the problems that intel, AMD, and ARM suffer from. Nobody's bothering to fix a single thing and are just passing the buck like typical lazy developers.
mattbillenstein 8y ago
Dammit, most of you reading this do not need any of this shit! Build simple things!
- TuringTest 8y ago
  You don't need them until you're hired by a client with nation-wide deployment requirements, and then you need them.
  - aprdm 8y ago
    Not necessarily. Needless to say containers and kubernetes are pretty new technologies and nation wide deployment requirement existed before docker / k8s. Most of the old big companies aren't using docker or k8s for their core services. They're all using legacy fat apps that are load balanced in baremetal or vms.
    - TuringTest 8y ago
      I doubt a home-made solution to load balancing will be seen as simpler, which is what's being argued by the comment I replied to. The point isn't that you need Kubernetes specifically, it's that requiring a system that scales well is not as uncommon as the OP puts it.
      - mattbillenstein 8y ago
        k8ts is not the only avenue to building scalable systems -- scaling is mostly about architecture, not how you run your software.
        TuringTest 8y ago
        > k8ts is not the only avenue to building scalable systems I had already conceded that. What's your point?
013a 8y ago
I really worry about cost in this future of leaning on something like ACI/Fargate to actually run the containers. An m5.large instance (2vcpu/8gb) costs $70/mo on-demand ($44/mo with a 1 year reservation). A similar Fargate runtime costs $146/mo. A b2ms Azure instance (2vcpu/8gb) costs $60/mo on-demand ($39/mo 1 year reservation). Azure Container Instances at a similar provisioning level costs $176/mo by my calculations. That's not a small difference. That's, like, 3x. Point being, I love virtual-kublet from the perspective of a scale-up just trying to get a product out the door. But for established companies, I still think the core idea of a container on a VM you control is going to rule. Fortunately Kubernetes allows amazingly easy flexibility to switch, and that's a reason why it might be the most important technology created in recent history.
- hedora 8y ago
  Also, the instance prices are already some crazy multiple of buying a machine, racking it, and running dozens of VM’s on it for 5 years. I don’t know the prices for hyperconverged or converged infrastructure off the top of my head, but doubt the amazon rates are even close to competitive unless you’re at a tiny scale (or need a ton of tiny presences in different regions).
markbnj 8y ago
I get the multi-tenancy argument, for certain use cases. I'm not sure I understand the point about greater resource utilization. Presumably they mean workloads can be scheduled more densely given a set of hardware resources, vs. containers... but I'd think that containers would score better on that metric than VMs. Can someone expound? The best part of the piece for me was "kubesprawl," my new favorite word for the week. We've seen it ourselves to some extent, but we are at least aware of it and try to exert some pressure in the other direction. Beyond that I am not particularly bothered by the idea of running lots of clusters for different purposes.
emmelaich 8y ago
I'm picking the eyes out the article here but I had to twitch at this ... > Linux containers were not built to be secure isolated sandboxes (like Solaris Zones or FreeBSD Jails). Instead they’re built upon a shared kernel model ... Solaris Zones and BSD Jails both use a shared kernel. And I'd bet you they were far from perfect in security isolation. Now it may be true that security wasn't Linux containers prime reason for being but we have an existence proof that they can be made secure enough -- anyone can get a trial Openshift container for the price of a login.
peterwwillis 8y ago
I'm convinced that Kubernetes is very good for job security, and not much else. Unless you're a managed services host, you should probably not be running it. Please, for the love of all that is holy, use a cloud services provider if you need K8s-style service features. If you don't, then just cobble together your infrastructure in the simplest way possible that uses DevOps principles, methods and practices.
- rantanplan 8y ago
  Could you elaborate a bit?
  - peterwwillis 8y ago
    K8s is a very complex system. And the more complex the requirements, the more complex one needs to make K8s through additional software that isn't baked in. Complex systems are costly to run, but more importantly, they're costly to maintain due to the typical level of service required, the number of employees needed to maintain it, and the amount of specialized knowledge required. The system is also under constant maintenance due to its short release cycle. Basically, you need to build an entire cloud services team just to keep it running smoothly. (not for "test labs", but for real production services) And on top of all this, if you're running it on your own hardware, you don't even get the benefit of reduced infrastructure costs. Because this is not only hard to get right, but very costly, it is much cheaper and easier to pay someone to do all this for you. It is almost guaranteed that doing it yourself will not give you any significant advantage, cost savings, or increased development velocity. On top of this, most people don't even need k8s. K8s is a containerized microservice mesh network. If you don't need containers and you aren't running microservices, you may be trying to fit a square peg in a round hole. Even if you did need k8s, the benefit may be small if you don't have complex requirements. Most people can get high-quality, reliable end results with simple, general-purpose solutions using DevOps principles and tools. If you're not Google or Facebook, you probably just need immutable infrastructure-as-code, monitoring, logging, continuous integration/deployment, and maybe autoscaling. You don't need an orchestration framework to deliver all that. And by going with less complex implementations, it will be easier and more cost-effective to maintain. At the end of the day, if you need k8s, use it. But I really worry about most people who hop on the k8s bandwagon because they see a lot of HN posts about it, or because Google touts it.
xaduha 8y ago
I'd be fine with that if they were using LKVM (from kvmtool) or that new Firecracker. But as far as I can tell most projects in this area use QEMU still, which is an outstanding piece of software, but doesn't scream micro-vm to me.
Annatar 8y ago
And people still wrangle with this... how can it be easier to struggle like this than learn how to use SmartOS and Triton? Kubernetes is a solution to severe OS virtualization deficiency in Linux, most notably orchestration. You, know, the problem which is non-existant in SmartOS with Triton and large scale configuration management with operating system packages. Every so often this hits "Hacker News" and people will rather struggle than master something new. But one cannot polish a turd.
- jskaggz 8y ago
  At the risk of tarnishing my reputation amongst the hacker news docker/kubernetes hypecycle elite, have an upvote. The I.T. industry in general is funny. New technologies come and go like pop stars. Docker == ke$ha, Kubernetes == ice cube, triton is fred astaire. They all have their off moments. I personally like my platform stable, performant, secure, and boring. If I spent all of my time keeping up on the latest trends on how to spin up machines, I'd have little time to work on actual product. Something good will come out of this influx of cash, marketing, and cloud sales, eventually. Fits and starts. /me goes back to coding and deploying on triton, while patiently watching the docker/kubernetes show.
  - Annatar 8y ago
    That was a big risk you took. You have guts for standing up to the hype machinery.
    - jskaggz 8y ago
      I'm still fighting the sneaking suspicion that putting kubernetes/etc out to the general public and having such a fast release cycle was just a genius play by the big cloud vendors to acquire customers (who will realize running this stuff on premises isn't as cheap as they thought it was after doing the math (all the math, security, training, operational expenses, personnel training/expenses, moving from docker->moby->rkt->gvisor->firecracker->now vm expenses, blah blah etc)). This current tech wave is kind of disheartening. Everybody is focused on hosting...can we not get icecube to play a show for folks that are pushing the envelope with technology as applied to the medical field, or saving the environment, yo? Triton on-prem is a snap. Boot the headnode from usb, boot the cluster nodes from usb+pxe, and lets get to kicking ass, fighting the good fight focusing on real groundbreaking applications. *edit: I'm still a little butt-hurt after kubernetes being rammed down my throat in a large enterprise environment. Apologies to those that are fighting the good fight with kubernetes, I know you're out there, and big high 5 :)
CyanLite4 8y ago
TLDR: Kubernetes is an anti-pattern.

220 points by MordodeMaru 8y ago

As 2018 comes to a close its time to drag out the hubris and make a bold prediction. The future of Kubernetes is Virtual Machines, not Containers. I’d say that’s less of a prediction than a matter of fact since it’s already happened in 2018 for AWS and GCP. This post is kind of irrelevant for consumers imo, in that the future of Kubernetes is still the container interface, regardless of whether your vendor decides to run it in a container or a VM.

halbritt 8y ago
How has it already happened? It's true that EKS and GKE nodes are virtual machines, but that's entirely beside the point of this article which seems to make the point that the workloads running inside of Kubernetes will also be virtual machines. Interesting notion, but I don't see it. The reason for kubesprawl as it is today is a result of the fact that today Kubernetes is hard to tune for disparate workloads which leads most folks to just punt and stand up multiple clusters. That said, people are starting to figure it out and more tools like the vertical pod autoscaler are coming. Eventually the more efficient choice will be to run disparate workloads across the same set of hardware.

k__ 8y ago

I had the impression DevOps ppl would cling to containers as the final form of VMs? But I'm mostly a high level front-end guy doing back-ends with serverless tech only.

stevenacreman 8y ago

It's still containers. On a cloud provider the Kubernetes workers are VM's which orchestrate containers. With Kata Containers you're just spawning containers inside micro-vm's.

int0x80 8y ago
That is one big difference, with big implications.
- stevenacreman 8y ago
  Just wanted to clarify that Kubernetes was still scheduling containers. Even if VM's are being used to isolate them. It's not all or nothing either. Containerd will support running a mix of containers and kata-containers across workers. For anyone interested in this topic I wrote about some other container runtimes here: https://kubedex.com/kubernetes-container-runtimes/ https://kubedex.com/kubernetes-container-runtimes/
  - amscanne 8y ago
    Note that this is not the case for virtual kubelet-based implementations, and your point here and above are specific to how Kata works (the article is talking more generally).
  - int0x80 8y ago
    Yeah.. I think I see what you say, I mean, the end user interface is the quite the same but this has big implications anyway for the systems design POV, so is not a small thing :)
jacques_chester 8y ago
> It's still containers. The security profiles of containers and VMs, including kernel-based VMs, are different. VMs still have a significant edge, because the attack surface is smaller and doesn't have many competing missions.
- tptacek 8y ago
  The attack surface argument is debatable, depending on how the system is designers, since virtualization introduces the hypervisor surface.

amscanne 8y ago

An important element of Kubernetes is that it standardizes the infrastructure control plane, and allows different pieces to be plugged in (for networking, storage, etc.). The "virtual kubelet" essentially throws that all that away and keeps Kubernetes "in API only". For example, with virtual kubelets, scheduling is meaningless and networking and storage are restricted to whatever the virtual kubelet target supports (if useable at all). Personally, I think the value proposition is tenuous -- you can create VMs today, doing so via the Kubernetes API isn't suddenly revolutionary. Just like throwing something in a "hardware virtualized" instance doesn't suddenly make the whole system secure. Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs).

serverascode 8y ago
I agree, I'm not sure the virtual kubelet concept is a great idea overall. It sounds good on the surface, but most of the time these kind of abstractions do more harm then good, as far as I'm concerned. But, I could be wrong. :)
jacques_chester 8y ago
> Personally, I think the value proposition is tenuous Multi-tenancy is a pretty compelling value proposition when you reach any kind of scale. If you're in a regulated sector, it's non-negotiable. Relying on the cluster as the security boundary is very effective ... and very wasteful. > Containers and Kubernetes are compelling for a variety of reasons, improving them to handle multi-tenancy is a broad challenge but I don't think the answer is to reduce the standard to what we have today (a bunch of disparate VMs). I think the argument is that rather than the painful (and it will be very painful) and probably incomplete quest to retrofit multi-tenancy into a single-tenancy design, we can introduce multi-tenancy where it basically actually matters: at the worker node. At first glance it's confusing to go from "one master, many nodes" to "one node pool, many masters". But it actually works better on every front. Workload efficiency goes up. Security surface area between masters becomes close to nil. Very cheap VMs are the means to that end. Disclosure: I work for Pivotal and this argument fits our basic doctrine of how Kubernetes ought to be used.

orliesaurus 8y ago

This feels like an infinite russian doll problem, putting k8s inside a bunch of vms I mean.. but I am glad that while everyone is still trying to learn how to properly do k8s these folks are thinking of the next thing

dehrmann 8y ago
And running a Java services in the JVM. It's not real infrastructure unless you have at least four layers of VMs.

old-gregg 8y ago

Kubernetes has a real chance to succeed where OpenStack failed. Good people at AWS have good reasons to be worried and will push us to a proprietary form of "serverless".

serverascode 8y ago
I think they are quite different solutions, and each benefits from the other. Further, they also tend to run into the same concerns at the about the same time in their life time, eg. concerns around complexity.
lykr0n 8y ago
OpenStack's failure is that it's an extremely complex system that looks to be nearly impossible to deploy by yourself. You have to use a Distro that does everything for you- their way. Kubernetes is simple enough to setup yourself. They have well documented tooling, and a solid do it yourself guide. OpenStack has none of that (that I can find). You select RedHat Openstack (RDO) or Canonical OpenStack (MAAS), and you have to use their all-in-one system to have a deployment- and that requires a narrow set of variables which every environment might not have. Which is insane- and will hinder adoption. EDIT. Not 100% correct, see below comments.
- bamboozled 8y ago
  The issue with Kubernetes isn't that it's hard to setup, but it's difficult to keep operate long term. Simple things like balancing workloads across availability-zones is still not trivial. I've heard many stories of people who tried to run Kubernetes themselves in production and didn't have a great experience for it.
  - halbritt 8y ago
    I've been running it in production for stateful workloads for over a year and it's been going swimmingly. I've never had better uptime or efficiency.
- serverascode 8y ago

trjordan 8y ago

This is, like all good HN articles, technically correct and practically incorrect. It is correct that containers leak, and people know this. Multi-cluster strategies are real, and they shouldn't be. It should be OK to have one big cluster[1]. Until Kubernetes fixes this, there will be some friction to adopt it, based on real use cases like untrusted code and noisy neighbors. It is incorrect because users (e.g. non-infrastructure engineers) don't know or care about the precise definition of containers and VMs are. The point of "containers" is that I can define something that acts like an operating system from the ground up, and it builds quickly and runs quickly in production. Kubernetes doesn't win by forcing users to think about VMs. Kubernetes wins by adopting a VM standard that can be built by Dockerfiles. Infra engineers will love it. But besides them? Nobody will care, because Docker for Mac will look the same. [1] Maybe 1 cluster per region? There's a whole fascinating topic that starts with the question "when building a PaaS, do you expose region placement to devs?" The answer implies a ton of stuff about what exactly it's reasonable to expect from a PaaS and how much infrastructure your average dev has to know.

deleted 8y ago
[deleted]
georgebarnett 8y ago
> It should be OK to have one big cluster Assuming you're deploying your Kube cluster in the cloud, the costs of having multiple clusters is really reduced. You don't have to allocate physical machines or worry about utilisation as much - you just pick a node size and autoscale. What that enables is thinking about other concerns when deciding how many clusters and where they are is right for your team. There are operational reasons why having multiple clusters is a good idea. At the simplest level, making a config change and only risking a portion of the infrastructure is an example.
- trjordan 8y ago
  Config blast radius, hardware sizing, networking partitions, disaster recovery, dev/prod separation -- lots of good reasons to have multiple clusters! As a gut check, most of these reasons apply to AWS AZs as well. If your Kubernetes strategy calls for more than one per region per AWS account, it means that you, organizationally, don't trust containers as much as you trust VMs on AWS. And right now, you're probably right to do so.

ec109685 8y ago

The nested kubernetes idea is interesting (linked to from the post). However, amazon and google aren’t using nested control planes for their infrastructure. Why is a single control plane good for them, but not for a Kubernetes deployment?

jchw 8y ago

"virtual machines" may be the wrong word. Containers is probably still the right word. What we probably want, is actually isolated containers. And hell, you can nearly get that today. Combining Docker with gVisor is a potential solution to the soft tenancy problem as far as I can tell, and Kubernetes supports using it. (And gVisor is by no stretch of the imagination a 'VM' - it is, at best, a tiny hypervisor, and maybe less than that.)

mcguire 8y ago

As someone who has been a sysadmin and system programmer for 25 years and a crusty rat bastard for almost that long, I have to wonder how long it is going to be until someone realizes that a piece of hardware doing a task is more efficient than 27 layers of virtual machines and a long pair of tongs.

nimbius 8y ago
amen. At first i figured containers were a good way to run code with difficult dependencies and weird operating environments, but that still doesnt make sense. - systemd and uwsgi for example play well when run as a single user per wsgi application under a single nginx/lb. - php-fpm already handles a ton of overhead from php apps. - ansible deployments called from gitlab-ci can roll out apps just as well as deploying from the registry. then i figured maybe they were on to something with the autoscaling thing...but that seems like a meaningless feature. Every good project already has metrics and forecasting...it would be absurd to think a final product like imgur.com or twitter does not know (down to the byte) how much storage theyll need in 4 months and the potential drivers. auto-scaling infrastructure just betrays the fact that most developers throw resources at load problems instead of waiting for ops to figure out the actual issue.
- latchkey 8y ago
  I helped build an e-commerce marketplace, which is essentially a shopping cart system that was used by third parties. We had no idea how to predict traffic loads because we had no idea when one of our customers would run a successful campaign. Autoscaling on google appengine was a lifesaver for us. Our very small dev team focused on building features because we had zero devops and we didn't have to carry a pager.
- rhinoceraptor 8y ago
  I don't think most developers or admins really have the toolset/skills to debug to root cause. Most people see debugging as the process of making problems go away, not as the science of understanding a problem. You need both developer and admin skills, as well as a pretty good understanding of the whole stack - the application, the framework/libraries, the application runtime/support libraries, the database, the kernel and the network.

elsonrodriguez 8y ago

The future of Kubernetes (for some interesting use cases), is virtual machines. For the rest of us the virtual kubelet project represents a bridge between Kubernetes and Serverless. I'd much rather pay my cloud provider to run my k8s workloads (billed by pod requests/limits) than pay for a control plane and three nodes just to run my workloads.

state_less 8y ago

While the use of namespaces seemed a bit discounted in the article, I think part of the problem is that it's an underutilized and poorly conveyed feature, but maybe that's just me? I'd like to see a better out-of-the-box vanilla K8S user management interface in the dashboard. Maybe the default UX on cluster creation is a user/service account that isn't cluster admin that is limited to 1 or more namespaces. There should be a better dashboard to configure and create users that is front and center when you first create a cluster. You have to work a bit to get something you can use to authenticate as cluster admin like role. This should help direct people towards creating more users and isolated namespaces.

justicezyx 8y ago
Also gVisor.
halbritt 8y ago
My org is using namespaces for tenant isolation. Works quite well.
- geerlingguy 8y ago
  Namespaces++. Unless you’re just running one app in a Kubernetes cluster (where default == that app), namespaces solve a plethora of problems.
  - halbritt 8y ago
    Resource limits, network policy boundaries, etc.
    - smarterclayton 8y ago
      That’s what we designed them for :) I do think we’ve not explored enough of the per namespace policy stuff though - i’d like both podpreset and a reasonably simple scheduling policy (toleration + node selector control to replace the annotation based system) to make it in, as well as a simpler namespace initialization path so you can more easily lock down the contents of a namespace without having to proxy the create namespace API call.

lightedman 8y ago

Can't handle real machines, so let's virtualize everything despite the fact we don't have the physical machines fixed or sussed-out, yet. Bravo. Now I see why we've got the problems that intel, AMD, and ARM suffer from. Nobody's bothering to fix a single thing and are just passing the buck like typical lazy developers.

mattbillenstein 8y ago

Dammit, most of you reading this do not need any of this shit! Build simple things!

TuringTest 8y ago
You don't need them until you're hired by a client with nation-wide deployment requirements, and then you need them.
- aprdm 8y ago
  Not necessarily. Needless to say containers and kubernetes are pretty new technologies and nation wide deployment requirement existed before docker / k8s. Most of the old big companies aren't using docker or k8s for their core services. They're all using legacy fat apps that are load balanced in baremetal or vms.
  - TuringTest 8y ago
    I doubt a home-made solution to load balancing will be seen as simpler, which is what's being argued by the comment I replied to. The point isn't that you need Kubernetes specifically, it's that requiring a system that scales well is not as uncommon as the OP puts it.
    - mattbillenstein 8y ago
      k8ts is not the only avenue to building scalable systems -- scaling is mostly about architecture, not how you run your software.
      - TuringTest 8y ago
        > k8ts is not the only avenue to building scalable systems I had already conceded that. What's your point?

013a 8y ago

I really worry about cost in this future of leaning on something like ACI/Fargate to actually run the containers. An m5.large instance (2vcpu/8gb) costs $70/mo on-demand ($44/mo with a 1 year reservation). A similar Fargate runtime costs $146/mo. A b2ms Azure instance (2vcpu/8gb) costs $60/mo on-demand ($39/mo 1 year reservation). Azure Container Instances at a similar provisioning level costs $176/mo by my calculations. That's not a small difference. That's, like, 3x. Point being, I love virtual-kublet from the perspective of a scale-up just trying to get a product out the door. But for established companies, I still think the core idea of a container on a VM you control is going to rule. Fortunately Kubernetes allows amazingly easy flexibility to switch, and that's a reason why it might be the most important technology created in recent history.

hedora 8y ago
Also, the instance prices are already some crazy multiple of buying a machine, racking it, and running dozens of VM’s on it for 5 years. I don’t know the prices for hyperconverged or converged infrastructure off the top of my head, but doubt the amazon rates are even close to competitive unless you’re at a tiny scale (or need a ton of tiny presences in different regions).

markbnj 8y ago

I get the multi-tenancy argument, for certain use cases. I'm not sure I understand the point about greater resource utilization. Presumably they mean workloads can be scheduled more densely given a set of hardware resources, vs. containers... but I'd think that containers would score better on that metric than VMs. Can someone expound? The best part of the piece for me was "kubesprawl," my new favorite word for the week. We've seen it ourselves to some extent, but we are at least aware of it and try to exert some pressure in the other direction. Beyond that I am not particularly bothered by the idea of running lots of clusters for different purposes.

emmelaich 8y ago

I'm picking the eyes out the article here but I had to twitch at this ... > Linux containers were not built to be secure isolated sandboxes (like Solaris Zones or FreeBSD Jails). Instead they’re built upon a shared kernel model ... Solaris Zones and BSD Jails both use a shared kernel. And I'd bet you they were far from perfect in security isolation. Now it may be true that security wasn't Linux containers prime reason for being but we have an existence proof that they can be made secure enough -- anyone can get a trial Openshift container for the price of a login.

peterwwillis 8y ago

I'm convinced that Kubernetes is very good for job security, and not much else. Unless you're a managed services host, you should probably not be running it. Please, for the love of all that is holy, use a cloud services provider if you need K8s-style service features. If you don't, then just cobble together your infrastructure in the simplest way possible that uses DevOps principles, methods and practices.

rantanplan 8y ago
Could you elaborate a bit?
- peterwwillis 8y ago
  K8s is a very complex system. And the more complex the requirements, the more complex one needs to make K8s through additional software that isn't baked in. Complex systems are costly to run, but more importantly, they're costly to maintain due to the typical level of service required, the number of employees needed to maintain it, and the amount of specialized knowledge required. The system is also under constant maintenance due to its short release cycle. Basically, you need to build an entire cloud services team just to keep it running smoothly. (not for "test labs", but for real production services) And on top of all this, if you're running it on your own hardware, you don't even get the benefit of reduced infrastructure costs. Because this is not only hard to get right, but very costly, it is much cheaper and easier to pay someone to do all this for you. It is almost guaranteed that doing it yourself will not give you any significant advantage, cost savings, or increased development velocity. On top of this, most people don't even need k8s. K8s is a containerized microservice mesh network. If you don't need containers and you aren't running microservices, you may be trying to fit a square peg in a round hole. Even if you did need k8s, the benefit may be small if you don't have complex requirements. Most people can get high-quality, reliable end results with simple, general-purpose solutions using DevOps principles and tools. If you're not Google or Facebook, you probably just need immutable infrastructure-as-code, monitoring, logging, continuous integration/deployment, and maybe autoscaling. You don't need an orchestration framework to deliver all that. And by going with less complex implementations, it will be easier and more cost-effective to maintain. At the end of the day, if you need k8s, use it. But I really worry about most people who hop on the k8s bandwagon because they see a lot of HN posts about it, or because Google touts it.

xaduha 8y ago

I'd be fine with that if they were using LKVM (from kvmtool) or that new Firecracker. But as far as I can tell most projects in this area use QEMU still, which is an outstanding piece of software, but doesn't scream micro-vm to me.

Annatar 8y ago

And people still wrangle with this... how can it be easier to struggle like this than learn how to use SmartOS and Triton? Kubernetes is a solution to severe OS virtualization deficiency in Linux, most notably orchestration. You, know, the problem which is non-existant in SmartOS with Triton and large scale configuration management with operating system packages. Every so often this hits "Hacker News" and people will rather struggle than master something new. But one cannot polish a turd.

jskaggz 8y ago
At the risk of tarnishing my reputation amongst the hacker news docker/kubernetes hypecycle elite, have an upvote. The I.T. industry in general is funny. New technologies come and go like pop stars. Docker == ke$ha, Kubernetes == ice cube, triton is fred astaire. They all have their off moments. I personally like my platform stable, performant, secure, and boring. If I spent all of my time keeping up on the latest trends on how to spin up machines, I'd have little time to work on actual product. Something good will come out of this influx of cash, marketing, and cloud sales, eventually. Fits and starts. /me goes back to coding and deploying on triton, while patiently watching the docker/kubernetes show.
- Annatar 8y ago
  That was a big risk you took. You have guts for standing up to the hype machinery.
  - jskaggz 8y ago
    I'm still fighting the sneaking suspicion that putting kubernetes/etc out to the general public and having such a fast release cycle was just a genius play by the big cloud vendors to acquire customers (who will realize running this stuff on premises isn't as cheap as they thought it was after doing the math (all the math, security, training, operational expenses, personnel training/expenses, moving from docker->moby->rkt->gvisor->firecracker->now vm expenses, blah blah etc)). This current tech wave is kind of disheartening. Everybody is focused on hosting...can we not get icecube to play a show for folks that are pushing the envelope with technology as applied to the medical field, or saving the environment, yo? Triton on-prem is a snap. Boot the headnode from usb, boot the cluster nodes from usb+pxe, and lets get to kicking ass, fighting the good fight focusing on real groundbreaking applications. *edit: I'm still a little butt-hurt after kubernetes being rammed down my throat in a large enterprise environment. Apologies to those that are fighting the good fight with kubernetes, I know you're out there, and big high 5 :)

CyanLite4 8y ago

TLDR: Kubernetes is an anti-pattern.