14 ms·

Kubernetes reinvented virtual machines in a good sense

186 points by paulgb 4y ago

secondcoming 4y ago
> On the darker side, inefficient scaling (due to uneven daily/yearly traffic distribution), overly complicated deployments (delivering code to many boxes quickly is hard), and fragile service discovery (have you tried running consul or zookeeper at scale?) would lead to higher operational costs. Obviously, everyone's use-case is different but none of this has been my experience with VMs: - Both AWS and GCP allow you to define custom scaling policies for an instance group. - Deploying new code just means updating a template with your new image and pushing that out. Both GCP and AWS would then cycle the existing machines automatically to use this new image. - At least in our case, there is no need for Service Discovery; you just need to know where the Load Balancer is. We have moved some things to Kubernetes and now have to deal with inexplicable occasional failures. That and an everyday massive YAML headache. YMMV
- awoimbee 4y ago
  IMO it takes time to get up to speed but then it's pure bliss in terms of reliability. And don't go to yaml hell, use pulumi or equivalent.
  - x86x87 4y ago
    > it's pure bliss in terms of reliability narrator: it is not. you're not trying very hard if you haven't hit a case where k8s will lead to you banging your heads on any available walls trying to understand a behavior in an edge case (here is one that made walls hate me: https://github.com/kubernetes/kubernetes/issues/96858 https://github.com/kubernetes/kubernetes/issues/96858 and I know of a half dozen developers that learned about this the hard way).
    - hanikesn 4y ago
      Looks to me like an example of aws not having good defaults put of the box and not an k8s specific issue. Having to specific a `sleep 30` shutdown hook on pods is quite annoying though and some built-in should be added instead.
      - shaklee3 4y ago
        exactly. if you read the bug it's clearly not a k8s issue. they cannot control aws jabbering elb bugs.
        x86x87 4y ago
        You're missing the point. There is no cloud agnosticism and you need to understand the underlying cloud primitives although allegedly k8s isolates you from this.
        shaklee3 4y ago
        I'm not missing the point. The OP was specifically talking about a k8s bug that wasn't a k8s bug at all. You don't blame Linux for a poorly written application. If the cloud providers choose to add a bunch of uncommon options that has no counterpart in k8s that's their choice. but it doesn't mean k8s is a magic bullet that makes it so you don't have to do the bare minimum to understand the cloud providers.
jbverschoor 4y ago
I thought we had this kind of covered 20 years ago with J2EE
- jayd16 4y ago
  Doesn't really handle multiple process isolation and IPC as well.
  - cassianoleal 4y ago
    It also doesn't really handle anything that doesn't run on the JVM.
    - jayd16 4y ago
      Well sure but that's true of any VM, right? You still need to make sure your images run on the k8s instance even if that's a lower bar.
      - cassianoleal 4y ago
        I don't need to rewrite my apps in order to run them on a different architecture. Recompiling is enough for compiled languages, and for interpreted languages you just need to add the interpreter appropriate to the architecture. It's definitely not the same thing.
        jayd16 4y ago
        I suppose you missed that it was a bit tongue in cheek but if you want to take it seriously, I think it still stands. You can just compile to JVM bytecode instead of the assembly of your vm of choice. It's indeed different in practice but I don't think it's all that different in principle. Really though, I was just pointing out the tautology of your previous comment. It boils down to "it doesn't run anything that doesn't run on it." https://stackoverflow.com/questions/4221605/compiling-c-for-the-jvm https://stackoverflow.com/questions/4221605/compiling-c-for-...
rwmj 4y ago
Wouldn't you want to use KubeVirt for this?
z3t4 4y ago
The bus factor - your engineering skills that took you 20+ years to acquire are worth nothing if you can't be replaced...
- hinkley 4y ago
  It’s astounding how long companies can ignore this about their people. Acknowledging that kind of power in an employee probably messes up the power dynamic in uncomfortable ways. I recently learned that one of the heads of SpaceX is heading his own company now. It’ll be interesting to see if his bus number was high enough that the wheels stay on, if the symptoms will be obvious if it wasn’t, or if they will be explained away by other phenomenon. I have had enough successful bosses that credited the wrong actions that it stands to reason that most failures are also blamed on the wrong chain of events. I know my own narratives in 5 Why’s situations are often in stark contrast to other theories.
  - mgfist 4y ago
    > It’s astounding how long companies can ignore this about their people. Is that really a problem? Companies churn people exceptionally fast these days. That inherently involves dealing with the bus problem.
    - hinkley 4y ago
      Define 'fast' and 'these days'. Lots of companies are affected by bus problems, but I'm not so sure they're successful at dealing with it. I'm also not sure this is a new phenomenon, and it's certainly not universal. If anything I can think of a couple of projects that suffer from too little turnover, and the resultant echo chamber. As for 'dealing', look at all the status Google services have lost in the last ten years due to systemic problems with valuing novelty over stability. There are perverse incentives not to maintain things you have delivered, even if you are still at the company.
x86x87 4y ago
Did K8s reinvent VMs though? People keep forgetting that vm!=container and the security posture is completely different. Also, for the amounts of complexity K8s adds it's not worth the overhead in 95% of cases (i get it that it's cool and everyone and their dog want to claim they have experience with it - I am affraid that once the "movement" fizzles out we're going to be stuck with a lot of things that are over-overengineered)
- deleted 4y ago
  [deleted]
- k8sToGo 4y ago
  Do you know kubernetes or are you just jumping on the hate train? Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?
  - jmspring 4y ago
    He has a point, containers and VMs are not the same.
    - k8sToGo 4y ago
      Yes that part is true. I was not referring to that part.
  - de6u99er 4y ago
    > Do you know kubernetes or are you just jumping on the hate train? It seems rather that you don't know what kubernetes is. His comment is justified. You can put your pitchfork back. > Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines? It seems that you don't understand ehat he's trying to say.
    - k8sToGo 4y ago
aetherspawn 4y ago
Good for some use cases, but still plenty of servers that are hostile to K8s because ie you have to login and enter a license key, or something like that.
avereveard 4y ago
*chroots
tannhaeuser 4y ago
That doesn't make sense? K8s runs Docker images, and a Docker image explicitly is not a VM but, in fact, a lightweight replacement for a full-blown VM. Of course, running Docker on Windows or Mac OS requires a Linux VM, though.
- jiggawatts 4y ago
  You can run Windows images on Windows clients using Docker.
  - pjmlp 4y ago
    Docker still requires a mini WSL instance running.
    - koffiezet 4y ago
      Not for windows native images (which are a thing, but does anyone actually use them?) So in theory, no you don't need WSL.
      - pjmlp 4y ago
        Yes we use them, for example Sitecore uses Windows Containers, and Docker Desktop does rely on WSL being there no matter what.
sureglymop 4y ago
The containerization features used by projects like Docker and Kubernetes are provided by the Linux kernel. Containers are not VMs, they're just regular processes.
- x86x87 4y ago
  Containers are not regular processes. One could argue that you can have a container without any process in it. A container is an abstract idea that normally comes together through the synergy of [at least] cgroups, namespaces and overlay filesystems.
  - sureglymop 4y ago
    They are regular linux processes, you can find them in /proc or display them with ps... Croups, namespaces etc are Linux features that can be used with any process. But i guess i see now what you are saying. That it's a "meta" term for that idea of abstraction.
hinkley 4y ago
Does anyone know of a good, sober treatise on the suitability of microkernels for multitenancy and/or split priority systems? Hypervisors just seem like particularly impoverished microkernels to me. It reminds me of social programs working hard to avoid the term Socialism because of emotional baggage. Oh this is definitely not a microkernel? Is it a full privilege program that handles a small number of system-wide concerns and delegates the maximum number of other concerns to processes running at a lower privilege level? Yes? Then can we call it a… no? Okay buddy.
- alexjurkiewicz 4y ago
  Most people use kernel to mean "operating system kernel" only. Your definition is a superset of that, so it fits the general definition but then expands it in a way that doesn't have broad consensus. That's why people disagree with you.
- gwd 4y ago
  > Hypervisors just seem like particularly impoverished microkernels to me. Or, "Microkernels done right": https://www.usenix.org/legacy/event/hotos05/final_papers/full_papers/hand/hand.pdf https://www.usenix.org/legacy/event/hotos05/final_papers/ful... > At the last HotOS, Mendel Rosenblum gave an ‘outrageous’ opinion that the academic obsession with microkernels during the past two decades produced many publications but little impact. He argued that virtual machine monitors (VMMs) had had considerably more practical uptake, despite—or perhaps due to—being principally developed by industry. > In this paper, we investigate this claim in light of our experiences in developing the Xen [1] virtual machine monitor. We argue that modern VMMs present a practical platform which allows the development and deployment of innovative systems research: in essence, VMMs are microkernels done right. (Edited to add relevant paragraphs from the abstract.)
- soraminazuki 4y ago
  You just described one of the key ideas behind unikernels ;)
202206241203 4y ago
Kubernetes... It's something they used to utilise in the past before cloud-native, right?
gwd 4y ago
Most of the comments here are missing the point of the article. Yes, Docker containers != VMs. That's the point: An old-school VM had all the components needed to run one instance of a service; this might include the core app itself, along with "support processes", like a reverse proxy for TLS termination, and maybe some other services. By design, you cannot run all of those in a single docker container. The point of the article is to say that Kubernetes pods ~= VMs: A pod has all the components required for a single instance of a service, each one in its own docker container, but with increased access to other containers in the pod (just as processes have increased access to other processes in the same VM). So if you're coming from a VM deployment world, when you see "pod" think "vm", and when you see "sidecar" think "support process".
- datalopers 4y ago
  > By design, you cannot run all of those in a single docker container. Sure you can. The notion that “one container, one process” is the biggest misstep of containerization.
  - mccorrinall 4y ago
    Why do you think so? In my opinion it makes debugging, deployment and keeping the architecture clean much easier.
  - gwd 4y ago
    Note that I said "Docker container". And that's an oversimplification, but from the article: > Docker containers have been deliberately designed to contain just one application. One container - one Nginx; one container - one Python web server; one container - one daemon. The lifecycle of a container would be bound to the lifecycle of that application. And running an init process like systemd as a top-level entrypoint was specifically discouraged. My guess is that the goal of this is to leave "lifecycle" issues at the container boundary. If you put nginx and your golang webapp in the same container, and one of them crashes, Docker / kubernetes you have to have logic inside the container to detect that and restart the correct one. If the goal is to have Docker / Kubernetes is to take care of all that for you, making it one thing per container is the best option.
siliconc0w 4y ago
K8s can actually run VMs as well (kube-virt) so you can theoretically align on a common platform for both VMs and Containers. In practice most people just use their cloud provider's implementation of K8s. It's complex but if I had to run on bare-metal I'd probably look in that direction vs playing for an equally complex proprietary solution.
neeleshs 4y ago
A lot of comments here about the complexity. But a takeaway from the article is that driving a car is different from repairing it. We are a startup, had zero knowledge of k8s before. Without GKE, we would have lost days and days of work to imitate parts of k8s on VMs. Scaling up and down, everything as a container (even cron jobs. Otherwise we had to keep a vm running and use some scheduler). GKE has been a massive advantage for us. The complexity bits may be true for someone managing a k8s cluster. Even with just 3-4 services, managing everything as cattle has been amazing.
- dijit 4y ago
  To be perfectly fair with you, GKE is another level from every other k8s installation target. EKS and AKS are not nearly as well polished and put together, and god help you if you decide to run on-prem. The problem is of course that you’re quite right that it’s like driving a car, but ultimately unless you have a good support contract with your car manufacturer for a specific use of the car, then you have to take it to another mechanic and they’re going to put you over a barrel. (Maybe the analogy falls apart here)
- tc08 4y ago
  I couldn’t agree more. This and the cloud foundation toolkit for terraform is why we chose GCP, and it got us up and running with an impressive infrastructure in a few days.
  - x86x87 4y ago
    Is your goal really to be up and running in a few days? My goal is to run for years without significant ops effort. Maintainence cost always, always dwarves development cost.
    - neeleshs 4y ago
      They are not mutually exclusive with something like GKE. We went up and running in a few days and been going strong for several years now.
dirkg 4y ago
How do lightweight MicroVMs which have the speed of containers but the isolation of a real vm, as implemented in FIrecracker etc, change the landscape? Or do they? I imagine launching a k8s pod or container is still faster and offers much more management and the security benefits of a real vm arent really applicable or desirable?
- x86x87 4y ago
  the revolution will start across this dimension at some point in time. We're looking at unikernels (https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel) and boot times of nanoseconds for these super specialized VMs. The tech and the ideas around it are not quite there but this is an exciting space to watch in the next 5-10 years.
- AtlasBarfed 4y ago
  That is so close to a simple process in a multitasking OS ... I get it, you still need dependency isolation, you're just reducing the dependencies to the minimum number as possible, but sometimes I wonder if something (besides cgroups) was missed in OS design.
- vbezhenar 4y ago
  Launching a container is implementation detail for Kubernetes. I could totally see Kubernetes which runs some pods in containers and some pods in VMs with tiny kernel. May be it even exists, I'm too lazy to google it.
- koffiezet 4y ago
  Microvms are just the workloads, while Kubernetes is a platform, mostly agnostic of that workload. Yes that's currently mostly containers, but you can already schedule VMs on k8s with KubeVirt.
draw_down 4y ago
tzury 4y ago
Has anyone ideitified the tools used for the diagrams?
- arunc 4y ago
  Looks like illustrator, but I could be wrong
- jq-r 4y ago
  Pretty sure that’s Excalidraw.
roschdal 4y ago
Kubernetes is horribly complex.
- x86x87 4y ago
  That's the tragedy of software development. People like complex things because it shows how smart they are and how hard the problem they are solving really is.
zxcvbn4038 4y ago
Kubernetes reinvented virtual machines? That’s a bold statement. I’m still waiting for them to catch up to where OS/360 was 45 years ago. People forget that mainframe did it first, mainframe still does it best, the only thing that’s changed is that now you can do it on consumer grade electronics for less then 5 million dollars.
- koffiezet 4y ago
  Can you elaborate on what mainframes do better? I'm genuinely curious (not being too familiar with mainframes).
Skinner112 4y ago
[dead]

186 points by paulgb 4y ago

> On the darker side, inefficient scaling (due to uneven daily/yearly traffic distribution), overly complicated deployments (delivering code to many boxes quickly is hard), and fragile service discovery (have you tried running consul or zookeeper at scale?) would lead to higher operational costs. Obviously, everyone's use-case is different but none of this has been my experience with VMs: - Both AWS and GCP allow you to define custom scaling policies for an instance group. - Deploying new code just means updating a template with your new image and pushing that out. Both GCP and AWS would then cycle the existing machines automatically to use this new image. - At least in our case, there is no need for Service Discovery; you just need to know where the Load Balancer is. We have moved some things to Kubernetes and now have to deal with inexplicable occasional failures. That and an everyday massive YAML headache. YMMV

awoimbee 4y ago
IMO it takes time to get up to speed but then it's pure bliss in terms of reliability. And don't go to yaml hell, use pulumi or equivalent.
- x86x87 4y ago
  > it's pure bliss in terms of reliability narrator: it is not. you're not trying very hard if you haven't hit a case where k8s will lead to you banging your heads on any available walls trying to understand a behavior in an edge case (here is one that made walls hate me: https://github.com/kubernetes/kubernetes/issues/96858 https://github.com/kubernetes/kubernetes/issues/96858 and I know of a half dozen developers that learned about this the hard way).
  - hanikesn 4y ago
    Looks to me like an example of aws not having good defaults put of the box and not an k8s specific issue. Having to specific a `sleep 30` shutdown hook on pods is quite annoying though and some built-in should be added instead.
    - shaklee3 4y ago
      exactly. if you read the bug it's clearly not a k8s issue. they cannot control aws jabbering elb bugs.
      - x86x87 4y ago
        You're missing the point. There is no cloud agnosticism and you need to understand the underlying cloud primitives although allegedly k8s isolates you from this.
        shaklee3 4y ago
        I'm not missing the point. The OP was specifically talking about a k8s bug that wasn't a k8s bug at all. You don't blame Linux for a poorly written application. If the cloud providers choose to add a bunch of uncommon options that has no counterpart in k8s that's their choice. but it doesn't mean k8s is a magic bullet that makes it so you don't have to do the bare minimum to understand the cloud providers.

jbverschoor 4y ago

I thought we had this kind of covered 20 years ago with J2EE

jayd16 4y ago
Doesn't really handle multiple process isolation and IPC as well.
- cassianoleal 4y ago
  It also doesn't really handle anything that doesn't run on the JVM.
  - jayd16 4y ago
    Well sure but that's true of any VM, right? You still need to make sure your images run on the k8s instance even if that's a lower bar.
    - cassianoleal 4y ago
      I don't need to rewrite my apps in order to run them on a different architecture. Recompiling is enough for compiled languages, and for interpreted languages you just need to add the interpreter appropriate to the architecture. It's definitely not the same thing.
      - jayd16 4y ago
        I suppose you missed that it was a bit tongue in cheek but if you want to take it seriously, I think it still stands. You can just compile to JVM bytecode instead of the assembly of your vm of choice. It's indeed different in practice but I don't think it's all that different in principle. Really though, I was just pointing out the tautology of your previous comment. It boils down to "it doesn't run anything that doesn't run on it." https://stackoverflow.com/questions/4221605/compiling-c-for-the-jvm https://stackoverflow.com/questions/4221605/compiling-c-for-...

rwmj 4y ago

Wouldn't you want to use KubeVirt for this?

z3t4 4y ago

The bus factor - your engineering skills that took you 20+ years to acquire are worth nothing if you can't be replaced...

hinkley 4y ago
It’s astounding how long companies can ignore this about their people. Acknowledging that kind of power in an employee probably messes up the power dynamic in uncomfortable ways. I recently learned that one of the heads of SpaceX is heading his own company now. It’ll be interesting to see if his bus number was high enough that the wheels stay on, if the symptoms will be obvious if it wasn’t, or if they will be explained away by other phenomenon. I have had enough successful bosses that credited the wrong actions that it stands to reason that most failures are also blamed on the wrong chain of events. I know my own narratives in 5 Why’s situations are often in stark contrast to other theories.
- mgfist 4y ago
  > It’s astounding how long companies can ignore this about their people. Is that really a problem? Companies churn people exceptionally fast these days. That inherently involves dealing with the bus problem.
  - hinkley 4y ago
    Define 'fast' and 'these days'. Lots of companies are affected by bus problems, but I'm not so sure they're successful at dealing with it. I'm also not sure this is a new phenomenon, and it's certainly not universal. If anything I can think of a couple of projects that suffer from too little turnover, and the resultant echo chamber. As for 'dealing', look at all the status Google services have lost in the last ten years due to systemic problems with valuing novelty over stability. There are perverse incentives not to maintain things you have delivered, even if you are still at the company.

x86x87 4y ago

Did K8s reinvent VMs though? People keep forgetting that vm!=container and the security posture is completely different. Also, for the amounts of complexity K8s adds it's not worth the overhead in 95% of cases (i get it that it's cool and everyone and their dog want to claim they have experience with it - I am affraid that once the "movement" fizzles out we're going to be stuck with a lot of things that are over-overengineered)

deleted 4y ago
[deleted]
k8sToGo 4y ago
Do you know kubernetes or are you just jumping on the hate train? Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines?
- jmspring 4y ago
  He has a point, containers and VMs are not the same.
  - k8sToGo 4y ago
    Yes that part is true. I was not referring to that part.
- de6u99er 4y ago
  > Do you know kubernetes or are you just jumping on the hate train? It seems rather that you don't know what kubernetes is. His comment is justified. You can put your pitchfork back. > Can't you argue with that logic against everything remotely complex? If you have people who know how to use it, then why not use it? Would you say the same about complicated engineered car engines? It seems that you don't understand ehat he's trying to say.
  - k8sToGo 4y ago

aetherspawn 4y ago

Good for some use cases, but still plenty of servers that are hostile to K8s because ie you have to login and enter a license key, or something like that.

avereveard 4y ago

*chroots

tannhaeuser 4y ago

That doesn't make sense? K8s runs Docker images, and a Docker image explicitly is not a VM but, in fact, a lightweight replacement for a full-blown VM. Of course, running Docker on Windows or Mac OS requires a Linux VM, though.

jiggawatts 4y ago
You can run Windows images on Windows clients using Docker.
- pjmlp 4y ago
  Docker still requires a mini WSL instance running.
  - koffiezet 4y ago
    Not for windows native images (which are a thing, but does anyone actually use them?) So in theory, no you don't need WSL.
    - pjmlp 4y ago
      Yes we use them, for example Sitecore uses Windows Containers, and Docker Desktop does rely on WSL being there no matter what.

sureglymop 4y ago

The containerization features used by projects like Docker and Kubernetes are provided by the Linux kernel. Containers are not VMs, they're just regular processes.

x86x87 4y ago
Containers are not regular processes. One could argue that you can have a container without any process in it. A container is an abstract idea that normally comes together through the synergy of [at least] cgroups, namespaces and overlay filesystems.
- sureglymop 4y ago
  They are regular linux processes, you can find them in /proc or display them with ps... Croups, namespaces etc are Linux features that can be used with any process. But i guess i see now what you are saying. That it's a "meta" term for that idea of abstraction.

hinkley 4y ago

Does anyone know of a good, sober treatise on the suitability of microkernels for multitenancy and/or split priority systems? Hypervisors just seem like particularly impoverished microkernels to me. It reminds me of social programs working hard to avoid the term Socialism because of emotional baggage. Oh this is definitely not a microkernel? Is it a full privilege program that handles a small number of system-wide concerns and delegates the maximum number of other concerns to processes running at a lower privilege level? Yes? Then can we call it a… no? Okay buddy.

alexjurkiewicz 4y ago
Most people use kernel to mean "operating system kernel" only. Your definition is a superset of that, so it fits the general definition but then expands it in a way that doesn't have broad consensus. That's why people disagree with you.
gwd 4y ago
> Hypervisors just seem like particularly impoverished microkernels to me. Or, "Microkernels done right": https://www.usenix.org/legacy/event/hotos05/final_papers/full_papers/hand/hand.pdf https://www.usenix.org/legacy/event/hotos05/final_papers/ful... > At the last HotOS, Mendel Rosenblum gave an ‘outrageous’ opinion that the academic obsession with microkernels during the past two decades produced many publications but little impact. He argued that virtual machine monitors (VMMs) had had considerably more practical uptake, despite—or perhaps due to—being principally developed by industry. > In this paper, we investigate this claim in light of our experiences in developing the Xen [1] virtual machine monitor. We argue that modern VMMs present a practical platform which allows the development and deployment of innovative systems research: in essence, VMMs are microkernels done right. (Edited to add relevant paragraphs from the abstract.)
soraminazuki 4y ago
You just described one of the key ideas behind unikernels ;)

202206241203 4y ago

Kubernetes... It's something they used to utilise in the past before cloud-native, right?

gwd 4y ago

Most of the comments here are missing the point of the article. Yes, Docker containers != VMs. That's the point: An old-school VM had all the components needed to run one instance of a service; this might include the core app itself, along with "support processes", like a reverse proxy for TLS termination, and maybe some other services. By design, you cannot run all of those in a single docker container. The point of the article is to say that Kubernetes pods ~= VMs: A pod has all the components required for a single instance of a service, each one in its own docker container, but with increased access to other containers in the pod (just as processes have increased access to other processes in the same VM). So if you're coming from a VM deployment world, when you see "pod" think "vm", and when you see "sidecar" think "support process".

datalopers 4y ago
> By design, you cannot run all of those in a single docker container. Sure you can. The notion that “one container, one process” is the biggest misstep of containerization.
- mccorrinall 4y ago
  Why do you think so? In my opinion it makes debugging, deployment and keeping the architecture clean much easier.
- gwd 4y ago
  Note that I said "Docker container". And that's an oversimplification, but from the article: > Docker containers have been deliberately designed to contain just one application. One container - one Nginx; one container - one Python web server; one container - one daemon. The lifecycle of a container would be bound to the lifecycle of that application. And running an init process like systemd as a top-level entrypoint was specifically discouraged. My guess is that the goal of this is to leave "lifecycle" issues at the container boundary. If you put nginx and your golang webapp in the same container, and one of them crashes, Docker / kubernetes you have to have logic inside the container to detect that and restart the correct one. If the goal is to have Docker / Kubernetes is to take care of all that for you, making it one thing per container is the best option.

siliconc0w 4y ago

K8s can actually run VMs as well (kube-virt) so you can theoretically align on a common platform for both VMs and Containers. In practice most people just use their cloud provider's implementation of K8s. It's complex but if I had to run on bare-metal I'd probably look in that direction vs playing for an equally complex proprietary solution.

neeleshs 4y ago

A lot of comments here about the complexity. But a takeaway from the article is that driving a car is different from repairing it. We are a startup, had zero knowledge of k8s before. Without GKE, we would have lost days and days of work to imitate parts of k8s on VMs. Scaling up and down, everything as a container (even cron jobs. Otherwise we had to keep a vm running and use some scheduler). GKE has been a massive advantage for us. The complexity bits may be true for someone managing a k8s cluster. Even with just 3-4 services, managing everything as cattle has been amazing.

dijit 4y ago
To be perfectly fair with you, GKE is another level from every other k8s installation target. EKS and AKS are not nearly as well polished and put together, and god help you if you decide to run on-prem. The problem is of course that you’re quite right that it’s like driving a car, but ultimately unless you have a good support contract with your car manufacturer for a specific use of the car, then you have to take it to another mechanic and they’re going to put you over a barrel. (Maybe the analogy falls apart here)
tc08 4y ago
I couldn’t agree more. This and the cloud foundation toolkit for terraform is why we chose GCP, and it got us up and running with an impressive infrastructure in a few days.
- x86x87 4y ago
  Is your goal really to be up and running in a few days? My goal is to run for years without significant ops effort. Maintainence cost always, always dwarves development cost.
  - neeleshs 4y ago
    They are not mutually exclusive with something like GKE. We went up and running in a few days and been going strong for several years now.

dirkg 4y ago

How do lightweight MicroVMs which have the speed of containers but the isolation of a real vm, as implemented in FIrecracker etc, change the landscape? Or do they? I imagine launching a k8s pod or container is still faster and offers much more management and the security benefits of a real vm arent really applicable or desirable?

x86x87 4y ago
the revolution will start across this dimension at some point in time. We're looking at unikernels (https://en.wikipedia.org/wiki/Unikernel https://en.wikipedia.org/wiki/Unikernel) and boot times of nanoseconds for these super specialized VMs. The tech and the ideas around it are not quite there but this is an exciting space to watch in the next 5-10 years.
AtlasBarfed 4y ago
That is so close to a simple process in a multitasking OS ... I get it, you still need dependency isolation, you're just reducing the dependencies to the minimum number as possible, but sometimes I wonder if something (besides cgroups) was missed in OS design.
vbezhenar 4y ago
Launching a container is implementation detail for Kubernetes. I could totally see Kubernetes which runs some pods in containers and some pods in VMs with tiny kernel. May be it even exists, I'm too lazy to google it.
koffiezet 4y ago
Microvms are just the workloads, while Kubernetes is a platform, mostly agnostic of that workload. Yes that's currently mostly containers, but you can already schedule VMs on k8s with KubeVirt.

draw_down 4y ago

tzury 4y ago

Has anyone ideitified the tools used for the diagrams?

arunc 4y ago
Looks like illustrator, but I could be wrong
jq-r 4y ago
Pretty sure that’s Excalidraw.

roschdal 4y ago

Kubernetes is horribly complex.

x86x87 4y ago
That's the tragedy of software development. People like complex things because it shows how smart they are and how hard the problem they are solving really is.

zxcvbn4038 4y ago

Kubernetes reinvented virtual machines? That’s a bold statement. I’m still waiting for them to catch up to where OS/360 was 45 years ago. People forget that mainframe did it first, mainframe still does it best, the only thing that’s changed is that now you can do it on consumer grade electronics for less then 5 million dollars.

koffiezet 4y ago
Can you elaborate on what mainframes do better? I'm genuinely curious (not being too familiar with mainframes).

Skinner112 4y ago

[dead]