9 ms·

Containers are not just for Kubernetes

kyrra 5y ago
For Google cloud, I think before Cloud Run existed, they created a "create-with-container" option that deploys a container on a VM for you. https://cloud.google.com/sdk/gcloud/reference/compute/instances/create-with-container https://cloud.google.com/sdk/gcloud/reference/compute/instan... (I'm a googler, but do not work anywhere near your cloud. I used this for a personal project.)
- jillesvangurp 5y ago
  That still exists. We actually switched to that from Cloud Run. Cloud Run was a little too restrictive with background threads in our application. These basically get throttled in between requests and we kind of needed them not to be throttled. Otherwise it is actually great and works as advertised. The process of switching to using simple vms was a gcloud one liner to spin up a vm with the same docker image. That basically worked the first time we tried it. We ran with this for a few months. Then we finally sat down and got ourselves a load balancer with an instance group and an instance template. We deploy by telling gcloud to replace the template on the instance group. It does a rolling restart and it uses our health checks to ensure it does the right things. And it's stupidly easy to script together. We use Github actions for automated deploys. It does rolling restarts, has health checks, monitoring, logging, etc. built in. Everything you need to run a simple monolith. It's similar to what Amazon did with ECS. Just less convoluted and way more easier to use. People assume they need all this crazy complicated infrastructure. But it's all commodity off the shelf stuff these days and it doesn't have to be that hard to set up. Could we automate the above? Sure. But what's the point. It's 1 command to create instance templates, another command to tell the LB to run it. Provisioning the LB with a wild card certificate is a five minute job using the UI. It will tell you the commands if you need them. But it's something you do once per environment.
  - intrepidsoldier 5y ago
    Agreed. People choose technologies for their own reasons. The full spectrum is surviving and thriving: https://ramansharma.substack.com/p/multiple-stops-on-the-cloud-compute-continuum-all-with-their-own-merits-6b921ed82f2 https://ramansharma.substack.com/p/multiple-stops-on-the-clo...
- pqb 5y ago
  I am shocked, because I see it for the first time. Recently, I have been browsing gcloud documentation like a crazy to prepare myself to PCA certification but I have never seen this doc page before [0]. Thanks a lot for a mentioning it! [0]: https://cloud.google.com/compute/docs/containers/deploying-containers https://cloud.google.com/compute/docs/containers/deploying-c...
trabant00 5y ago
My opinion at the moment, subject to change: Docker and Kubernetes have a high price as they add a lot of complexity and make debugging harder. Especially for networking and state-full. Most of the problems they claim to solve where already solved years ago by more mature solutions: - reproducible builds and dependencies management: distribution package managers - high availability: load balancers, keepalive, dynamic routing, clustering and most importantly: data replication (where is this in k8s?) since downtime is nothing compared to data loss - resource limitation: cgroups - isolation: the good old user and groups, vlans, firewalls - orchestration, rollouts, rollbacks: CD pipelines, configuration management Technically the only thing I find easier with k8s is bin packing. The real advantage I find in containers and k8s is they became a standard. You don't have to learn the architecture that each company has designed for itself and dig into custom built solutions. So you can easily switch jobs and they can easily replace you. If you throw that away with custom container implementations for each PaaS offering and fragmentation of orchestrator solutions then you only get the complexity and no benefit. We're going to see in a few more years where it goes but I'm not exactly optimistic.
- dilyevsky 5y ago
  “Kubernetes is too complex - let me just string together 2/3rds of what it does with a few thousand lines of bash…”
  - dijit 5y ago
    There has to be a name for the fallacy that stipulates “the alternative to complex systems is bash”.
    - dilyevsky 5y ago
      It maybe a fallacy alright but this exactly what usually happens
  - stephenr 5y ago
    TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.
tyingq 5y ago
LXD is worth a look if you want central management of containers with very little in the way of extra layers. It can also spin up normal VMs if containers don't work for you. And the control plane data is spread across a raft-enabled and distributed sqlite.
- Helmut10001 5y ago
  In case anyone is confused with the terms or tools, see the Proxmox [1] docs for an explanation of the differences between LXC vs LXD vs Proxmox Containers vs Docker [1]: https://pve.proxmox.com/wiki/FAQ https://pve.proxmox.com/wiki/FAQ
  - zekrioca 5y ago
    Yes, I do think they should have chosen a better name than "LXD". It really confuses itself with LXC, specially so because all of its commands start with "lxc"..
josteink 5y ago
I use LXC for isolation with VM-like environments, without the regular VM overhead. Never bothered with Docker or Kubernetes, as I already have my needs solved. Call me a dinosaur if you like, but it’s fundamentally simple and it works for me. To expand on the title: Containers are not just Docker.
- zekrioca 5y ago
  You would be impressed by LXD. If you ever needs LXC containers working together as in a cluster, try it out.
KronisLV 5y ago
> Along came the Kubernetes wave. It wasn’t meant to solve any of the above problems. It was supposed to provide efficient utilization of compute resources while improving reliability and availability. So, both containers and Kubernetes serve different purposes. I feel like Kubernetes tried doing way more than that: it added additional abstractions like ingresses and load balancers, abstracted away storage with its drivers and PVC concept, added support for all sorts of cases when something should be run (e.g. if there's a task you'd like to schedule or even just run once), introduced namespaces and its API is pretty extensive and extensible, especially with CRDs. Just look at how many projects have propped up around it, from networking solutions (Calico, Flannel, Istio, linkerd, ...), to storage solutions (Longhorn, Rook, ...), to all sorts of additional software, like Helm, Kustomize, Skaffold etc. Of course, that's not to say that everyone needs all of that complexity, as others have rightfully pointed out in the comments. Personally, i think that the sweet spot is focusing on OCI as the standard to go for and hosting most of the software that you need inside of containers with a lightweight orchestrator, only moving to something more complicated once the need actually arises. For example, Nginx in a container on Docker Swarm is far more simpler while also being a bit more flexible than Kubernetes ingress: just this week i needed to expose a Sonatype Nexus Docker registry on server port 5000, 5001, ... whilst also wanting the reverse proxy/ingress to handle SSL/TLS termination with a wildcard cert. Kubernetes ingress doesn't support these non-standard ports, whereas NodePort would have necessitated running on the 32000-... port range and even then i'd need to setup a separate Nginx instance if i wanted to use it for certificate management, as opposed to importing the certs into Nexus itself, which kind of defeats the purpose. Also, needing to trick a Traefik ingress (the default in k3s) into using a custom default wildcard cert necessitated a HelmChartConfig, ConfigMap, Secret (TLS) and TLSStore, which seems badly documented, cryptic and also just wasn't too user friendly - and felt like a step back when compared to just needing a nginx.conf (or the Traefik equivalent) and the actual .crt and .key files. Whereas with Docker Swarm, i can just have one Nginx container and expose whatever ports i want, because i'm not pigeonholed into a particular type of abstraction - however, that comes at the expense of not having easy to configure ingress, since i need to change the Nginx config manually. Regardless, the simplicity of Docker Swarm and the likes of Hashicorp Nomad is actually a boon - in the case of the former, pair it up with Portainer and it should be pretty smooth sailing from there onwards, even easier than using Rancher with Kubernetes! Admittedly, it all probably depends on the scale that you operate at - because once you really need to think about node draining, resource rebalancing upon adding new nodes, autoscaling and other advanced concerns, then you might want to look into Kubernetes, but for the most companies and projects out there, that's hardly necessary. For most of them, regular containers with some lightweight orchestrator running on top should be more than enough. I think Docker here wins out as the runtime as well as opposed to containerd or podman, because of its historical significance and how much software has been build around it (hell, even Rancher in Docker basically takes the Kubernetes in Docker approach which is actually really nice, apart from it all breaking on newer RPM distros because of a lack of cgroups v2 support). Now, there are other container technologies out there as well (e.g. LXC or some might even suggest the BSD jails), but i think that we're not in too bad of a spot right now. Luckily alternatives to Kubernetes are still maintained and are pretty usable, sometimes even Docker Compose can be enough for prototyping or dev environments and if you ever find yourself needing to look into using Kubernetes, projects like k0s or k3s are pretty much excellent for that. Of course, i'd also suggest a modicum of supporting software, like Rancher for a nice web UI and Rancher Desktop to not have to pay for Docker Desktop, if you don't need it. In my eyes, this approach also fully eliminates vendor lock - all i need is a bunch of VPSes on any cloud host, on which i can initialize a container cluster with the likes of Ansible and migrating is similarly easy (especially with bind mounts for storage). PaaS platforms can be nice for when you don't want to deal with managing your own servers, but i much prefer to keep my apps as portable as possible. That's not to say that the PaaS approach is not valid, as it can indeed save a lot of time and effort, if you haven't invested in automating the setup and management of your own nodes!
ipdashc 5y ago
Huh, this is kind of an odd post to me. I've used Docker for several years for many projects, plus other container technologies (LXC, systemd-nspawn). But I've never touched Kubernetes in my life. Could barely tell you anything about it. It's just never been necessary. Is Kubernetes really that popular out there in the wider software world? Or is this author just in a bubble?
- duskwuff 5y ago
  Have you ever used docker-compose to define a system architecture consisting of multiple containers? If so: Kubernetes is kind of like a fancier version of that which can scale to a large cluster, along with tools for handling common tasks associated with these deployments like name resolution, HTTP/S ingress, automatic scaling, and periodic tasks. It scratches a lot of itches for large-scale production environments.
- KronisLV 5y ago
  > Is Kubernetes really that popular out there in the wider software world? I guess it could be a regional thing, but for the most part i'd be inclined to say yes. Most of the larger cloud vendors offer managed Kubernetes in one way or another (Azure, GCP, AWS, Scaleway etc.), which you cannot say about any other orchestrator out there. Most of the large enteprises that i've worked with also are moving towards introducing Kubernetes internally, sometimes with pretty large clusters for multiple projects at once! Similarly, most of the tooling out there is focused on Kubernetes nowadays (networking, storage, deployment solutions), which isn't the case for other orchestrators. Of course, many out there don't even need an orchestrator - Docker or even Docker Compose (or the podman equivalents, or other solutions like that) are oftentimes enough for smaller scale projects, it can be a while before you would enjoy the benefits from the likes of Docker Swarm, Hashicorp Nomad or even one of the many Kubernetes distros out there. Once you need to scale past one (or a few servers) and need to orchestrate deployments across all of them, it might just be worth to look into them, though.
  - intrepidsoldier 5y ago
    Kubernetes is indeed popular. Inexplicably so, I must admit.

For Google cloud, I think before Cloud Run existed, they created a "create-with-container" option that deploys a container on a VM for you. https://cloud.google.com/sdk/gcloud/reference/compute/instances/create-with-container https://cloud.google.com/sdk/gcloud/reference/compute/instan... (I'm a googler, but do not work anywhere near your cloud. I used this for a personal project.)

jillesvangurp 5y ago
That still exists. We actually switched to that from Cloud Run. Cloud Run was a little too restrictive with background threads in our application. These basically get throttled in between requests and we kind of needed them not to be throttled. Otherwise it is actually great and works as advertised. The process of switching to using simple vms was a gcloud one liner to spin up a vm with the same docker image. That basically worked the first time we tried it. We ran with this for a few months. Then we finally sat down and got ourselves a load balancer with an instance group and an instance template. We deploy by telling gcloud to replace the template on the instance group. It does a rolling restart and it uses our health checks to ensure it does the right things. And it's stupidly easy to script together. We use Github actions for automated deploys. It does rolling restarts, has health checks, monitoring, logging, etc. built in. Everything you need to run a simple monolith. It's similar to what Amazon did with ECS. Just less convoluted and way more easier to use. People assume they need all this crazy complicated infrastructure. But it's all commodity off the shelf stuff these days and it doesn't have to be that hard to set up. Could we automate the above? Sure. But what's the point. It's 1 command to create instance templates, another command to tell the LB to run it. Provisioning the LB with a wild card certificate is a five minute job using the UI. It will tell you the commands if you need them. But it's something you do once per environment.
- intrepidsoldier 5y ago
  Agreed. People choose technologies for their own reasons. The full spectrum is surviving and thriving: https://ramansharma.substack.com/p/multiple-stops-on-the-cloud-compute-continuum-all-with-their-own-merits-6b921ed82f2 https://ramansharma.substack.com/p/multiple-stops-on-the-clo...
pqb 5y ago
I am shocked, because I see it for the first time. Recently, I have been browsing gcloud documentation like a crazy to prepare myself to PCA certification but I have never seen this doc page before [0]. Thanks a lot for a mentioning it! [0]: https://cloud.google.com/compute/docs/containers/deploying-containers https://cloud.google.com/compute/docs/containers/deploying-c...

trabant00 5y ago

My opinion at the moment, subject to change: Docker and Kubernetes have a high price as they add a lot of complexity and make debugging harder. Especially for networking and state-full. Most of the problems they claim to solve where already solved years ago by more mature solutions: - reproducible builds and dependencies management: distribution package managers - high availability: load balancers, keepalive, dynamic routing, clustering and most importantly: data replication (where is this in k8s?) since downtime is nothing compared to data loss - resource limitation: cgroups - isolation: the good old user and groups, vlans, firewalls - orchestration, rollouts, rollbacks: CD pipelines, configuration management Technically the only thing I find easier with k8s is bin packing. The real advantage I find in containers and k8s is they became a standard. You don't have to learn the architecture that each company has designed for itself and dig into custom built solutions. So you can easily switch jobs and they can easily replace you. If you throw that away with custom container implementations for each PaaS offering and fragmentation of orchestrator solutions then you only get the complexity and no benefit. We're going to see in a few more years where it goes but I'm not exactly optimistic.

dilyevsky 5y ago
“Kubernetes is too complex - let me just string together 2/3rds of what it does with a few thousand lines of bash…”
- dijit 5y ago
  There has to be a name for the fallacy that stipulates “the alternative to complex systems is bash”.
  - dilyevsky 5y ago
    It maybe a fallacy alright but this exactly what usually happens
- stephenr 5y ago
  TBH I'd rather read a few thousand lines of bash than the equivalent in Yet Another Migraine Looming spread across who knows how many containers that do who knows what. Source: I fix people's infra when it's fucked. By far the most fucked to date, are Kubernetes/etc put in place to solve problems solved with much simpler solutions.

tyingq 5y ago

LXD is worth a look if you want central management of containers with very little in the way of extra layers. It can also spin up normal VMs if containers don't work for you. And the control plane data is spread across a raft-enabled and distributed sqlite.

Helmut10001 5y ago
In case anyone is confused with the terms or tools, see the Proxmox [1] docs for an explanation of the differences between LXC vs LXD vs Proxmox Containers vs Docker [1]: https://pve.proxmox.com/wiki/FAQ https://pve.proxmox.com/wiki/FAQ
- zekrioca 5y ago
  Yes, I do think they should have chosen a better name than "LXD". It really confuses itself with LXC, specially so because all of its commands start with "lxc"..

josteink 5y ago

I use LXC for isolation with VM-like environments, without the regular VM overhead. Never bothered with Docker or Kubernetes, as I already have my needs solved. Call me a dinosaur if you like, but it’s fundamentally simple and it works for me. To expand on the title: Containers are not just Docker.

zekrioca 5y ago
You would be impressed by LXD. If you ever needs LXC containers working together as in a cluster, try it out.

KronisLV 5y ago

> Along came the Kubernetes wave. It wasn’t meant to solve any of the above problems. It was supposed to provide efficient utilization of compute resources while improving reliability and availability. So, both containers and Kubernetes serve different purposes. I feel like Kubernetes tried doing way more than that: it added additional abstractions like ingresses and load balancers, abstracted away storage with its drivers and PVC concept, added support for all sorts of cases when something should be run (e.g. if there's a task you'd like to schedule or even just run once), introduced namespaces and its API is pretty extensive and extensible, especially with CRDs. Just look at how many projects have propped up around it, from networking solutions (Calico, Flannel, Istio, linkerd, ...), to storage solutions (Longhorn, Rook, ...), to all sorts of additional software, like Helm, Kustomize, Skaffold etc. Of course, that's not to say that everyone needs all of that complexity, as others have rightfully pointed out in the comments. Personally, i think that the sweet spot is focusing on OCI as the standard to go for and hosting most of the software that you need inside of containers with a lightweight orchestrator, only moving to something more complicated once the need actually arises. For example, Nginx in a container on Docker Swarm is far more simpler while also being a bit more flexible than Kubernetes ingress: just this week i needed to expose a Sonatype Nexus Docker registry on server port 5000, 5001, ... whilst also wanting the reverse proxy/ingress to handle SSL/TLS termination with a wildcard cert. Kubernetes ingress doesn't support these non-standard ports, whereas NodePort would have necessitated running on the 32000-... port range and even then i'd need to setup a separate Nginx instance if i wanted to use it for certificate management, as opposed to importing the certs into Nexus itself, which kind of defeats the purpose. Also, needing to trick a Traefik ingress (the default in k3s) into using a custom default wildcard cert necessitated a HelmChartConfig, ConfigMap, Secret (TLS) and TLSStore, which seems badly documented, cryptic and also just wasn't too user friendly - and felt like a step back when compared to just needing a nginx.conf (or the Traefik equivalent) and the actual .crt and .key files. Whereas with Docker Swarm, i can just have one Nginx container and expose whatever ports i want, because i'm not pigeonholed into a particular type of abstraction - however, that comes at the expense of not having easy to configure ingress, since i need to change the Nginx config manually. Regardless, the simplicity of Docker Swarm and the likes of Hashicorp Nomad is actually a boon - in the case of the former, pair it up with Portainer and it should be pretty smooth sailing from there onwards, even easier than using Rancher with Kubernetes! Admittedly, it all probably depends on the scale that you operate at - because once you really need to think about node draining, resource rebalancing upon adding new nodes, autoscaling and other advanced concerns, then you might want to look into Kubernetes, but for the most companies and projects out there, that's hardly necessary. For most of them, regular containers with some lightweight orchestrator running on top should be more than enough. I think Docker here wins out as the runtime as well as opposed to containerd or podman, because of its historical significance and how much software has been build around it (hell, even Rancher in Docker basically takes the Kubernetes in Docker approach which is actually really nice, apart from it all breaking on newer RPM distros because of a lack of cgroups v2 support). Now, there are other container technologies out there as well (e.g. LXC or some might even suggest the BSD jails), but i think that we're not in too bad of a spot right now. Luckily alternatives to Kubernetes are still maintained and are pretty usable, sometimes even Docker Compose can be enough for prototyping or dev environments and if you ever find yourself needing to look into using Kubernetes, projects like k0s or k3s are pretty much excellent for that. Of course, i'd also suggest a modicum of supporting software, like Rancher for a nice web UI and Rancher Desktop to not have to pay for Docker Desktop, if you don't need it. In my eyes, this approach also fully eliminates vendor lock - all i need is a bunch of VPSes on any cloud host, on which i can initialize a container cluster with the likes of Ansible and migrating is similarly easy (especially with bind mounts for storage). PaaS platforms can be nice for when you don't want to deal with managing your own servers, but i much prefer to keep my apps as portable as possible. That's not to say that the PaaS approach is not valid, as it can indeed save a lot of time and effort, if you haven't invested in automating the setup and management of your own nodes!

ipdashc 5y ago

Huh, this is kind of an odd post to me. I've used Docker for several years for many projects, plus other container technologies (LXC, systemd-nspawn). But I've never touched Kubernetes in my life. Could barely tell you anything about it. It's just never been necessary. Is Kubernetes really that popular out there in the wider software world? Or is this author just in a bubble?

duskwuff 5y ago
Have you ever used docker-compose to define a system architecture consisting of multiple containers? If so: Kubernetes is kind of like a fancier version of that which can scale to a large cluster, along with tools for handling common tasks associated with these deployments like name resolution, HTTP/S ingress, automatic scaling, and periodic tasks. It scratches a lot of itches for large-scale production environments.
KronisLV 5y ago
> Is Kubernetes really that popular out there in the wider software world? I guess it could be a regional thing, but for the most part i'd be inclined to say yes. Most of the larger cloud vendors offer managed Kubernetes in one way or another (Azure, GCP, AWS, Scaleway etc.), which you cannot say about any other orchestrator out there. Most of the large enteprises that i've worked with also are moving towards introducing Kubernetes internally, sometimes with pretty large clusters for multiple projects at once! Similarly, most of the tooling out there is focused on Kubernetes nowadays (networking, storage, deployment solutions), which isn't the case for other orchestrators. Of course, many out there don't even need an orchestrator - Docker or even Docker Compose (or the podman equivalents, or other solutions like that) are oftentimes enough for smaller scale projects, it can be a while before you would enjoy the benefits from the likes of Docker Swarm, Hashicorp Nomad or even one of the many Kubernetes distros out there. Once you need to scale past one (or a few servers) and need to orchestrate deployments across all of them, it might just be worth to look into them, though.
- intrepidsoldier 5y ago
  Kubernetes is indeed popular. Inexplicably so, I must admit.