22 ms·

The container orchestrator landscape

271 points by chmaynard 4y ago

jordemort 4y ago
Hi, I'm the author of the article. This is my third for LWN and a follow-up to my previous one about container engines ( https://lwn.net/Articles/902049/ https://lwn.net/Articles/902049/ ) - I wasn't initially planning on writing this since I only have a wee bit of k8s experience, but enough people asked for it that I felt that I should. Please let me know if you'd like me to write about anything else :)
- igtztorrero 4y ago
  Super info, congratulations. I keep conclusions: Ten years ago, very little of this technology even existed, and things are still evolving quickly. Thanks
- heipei 4y ago
  Thanks for the great post! One small addition: Nomad recently implemented basic service discovery functionality into Nomad itself, so for simple use-cases you might get away with just a Nomad cluster without Consul running alongside: https://www.hashicorp.com/blog/nomad-service-discovery https://www.hashicorp.com/blog/nomad-service-discovery
- MoOmer 4y ago
  Great post - I hadn't heard of Nomad Pack, and I have a small-ish deployment of ~20 nodes. There are many reasons to love Nomad, like its straight-forward architecture, how it plays well with other binaries to augment its functionality (Consul & Vault), super easy service mesh with Consul Connect + Traefik, and the ease of defining complex applications/architectures with the Job Spec [0]. I've used k8s a few times for work, but I just love Nomad. [0]: https://www.nomadproject.io/docs/job-specification https://www.nomadproject.io/docs/job-specification
  - kungfufrog 4y ago
    I love nomad too. Quick question, how have you solved ingress using nomad + consul connect for service mesh? The big thing for me is needing mTLS support for some of my services.
    - chucky_z 4y ago
      Yea consul connect is a good choice for mTLS, it's enforced by default with no way to turn it off. You can also go DIY with something like Traefik and just configure everything to require TLS with client TLS pass-through for enforced mutual TLS, that puts you into 'require a set of central load balancers' but you still get all the benefits.
    - MoOmer 4y ago
      I use Consul Connect + Traefik annotations, and am planning to write about it soon. If you want a sneak peak, I'm happy to screen-share the config to get you going, ping me at the email in my profile!
  - candiddevmike 4y ago
    I'm not sure where I'd use Nomad Pack vs Waypoint or Terraform. Any HashiCorp folks care to chime in?
    - schmichael 4y ago
      Great question. I know it can be frustrating that we don't answer this directly anywhere, but Pack and Waypoint are evolving so quickly the guidance would likely need regular updating. HashiCorp has always tried to loosely couple our products so that while they work Better Together, they also work independently and allow users to choose the bits and pieces that fit their needs best. My general suggestions for this question are that Terraform is what you should be creating your Nomad clusters with. In the cloud you can use it to define your autoscaling groups, load balancers, and use Packer to create golden images for Nomad servers and clients. While Terraform is also useful for deploying "base" system jobs like log shippers or metrics aggregators, I generally don't recommend deploying all of your workloads with Terraform. You generally want more fine grained control over things like promoting-canaries-during-deployments than Terraform's architecture or interface could possibly support. So now your left with how to run all of your workloads. Lots of folks hit the Nomad API or use the Nomad CLI straight from the CI runners, but Waypoint and Pack offer some useful tools. Waypoint has more application developer oriented tooling than Pack. It's a developer experience oriented tool where Pack is more like Helm and oriented toward offering a package management solution for clusters. Since Waypoint manages building your project, it's a much more opinionated tool than Pack. You likely want to use Waypoint throughout your application development process. Pack is much more useful if you just want to "package up" a bunch of existing code, whether your own or 3rd party, and run it in your cluster. It's intended to eventually have an extensive public registry/catalog of popular software ready to run in your cluster. This isn't an area Waypoint is concerned with at all. Hope that helps!
- abriosi 4y ago
  Congratulations on the article. It is always nice to read sober writing Personally our team found a match using docker swarm. The reality is that we don't have a DevOps role and we needed something that anyone could deploy on Although the general sentiment is negative towards docker swarm, its simplicity is hard to beat This guide was useful for setting up our stack https://dockerswarm.rocks/ https://dockerswarm.rocks/
  - bityard 4y ago
    > It is always nice to read sober writing Are you used to reading drunken writing? :) I read this earlier too and was very impressed at how everything was explained in plain english. I wish I could send this article to myself 2-3 years ago, it would have saved me literally days of trying to figure out what parts of k8s do what and why.
    - benhoyt 4y ago
      > plain English Agreed. LWN is really good at this, partly because their editors have really good "tech BS" detectors (learned this when I was writing for them). It means you really have to understand what you're saying and can't just parrot buzzwords or lingo you've read on a tool's marketing website.
- chucky_z 4y ago
  One point around Nomad is that the point on it requiring low latency.... isn't really true. I run a cluster with some servers physically on different coasts of the United States with clients as far as Singapore. It all just works :). (This does need some tuning but once it works, it just works)
  - schmichael 4y ago
    Our docs are a bit overly conservative in their suggestions. Glad that configuration is working out for you! Clients (aka nodes-actually-running-your-work) can be run globally distributed without issue, and we've even added some more features to help tune their behavior: https://www.hashicorp.com/blog/managing-applications-at-the-edge-with-hashicorp-nomad https://www.hashicorp.com/blog/managing-applications-at-the-... Disclaimer: I am the HashiCorp Nomad Engineering Team Lead
- schmichael 4y ago
  Disclaimer: I'm the HashiCorp Nomad Engineering Team Lead This may be my favorite article on the subject, and I've read a lot! Others have suggested some minor corrections but overall you've done a fantastic job accurately and fairly portraying Nomad as far as I'm concerned. I really appreciate the final paragraph as I constantly field questions about whether or not a k8s-only world is a foregone conclusion. I think there's room for more than one option, and I'm glad you noticed HashiCorp thinks so too!
  - jordemort 4y ago
    Thanks, I'm glad you liked it! I've used Nomad a bit at a previous job; due to its simplicity, we were much more comfortable embedding it in the virtual appliance images we were building. Nomad meant we had to rewrite the k8s configs that the hosted version of the app was using, but even with one of the all-in-one k8s distros, we were worried about shipping that many hidden moving parts.
  - jerrac 4y ago
    What options are there for managing persistent storage in Nomad? I've been through the very basic initial getting started tutorial, but haven't had a chance to dig any deeper. When I was trying kube, I found Longhorn to be the one thing I liked. Is there anything like that for Nomad? I've been trying to get a MariaDB Galera cluster running on Docker Swarm for the past couple weeks and have been somewhat stymied a few things. Something like Longhorn would help a lot.
    - schmichael 4y ago
      Nomad supports CSI like Kubernetes. Unfortunately many CSI storage providers assume kubernetes or adhere to kubernetes behavior even when it deviates from the spec, so there are gotchas. That being said they’re often glaring. So please report an issue to Nomad and the upstream of you hit an issue! Nomad has 2 other storage features: 1. Host volumes where nodes can advertise a volume by name and jobs can request being placed on nodes with that volume available. I think a lot of folks run databases this way so they can statically define their database nodes and not worry about the numerous runtime points of failure CSI introduces. 2. Ephemeral disk stickiness and migration: during deployments an instance of a job’s intrinsic local storage can be set to either get reused on the same node (sticky) or migrated to wherever the new instance is placed. Ephemeral disks is the oldest storage solution and is only a best effort (unlike CSI which may be able to reliably reattach and existing volume to a new node in the case of node failure). However I’ve heard of a lot of people happily using it for already distributed databases like Elasticsearch or Cassandra where the database can tolerate nomad’s best effort failing. (Sorry for lack of links as I’m on mobile. Googling the key phrases you care about should get you to our docs quickly and easily)
- orsenthil 4y ago
  > Hi, I'm the author of the article. Wow. I really enjoyed this. Your writing was clear, simple and explained the landscape while navigating the marketing terms extremely well. I wish to read other articles that you have written.
  - jordemort 4y ago
    I've only written three so far, including this one; you'll find them all at LWN. Buying a subscription to LWN is the best way to ensure that they keep paying me to write more :D
    - orsenthil 4y ago
      Yup, I am a subscriber.
- POPOSYS 4y ago
  Thanks for your efforts, very nice article! It would be great if you took some time to write about the question of climate change and energy waste. In this regard it would be very interesting to better understand the impact tools like K8S have on global warming. Just from a pure practical POV I see many clusters burning a lot of CPU cycles with K8S processes that do nothing else than providing some kind of infrastructure for the real computing task. I have seen K8S burning 50% and more CPU time - this with not one single payload operation, just doing its cluster thing! This is a horrible waste of energy and we can not go ahead establishing such a bloated monstrosity as default data center OS of the future. Not all data centers are 100% solar - I guess people in the field are LOL at this point knowing the realities. So I personally come to the conclusion that a thing like K8S is the opposite of what is needed in a world that is heading to a climate disaster. So many hours spent into optimizing the Linux kernel (successfully!) are nullified by a tool that wastes energy recklessly. We need more than one article to help programmers of these tools understand why they should do their jobs with more love for future generations. Also we need lots of articles about how to write efficient code that does not waste CPU cycles.
- POPOSYS 4y ago
  Ah, and one more from another perspective: A very interesting series of articles would be "How to take down your K8S cluster" describing the many ways a cluster can be [mis]configured to easily be brought down by one single error / pod / application.
cad1 4y ago
Maybe it is implied by being on lwn.net, but the list is limited to open source orchestration platforms. In a broader discussion I would expect to see AWS ECS mentioned.
- jordemort 4y ago
  The implication is correct; I did not write about ECS because it is not open source.
  - hkrgl 4y ago
    Out of curiosity, what other non-open source platforms do people use?
- Kaytaro 4y ago
  They mentioned it indirectly- > "seemingly every tech company of a certain size has its own distribution and/or hosted offering to cater to enterprises"
KronisLV 4y ago
This seems like a pretty well written overview! As someone who rather liked Docker Swarm (and still likes it, running my homelab and private cloud stuff on it), it is a bit sad to see it winding down like it, even though there were attempts to capitalize on the nice set of simple functionality that it brought to the table like CapRover: https://caprover.com/ https://caprover.com/ Even though there is still some nice software to manage installs of it, like Portainer: https://www.portainer.io/ https://www.portainer.io/ (which also works for Kubernetes, like a smaller version of Rancher) Even though the resource usage is far lower than that of almost any Kubernetes distro that I've used (microk8s, K3s and K0s included), the Compose format being pretty much amazing for most smaller deployments and Compose still being one of the better ways to run things locally in addition to Swarm for remote deployments (Skaffold or other K8s local cluster solutions just feel complex in comparison). And yet, that's probably not where the future lies. Kubernetes won. Well, Nomad is also pretty good, admittedly. Though if you absolutely do need Kubernetes, personally I'd suggest that you look in the direction of Rancher for a simple UI to manage it, or at least drill down into the cluster state, without needing too much digging through a CLI: https://rancher.com/ https://rancher.com/ Lots of folks actually like k9s as well, if you do like the TUI approach a bit more: https://k9scli.io/ https://k9scli.io/ But for the actual clusters, assuming that you ever want to self-host one, ideally a turnkey solution, RKE is good, K0s is also promising, but personally I'd go with K3s: https://k3s.io/ https://k3s.io/ which has been really stable on DEB distros and mostly works okay on RPM ones (if you cannot afford OpenShift or to wait for MicroShift). My only pet peeve being that the Traefik ingress is a little bit under-documented (e.g. how to configure common use cases, like a SSL certificate, one with an intermediate certificate, maybe a wildcard, or perhaps just use Let's Encrypt, how to set defaults vs defining them per domain). For the folks with thicker wallets, though, I'd suggest to just give in and pay someone to run a cluster for you: that way you'll get something vaguely portable, will make lots of the aspects in regards to running it someone else's problem and will be able to leverage the actual benefits of working with the container orchestrator. Oh, also Helm is actually pretty nice, since the templates it generates for you by default are actually reasonable and can serve as a nice starting point for your own application deployment deescriptions, versus trying to write your own Kubernetes manifest from 0 which feels like an uphill battle, at least when compared with the simplicity of Compose. In regards to Nomad, however, I feel like a UI that would let you click around and get bunches of HCL to describe the deployment that you want would be pretty cool. At least when starting out, something like that can really flatten the learning curve. That's also kind of why I like Portainer or Rancher so much, or even something like: https://k8syaml.com/ https://k8syaml.com/ > To extend its reach across multiple hosts, Docker introduced Swarm mode in 2016. This is actually the second product from Docker to bear the name "Swarm" — a product from 2014 implemented a completely different approach to running containers across multiple hosts, but it is no longer maintained. It was replaced by SwarmKit, which provides the underpinnings of the current version of Docker Swarm. On an unrelated note, this, at least to me, feels like pretty bad naming and management of the whole initiative, though. Of course, if the features are there, it shouldn't be enough to scare anyone away from the project, but at the same time it could have been a bit simpler.
throwaway787544 4y ago
The main purpose of K8s is just to run microservices. It makes things convenient by automatically restarting services, running multiple copies across servers, routing from one load-balancing port to many ephemerally-allocated ports, collecting logs, providing a consistent control API, limit system resources, networking, secrets... ...All things Systemd does. Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone. All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. Managed Distributions of K8s go away if every Linux distro can do the same things out of the box. "Managing fleets" just becomes running more Linux VMs or not. You wouldn't need "clusters", you'd just run a bunch of boxes and run replicas of whatever service you wanted, and all resources would be managed using normal Linux system permissions and Systemd configs. No volume mounts or persistent storage setup, it's just regular filesystem mounts and disks. Containers would be optional, eliminating a huge class of complexity (static apps make a lot of containerization moot). And most importantly to admins - no more bullshit 15 month upgrade death march. Imagine just sticking to a stable Debian for 3 years but still having all the power of a distributed orchestrator. Imagine not having 50 components with APIs listening on the internet to be hacked. Imagine just using IPTables rather than custom plugins that change based on the Cloud provider just to filter network traffic, and not letting everything on the network access everything else by default. K8s is just too expensive, complicated, and time consuming to survive. As soon as Systemd can network as a cluster, the writing's on the wall. The only reason it hasn't happened yet is the Systemd developers are basically desktop-oriented Linux fogies rather than Cloud-focused modern devs. I also predict RedHat will block patches to make this happen, since their money comes from K8s. But someone just needs to make a Systemd fork using Systemd's own API and we can finally fix all the crap Systemd devs like to pretend is somebody else's bug.
- deleted 4y ago
  [deleted]
- loriverkutya 4y ago
  I have the "If you have a SystemD everything is a nail" feeling from this comment.
bamazizi 4y ago
One thing missing from the article is the "load balancer" aspect of the orchestrators. I believe K8S favours big cloud providers tremendously due to load balancer lock-in or handcuff or whatever else you want to call it. For Kubernetes there's a WIP Metallb but I personally haven't had confident success running on my own servers. K8S setup complexity is already highlighted but I'd add upgrade and maintenance pain to the mix as well. I've only recently started with Nomad and while it's so simple, I question my self at every step because coming from K8S I've gotten used to complexity and over configuration. Nomad allows you to bring your own loadbalancer, and i'm experimenting with Traefik & Nginx now.
- verdverm 4y ago
  k8s has good, non-cloud based ingress controllers. https://github.com/kubernetes/ingress-nginx https://github.com/kubernetes/ingress-nginx https://doc.traefik.io/traefik/providers/kubernetes-ingress/ https://doc.traefik.io/traefik/providers/kubernetes-ingress/
  - bamazizi 4y ago
    To run k8s on bare metal server clusters there's only Metallb option to bring user traffic into clusters NodePort services, via externally exposed IP address(es). I wasn't talking about k8s internal load balancers and reverse proxies
    - verdverm 4y ago
      You were talking about cloud, now you are talking about bare metal? Of course you have to route traffic to your cluster, but you are implying some cloud based load balancer lockin which just isn't true
      - chucky_z 4y ago
        I believe the OP is talking about LoadBalancer (https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/...), and I believe you're talking about Ingress (https://kubernetes.io/docs/concepts/services-networking/ingress/ https://kubernetes.io/docs/concepts/services-networking/ingr...).
hellohowareu 4y ago
I am trying to learn Kubernetes for work-- Docker + K8s + Helm. Does anyone have recommendation(s) regarding books, video courses, or live in-person trainings?
- paulgb 4y ago
  I learned a lot of the concepts just by typing search terms into YouTube. “TechWorld with Nana” is one channel that comes up a lot and has good, digestible explanations. Hussein Nasser too, but he has less content directly about Docker/k8s and more about networking. It's excellent though.
- mmh0000 4y ago
  For live training, I can not recommend Guru Labs[0] highly enough. They offer many courses, but specific to your question, lots of container related courses[1][2][3][4]. [0] https://www.gurulabs.com/linux-training/courses/ https://www.gurulabs.com/linux-training/courses/ [1] https://www.gurulabs.com/linux-training/courses/GL340/ https://www.gurulabs.com/linux-training/courses/GL340/ [2] https://www.gurulabs.com/linux-training/courses/GL360/ https://www.gurulabs.com/linux-training/courses/GL360/ [3] https://www.gurulabs.com/linux-training/courses/GL355/ https://www.gurulabs.com/linux-training/courses/GL355/ [4] https://www.gurulabs.com/linux-training/courses/GL390/ https://www.gurulabs.com/linux-training/courses/GL390/
- danwee 4y ago
  For K8s: 1. Read the book "Kubernetes up and Running" 2. After that you can start reading the https://kubernetes.io https://kubernetes.io documentation 3. Alongside the documentation reading, practise what you have learnt using minikube. 4. Only after, start with Helm (which is pretty bad, but if your company is using it...)
  - hellohowareu 4y ago
    Thanks! I was just curious about your perspective-- When do you mean by Helm is pretty bad? Could you elaborate? Also, how does it compare with alternatives?
- 4y ago
Alir3z4 4y ago
Docker Swarm all the way. Started with 1 node acting as manager and worker at the same time. Then added more 2 nodes, basically 3 nodes all acting as manager and workers at the same. Later kept 3 managers and 3 workers and now around 10 workers with the 3 managers. The cluster is full of apps, celery workers, static apps or SSR. To manage it I didn't use any UI. The CLI is really simple to use. Later to give less experienced people access to it i instant Portrainer. It has never stopped working, either upgrading or messing around on production environments. It just works. The only and only complain I have id the way it mess around with IPtable rules I have. The docker swarm networking overrides them. There are several approaches, while they work, they always feel like a hack to be honest. But yeah, they work. It's such a wonderful simple to use software that gets thing.
- JoyfulTurkey 4y ago
  I fully agree. Docker Swarm was perfect for my small employer with an IT staff of 3 people. Went with the 3 manager nodes, as recommended by Docker, to allow for upgrades without downtime, and things were rock solid. Also liked that encryption of the overlay networks was included. The only thing we missed/wanted was auto-scaling built-in, but we rarely had a need to scale the number of containers.
- bityard 4y ago
  What are you doing for storage? We're thinking of deploying Docker Swarm where I work onto a VMWare cluster and basically the options boil down to sharing a storage volume between VMs (which vSphere makes difficult) and replicating storage across nodes with something like gluster (which drastically increases storage used and has performance implications).
  - sofixa 4y ago
    > We're thinking of deploying Docker Swarm where I work onto a VMWare cluster IMHO that's an anti-pattern. Why would you run two orchestrators on top of one another? Containers also cause scheduling contention on the vSphere side, on top of the waste of CPU from the virtualisation layer. Unless you're in some very specific circumstances, run your container orchestrator on bare metal when on prem.
yummybear 4y ago
I do miss the days when you'd FTP to your server and sync changes.
- bityard 4y ago
  This is pretty much what I do with Ansible, and now my whole server configuration lives in a git repo that I can access from anywhere.
lifeisstillgood 4y ago
I was on a thread recently looking for this decades Byte magazine. How did we forget to mention lwn? Great article :-)
barberpole 4y ago
So it's not about canned orchestration?

271 points by chmaynard 4y ago

Hi, I'm the author of the article. This is my third for LWN and a follow-up to my previous one about container engines ( https://lwn.net/Articles/902049/ https://lwn.net/Articles/902049/ ) - I wasn't initially planning on writing this since I only have a wee bit of k8s experience, but enough people asked for it that I felt that I should. Please let me know if you'd like me to write about anything else :)

igtztorrero 4y ago
Super info, congratulations. I keep conclusions: Ten years ago, very little of this technology even existed, and things are still evolving quickly. Thanks
heipei 4y ago
Thanks for the great post! One small addition: Nomad recently implemented basic service discovery functionality into Nomad itself, so for simple use-cases you might get away with just a Nomad cluster without Consul running alongside: https://www.hashicorp.com/blog/nomad-service-discovery https://www.hashicorp.com/blog/nomad-service-discovery
MoOmer 4y ago
Great post - I hadn't heard of Nomad Pack, and I have a small-ish deployment of ~20 nodes. There are many reasons to love Nomad, like its straight-forward architecture, how it plays well with other binaries to augment its functionality (Consul & Vault), super easy service mesh with Consul Connect + Traefik, and the ease of defining complex applications/architectures with the Job Spec [0]. I've used k8s a few times for work, but I just love Nomad. [0]: https://www.nomadproject.io/docs/job-specification https://www.nomadproject.io/docs/job-specification
- kungfufrog 4y ago
  I love nomad too. Quick question, how have you solved ingress using nomad + consul connect for service mesh? The big thing for me is needing mTLS support for some of my services.
  - chucky_z 4y ago
    Yea consul connect is a good choice for mTLS, it's enforced by default with no way to turn it off. You can also go DIY with something like Traefik and just configure everything to require TLS with client TLS pass-through for enforced mutual TLS, that puts you into 'require a set of central load balancers' but you still get all the benefits.
  - MoOmer 4y ago
    I use Consul Connect + Traefik annotations, and am planning to write about it soon. If you want a sneak peak, I'm happy to screen-share the config to get you going, ping me at the email in my profile!
- candiddevmike 4y ago
  I'm not sure where I'd use Nomad Pack vs Waypoint or Terraform. Any HashiCorp folks care to chime in?
  - schmichael 4y ago
    Great question. I know it can be frustrating that we don't answer this directly anywhere, but Pack and Waypoint are evolving so quickly the guidance would likely need regular updating. HashiCorp has always tried to loosely couple our products so that while they work Better Together, they also work independently and allow users to choose the bits and pieces that fit their needs best. My general suggestions for this question are that Terraform is what you should be creating your Nomad clusters with. In the cloud you can use it to define your autoscaling groups, load balancers, and use Packer to create golden images for Nomad servers and clients. While Terraform is also useful for deploying "base" system jobs like log shippers or metrics aggregators, I generally don't recommend deploying all of your workloads with Terraform. You generally want more fine grained control over things like promoting-canaries-during-deployments than Terraform's architecture or interface could possibly support. So now your left with how to run all of your workloads. Lots of folks hit the Nomad API or use the Nomad CLI straight from the CI runners, but Waypoint and Pack offer some useful tools. Waypoint has more application developer oriented tooling than Pack. It's a developer experience oriented tool where Pack is more like Helm and oriented toward offering a package management solution for clusters. Since Waypoint manages building your project, it's a much more opinionated tool than Pack. You likely want to use Waypoint throughout your application development process. Pack is much more useful if you just want to "package up" a bunch of existing code, whether your own or 3rd party, and run it in your cluster. It's intended to eventually have an extensive public registry/catalog of popular software ready to run in your cluster. This isn't an area Waypoint is concerned with at all. Hope that helps!
abriosi 4y ago
Congratulations on the article. It is always nice to read sober writing Personally our team found a match using docker swarm. The reality is that we don't have a DevOps role and we needed something that anyone could deploy on Although the general sentiment is negative towards docker swarm, its simplicity is hard to beat This guide was useful for setting up our stack https://dockerswarm.rocks/ https://dockerswarm.rocks/
- bityard 4y ago
  > It is always nice to read sober writing Are you used to reading drunken writing? :) I read this earlier too and was very impressed at how everything was explained in plain english. I wish I could send this article to myself 2-3 years ago, it would have saved me literally days of trying to figure out what parts of k8s do what and why.
  - benhoyt 4y ago
    > plain English Agreed. LWN is really good at this, partly because their editors have really good "tech BS" detectors (learned this when I was writing for them). It means you really have to understand what you're saying and can't just parrot buzzwords or lingo you've read on a tool's marketing website.
chucky_z 4y ago
One point around Nomad is that the point on it requiring low latency.... isn't really true. I run a cluster with some servers physically on different coasts of the United States with clients as far as Singapore. It all just works :). (This does need some tuning but once it works, it just works)
- schmichael 4y ago
  Our docs are a bit overly conservative in their suggestions. Glad that configuration is working out for you! Clients (aka nodes-actually-running-your-work) can be run globally distributed without issue, and we've even added some more features to help tune their behavior: https://www.hashicorp.com/blog/managing-applications-at-the-edge-with-hashicorp-nomad https://www.hashicorp.com/blog/managing-applications-at-the-... Disclaimer: I am the HashiCorp Nomad Engineering Team Lead
schmichael 4y ago
Disclaimer: I'm the HashiCorp Nomad Engineering Team Lead This may be my favorite article on the subject, and I've read a lot! Others have suggested some minor corrections but overall you've done a fantastic job accurately and fairly portraying Nomad as far as I'm concerned. I really appreciate the final paragraph as I constantly field questions about whether or not a k8s-only world is a foregone conclusion. I think there's room for more than one option, and I'm glad you noticed HashiCorp thinks so too!
- jordemort 4y ago
  Thanks, I'm glad you liked it! I've used Nomad a bit at a previous job; due to its simplicity, we were much more comfortable embedding it in the virtual appliance images we were building. Nomad meant we had to rewrite the k8s configs that the hosted version of the app was using, but even with one of the all-in-one k8s distros, we were worried about shipping that many hidden moving parts.
- jerrac 4y ago
  What options are there for managing persistent storage in Nomad? I've been through the very basic initial getting started tutorial, but haven't had a chance to dig any deeper. When I was trying kube, I found Longhorn to be the one thing I liked. Is there anything like that for Nomad? I've been trying to get a MariaDB Galera cluster running on Docker Swarm for the past couple weeks and have been somewhat stymied a few things. Something like Longhorn would help a lot.
  - schmichael 4y ago
    Nomad supports CSI like Kubernetes. Unfortunately many CSI storage providers assume kubernetes or adhere to kubernetes behavior even when it deviates from the spec, so there are gotchas. That being said they’re often glaring. So please report an issue to Nomad and the upstream of you hit an issue! Nomad has 2 other storage features: 1. Host volumes where nodes can advertise a volume by name and jobs can request being placed on nodes with that volume available. I think a lot of folks run databases this way so they can statically define their database nodes and not worry about the numerous runtime points of failure CSI introduces. 2. Ephemeral disk stickiness and migration: during deployments an instance of a job’s intrinsic local storage can be set to either get reused on the same node (sticky) or migrated to wherever the new instance is placed. Ephemeral disks is the oldest storage solution and is only a best effort (unlike CSI which may be able to reliably reattach and existing volume to a new node in the case of node failure). However I’ve heard of a lot of people happily using it for already distributed databases like Elasticsearch or Cassandra where the database can tolerate nomad’s best effort failing. (Sorry for lack of links as I’m on mobile. Googling the key phrases you care about should get you to our docs quickly and easily)
orsenthil 4y ago
> Hi, I'm the author of the article. Wow. I really enjoyed this. Your writing was clear, simple and explained the landscape while navigating the marketing terms extremely well. I wish to read other articles that you have written.
- jordemort 4y ago
  I've only written three so far, including this one; you'll find them all at LWN. Buying a subscription to LWN is the best way to ensure that they keep paying me to write more :D
  - orsenthil 4y ago
    Yup, I am a subscriber.
POPOSYS 4y ago
Thanks for your efforts, very nice article! It would be great if you took some time to write about the question of climate change and energy waste. In this regard it would be very interesting to better understand the impact tools like K8S have on global warming. Just from a pure practical POV I see many clusters burning a lot of CPU cycles with K8S processes that do nothing else than providing some kind of infrastructure for the real computing task. I have seen K8S burning 50% and more CPU time - this with not one single payload operation, just doing its cluster thing! This is a horrible waste of energy and we can not go ahead establishing such a bloated monstrosity as default data center OS of the future. Not all data centers are 100% solar - I guess people in the field are LOL at this point knowing the realities. So I personally come to the conclusion that a thing like K8S is the opposite of what is needed in a world that is heading to a climate disaster. So many hours spent into optimizing the Linux kernel (successfully!) are nullified by a tool that wastes energy recklessly. We need more than one article to help programmers of these tools understand why they should do their jobs with more love for future generations. Also we need lots of articles about how to write efficient code that does not waste CPU cycles.
POPOSYS 4y ago
Ah, and one more from another perspective: A very interesting series of articles would be "How to take down your K8S cluster" describing the many ways a cluster can be [mis]configured to easily be brought down by one single error / pod / application.

cad1 4y ago

Maybe it is implied by being on lwn.net, but the list is limited to open source orchestration platforms. In a broader discussion I would expect to see AWS ECS mentioned.

jordemort 4y ago
The implication is correct; I did not write about ECS because it is not open source.
- hkrgl 4y ago
  Out of curiosity, what other non-open source platforms do people use?
Kaytaro 4y ago
They mentioned it indirectly- > "seemingly every tech company of a certain size has its own distribution and/or hosted offering to cater to enterprises"

KronisLV 4y ago

This seems like a pretty well written overview! As someone who rather liked Docker Swarm (and still likes it, running my homelab and private cloud stuff on it), it is a bit sad to see it winding down like it, even though there were attempts to capitalize on the nice set of simple functionality that it brought to the table like CapRover: https://caprover.com/ https://caprover.com/ Even though there is still some nice software to manage installs of it, like Portainer: https://www.portainer.io/ https://www.portainer.io/ (which also works for Kubernetes, like a smaller version of Rancher) Even though the resource usage is far lower than that of almost any Kubernetes distro that I've used (microk8s, K3s and K0s included), the Compose format being pretty much amazing for most smaller deployments and Compose still being one of the better ways to run things locally in addition to Swarm for remote deployments (Skaffold or other K8s local cluster solutions just feel complex in comparison). And yet, that's probably not where the future lies. Kubernetes won. Well, Nomad is also pretty good, admittedly. Though if you absolutely do need Kubernetes, personally I'd suggest that you look in the direction of Rancher for a simple UI to manage it, or at least drill down into the cluster state, without needing too much digging through a CLI: https://rancher.com/ https://rancher.com/ Lots of folks actually like k9s as well, if you do like the TUI approach a bit more: https://k9scli.io/ https://k9scli.io/ But for the actual clusters, assuming that you ever want to self-host one, ideally a turnkey solution, RKE is good, K0s is also promising, but personally I'd go with K3s: https://k3s.io/ https://k3s.io/ which has been really stable on DEB distros and mostly works okay on RPM ones (if you cannot afford OpenShift or to wait for MicroShift). My only pet peeve being that the Traefik ingress is a little bit under-documented (e.g. how to configure common use cases, like a SSL certificate, one with an intermediate certificate, maybe a wildcard, or perhaps just use Let's Encrypt, how to set defaults vs defining them per domain). For the folks with thicker wallets, though, I'd suggest to just give in and pay someone to run a cluster for you: that way you'll get something vaguely portable, will make lots of the aspects in regards to running it someone else's problem and will be able to leverage the actual benefits of working with the container orchestrator. Oh, also Helm is actually pretty nice, since the templates it generates for you by default are actually reasonable and can serve as a nice starting point for your own application deployment deescriptions, versus trying to write your own Kubernetes manifest from 0 which feels like an uphill battle, at least when compared with the simplicity of Compose. In regards to Nomad, however, I feel like a UI that would let you click around and get bunches of HCL to describe the deployment that you want would be pretty cool. At least when starting out, something like that can really flatten the learning curve. That's also kind of why I like Portainer or Rancher so much, or even something like: https://k8syaml.com/ https://k8syaml.com/ > To extend its reach across multiple hosts, Docker introduced Swarm mode in 2016. This is actually the second product from Docker to bear the name "Swarm" — a product from 2014 implemented a completely different approach to running containers across multiple hosts, but it is no longer maintained. It was replaced by SwarmKit, which provides the underpinnings of the current version of Docker Swarm. On an unrelated note, this, at least to me, feels like pretty bad naming and management of the whole initiative, though. Of course, if the features are there, it shouldn't be enough to scare anyone away from the project, but at the same time it could have been a bit simpler.

throwaway787544 4y ago

The main purpose of K8s is just to run microservices. It makes things convenient by automatically restarting services, running multiple copies across servers, routing from one load-balancing port to many ephemerally-allocated ports, collecting logs, providing a consistent control API, limit system resources, networking, secrets... ...All things Systemd does. Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone. All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. Managed Distributions of K8s go away if every Linux distro can do the same things out of the box. "Managing fleets" just becomes running more Linux VMs or not. You wouldn't need "clusters", you'd just run a bunch of boxes and run replicas of whatever service you wanted, and all resources would be managed using normal Linux system permissions and Systemd configs. No volume mounts or persistent storage setup, it's just regular filesystem mounts and disks. Containers would be optional, eliminating a huge class of complexity (static apps make a lot of containerization moot). And most importantly to admins - no more bullshit 15 month upgrade death march. Imagine just sticking to a stable Debian for 3 years but still having all the power of a distributed orchestrator. Imagine not having 50 components with APIs listening on the internet to be hacked. Imagine just using IPTables rather than custom plugins that change based on the Cloud provider just to filter network traffic, and not letting everything on the network access everything else by default. K8s is just too expensive, complicated, and time consuming to survive. As soon as Systemd can network as a cluster, the writing's on the wall. The only reason it hasn't happened yet is the Systemd developers are basically desktop-oriented Linux fogies rather than Cloud-focused modern devs. I also predict RedHat will block patches to make this happen, since their money comes from K8s. But someone just needs to make a Systemd fork using Systemd's own API and we can finally fix all the crap Systemd devs like to pretend is somebody else's bug.

deleted 4y ago
[deleted]
loriverkutya 4y ago
I have the "If you have a SystemD everything is a nail" feeling from this comment.

bamazizi 4y ago

One thing missing from the article is the "load balancer" aspect of the orchestrators. I believe K8S favours big cloud providers tremendously due to load balancer lock-in or handcuff or whatever else you want to call it. For Kubernetes there's a WIP Metallb but I personally haven't had confident success running on my own servers. K8S setup complexity is already highlighted but I'd add upgrade and maintenance pain to the mix as well. I've only recently started with Nomad and while it's so simple, I question my self at every step because coming from K8S I've gotten used to complexity and over configuration. Nomad allows you to bring your own loadbalancer, and i'm experimenting with Traefik & Nginx now.

verdverm 4y ago
k8s has good, non-cloud based ingress controllers. https://github.com/kubernetes/ingress-nginx https://github.com/kubernetes/ingress-nginx https://doc.traefik.io/traefik/providers/kubernetes-ingress/ https://doc.traefik.io/traefik/providers/kubernetes-ingress/
- bamazizi 4y ago
  To run k8s on bare metal server clusters there's only Metallb option to bring user traffic into clusters NodePort services, via externally exposed IP address(es). I wasn't talking about k8s internal load balancers and reverse proxies
  - verdverm 4y ago
    You were talking about cloud, now you are talking about bare metal? Of course you have to route traffic to your cluster, but you are implying some cloud based load balancer lockin which just isn't true
    - chucky_z 4y ago
      I believe the OP is talking about LoadBalancer (https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/...), and I believe you're talking about Ingress (https://kubernetes.io/docs/concepts/services-networking/ingress/ https://kubernetes.io/docs/concepts/services-networking/ingr...).

hellohowareu 4y ago

I am trying to learn Kubernetes for work-- Docker + K8s + Helm. Does anyone have recommendation(s) regarding books, video courses, or live in-person trainings?

paulgb 4y ago
I learned a lot of the concepts just by typing search terms into YouTube. “TechWorld with Nana” is one channel that comes up a lot and has good, digestible explanations. Hussein Nasser too, but he has less content directly about Docker/k8s and more about networking. It's excellent though.
mmh0000 4y ago
For live training, I can not recommend Guru Labs[0] highly enough. They offer many courses, but specific to your question, lots of container related courses[1][2][3][4]. [0] https://www.gurulabs.com/linux-training/courses/ https://www.gurulabs.com/linux-training/courses/ [1] https://www.gurulabs.com/linux-training/courses/GL340/ https://www.gurulabs.com/linux-training/courses/GL340/ [2] https://www.gurulabs.com/linux-training/courses/GL360/ https://www.gurulabs.com/linux-training/courses/GL360/ [3] https://www.gurulabs.com/linux-training/courses/GL355/ https://www.gurulabs.com/linux-training/courses/GL355/ [4] https://www.gurulabs.com/linux-training/courses/GL390/ https://www.gurulabs.com/linux-training/courses/GL390/
danwee 4y ago
For K8s: 1. Read the book "Kubernetes up and Running" 2. After that you can start reading the https://kubernetes.io https://kubernetes.io documentation 3. Alongside the documentation reading, practise what you have learnt using minikube. 4. Only after, start with Helm (which is pretty bad, but if your company is using it...)
- hellohowareu 4y ago
  Thanks! I was just curious about your perspective-- When do you mean by Helm is pretty bad? Could you elaborate? Also, how does it compare with alternatives?
4y ago

Alir3z4 4y ago

Docker Swarm all the way. Started with 1 node acting as manager and worker at the same time. Then added more 2 nodes, basically 3 nodes all acting as manager and workers at the same. Later kept 3 managers and 3 workers and now around 10 workers with the 3 managers. The cluster is full of apps, celery workers, static apps or SSR. To manage it I didn't use any UI. The CLI is really simple to use. Later to give less experienced people access to it i instant Portrainer. It has never stopped working, either upgrading or messing around on production environments. It just works. The only and only complain I have id the way it mess around with IPtable rules I have. The docker swarm networking overrides them. There are several approaches, while they work, they always feel like a hack to be honest. But yeah, they work. It's such a wonderful simple to use software that gets thing.

JoyfulTurkey 4y ago
I fully agree. Docker Swarm was perfect for my small employer with an IT staff of 3 people. Went with the 3 manager nodes, as recommended by Docker, to allow for upgrades without downtime, and things were rock solid. Also liked that encryption of the overlay networks was included. The only thing we missed/wanted was auto-scaling built-in, but we rarely had a need to scale the number of containers.
bityard 4y ago
What are you doing for storage? We're thinking of deploying Docker Swarm where I work onto a VMWare cluster and basically the options boil down to sharing a storage volume between VMs (which vSphere makes difficult) and replicating storage across nodes with something like gluster (which drastically increases storage used and has performance implications).
- sofixa 4y ago
  > We're thinking of deploying Docker Swarm where I work onto a VMWare cluster IMHO that's an anti-pattern. Why would you run two orchestrators on top of one another? Containers also cause scheduling contention on the vSphere side, on top of the waste of CPU from the virtualisation layer. Unless you're in some very specific circumstances, run your container orchestrator on bare metal when on prem.

yummybear 4y ago

I do miss the days when you'd FTP to your server and sync changes.

bityard 4y ago
This is pretty much what I do with Ansible, and now my whole server configuration lives in a git repo that I can access from anywhere.

lifeisstillgood 4y ago

I was on a thread recently looking for this decades Byte magazine. How did we forget to mention lwn? Great article :-)

barberpole 4y ago

So it's not about canned orchestration?