5 ms·

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad do

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s. However, when it first came out all the cool kids were using it, then most of them jumped ship to k8s. AirBnB's Chronos is now pretty much a dead project, Mesosphere's Marathon is now gimped (no UI) and major features moved into DCOS. At the same time, Mesosphere (now D2iQ?), now seems like it's more focused on k8s. k8s is everything plus the kitchen sink, and managed k8s isn't the killer feature I thought it would be. I don't blame people for not jumping on the k8s bandwagon at all.

rtempaccount1 6y ago
I'd agree that k8s has a lot of functionality built-in, another important thing to realise is what k8s doesn't do. In addition to the well-known integration points (Container Runtime/Network/Storage Interfaces), there's things like the lack of a good built-in user authentication mechanism with Kubernetes, which means you pretty much always need some external authentication service for your clusters. That's not too bad if your on one of the big managed providers (GKE/AKS/EKS) but can get complex for people who want to deploy on-prem.
- snuxoll 6y ago
  > That's not too bad if your on one of the big managed providers (GKE/AKS/EKS) but can get complex for people who want to deploy on-prem. Go spin up Keycloak, join it to your user-directory of choice (or not and just use the internal directory), configure it as your authentication provider, done.
  - threeseed 6y ago
    Or use Dex: https://github.com/dexidp/dex https://github.com/dexidp/dex Which has the advantage of not needing any external databases.
    - snuxoll 6y ago
      That works too depending on your requirements. Either way, authentication is not a hard problem to solve.
      - kissgyorgy 6y ago
        LOL. You clearly not worked with SSO or anything a bit more complex. It's a pretty hard problem, there are even companies whose whole portfolio is around authentication only!
  - raesene9 6y ago
    Right so in addition to the complexity of running k8s (which is the general point of the post) you now have to learn about OAuth servers and LDAP integration. In many corporates you also now have the challenges of cross-team/department work, for the k8s team to work with the AD team to get it setup. And still that won't get you away from the problem that without a first class user / group object in k8s people often end up running into problems with JML processes over time and mismatch between AuthN and AuthZ...
mpfundstein 6y ago
Nomad, Consul + Swarm is still a lovely solution and I prefer it a lot to K8s. K8s is a big monolith and often way too complex for my personal use cases. I hope Hashicorp sooner or later builds a proper replacement for Swarm so that we can have overlay networks without hassle. I know there is Weave, but never tried it.
- kitotik 6y ago
  Which part of Swarm are you looking to replace? Nomad already does the container part, and Consul Connect is the networking overlay / service mesh. Work is being done to get Nomad better integrated with Connect.
- tie_ 6y ago
  K8s is a lot of things, but a monolith it is not. Quite the opposite - the complexity comes from the large number of relatively simple components interacting in various ways.
- schmichael 6y ago
  Broader CNI and multi-interface networking support coming in 0.12! Would love to hear what else you miss from swarm: schmichael@hashicorp.com Update: Honestly if you haven't tried out our Consul Connect integration, please do. mTLS with just "connect { sidecar_proxy {} }" stanza and Consul running: https://www.nomadproject.io/docs/integrations/consul-connect/ https://www.nomadproject.io/docs/integrations/consul-connect...
abhorrence 6y ago
Chronos wasn't killed by k8s -- it was killed by Airflow.
corford 6y ago
FWIW, where I work we're going with Nomad over K8S. It gives us everything we want and nothing more (plus it's from a source we trust and love: HashiCorp). The nice thing with Nomad is, thanks to it's straightforward design/approach, it should be easy(ish) in the future to swap it out and go with something else if we outgrow it (or HashiCorp abandon it for some reason).
djsumdog 6y ago
I know quite a few startups going the Nomad route. It might not be as mainstream, but I don't think it's going to be phased out any time soon. Rancher was still maintaining their own scheduler for a while (not sure if they still do) because there were a lot of legacy customers still on it. Racher and RancherOS has pretty much moved to a full k8s management shop though.
gen220 6y ago
We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it all alive, and they’d be able to maintain the pieces of it that we use. Aurora configuration kinda sucks, but I think job configurations might just suck in general. That being said, it has been concerning to us to see the big institutional players move off of mesos and aurora. We wouldn’t like to be active maintainers, but we could. I think that’s the main difference between Mesos and K8s: with K8s, we wouldn’t want to maintain it, and we wouldn’t be able to (since it’s so large). Somehow mesos and aurora feel more manageable.
- harrisonjackson 6y ago
  5 dedicated folks = a lot of dollars. $500k-1M? Do those 5 folks keep things up reliably without burning out? Honestly curious - never used k8s or Mesos* in production and we have a much smaller team managing a much smaller infrastructure, I am sure. edit: meant to say mesos but applies to both
  - gen220 6y ago
    Yea 500k-1m is probably correct. But, to be fair, they’re not dedicated to only keeping the lights on, they probably spend 20% (1 eng year per year) of their collective hours fighting fires, so I don’t think burnout is too much of an issue. The remainder of their time is spent maintaining libraries and web interfaces. They’re a pretty standard “platform engineering” team. In my perspective, it doesn’t matter what technology you go with (K8s or otherwise): a responsibly-managed your platform team requires at least one team with a full on-call rotation (i.e. at least 4 engineers), depending on how wide your golden path is.
- ridv 6y ago
  I'm a former Apache Aurora maintainer. Aurora has been (and continues to be) awesome for us and I'm so happy to hear other folks are still using it and it's working out for them. Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as the remedy so I think you're right on the money with your statement. When the Project Management Committee (PMC) voted to put Aurora in the attic we were all super bummed but we just ran out of interested developers :(. The PMC agreed to kick off an "official" fork but so far it's just me maintaining it: https://github.com/aurora-scheduler/aurora https://github.com/aurora-scheduler/aurora
michaeldwan 6y ago
We're using Nomad to manage a large fleet of firecracker vms at fly.io. It's not as robust as k8s, but I think that's a feature. It's well documented, extensible, and predictable. Not a big community, but hashicorp folks are responsive on GitHub.
- schmichael 6y ago
  We're working on community! There's an office hours tomorrow: https://mobile.twitter.com/HashiCorp/status/1270126346103132170 https://mobile.twitter.com/HashiCorp/status/1270126346103132... There's also https://discuss.hashicorp.com/ https://discuss.hashicorp.com/ and we're discussing (harhar) how else to improve our community relations.
  - michaeldwan 6y ago
    That's great to hear!
whatsmyusername 6y ago
Swarm being dead is news to me, we use it for our on-prem bits without issue (though we only have a few of them). We use ECS for the majority of our container orchestration and love it.
random3 6y ago
Mesos was great. It was a much cleaner, flexible separation of concerns. Too bad Mesosphere killed it.
p_l 6y ago
I think it's more that Mesos committed suicide by DCOS - which sapped the strength from the community, something that first Google, then CNF work hard at avoiding. Another thing that might have been just me, but I really couldn't see any structure in Mesos deployments. Yes, you could run X, Y, Z on top of it to get various features, but they all had separate APIs, separate input files, etc. Having used k8s before that it was a huge blow, since at the time (late 2018) I was used to think that Mesos et al were more mature and advanced, and instead encountered k8s circa 1.2 with less cohesion :/
jatins 6y ago
I tried using Nomad once after being a little worn out by Kubernetes' complexity. For some reason, the Nomad abstractions didn't click on the first couple attempts. In comparison, Kubernetes' abstractions mapped 1:1 to my understanding of the service oriented architecture. I'd have probably gotten used to it had I spent more time using it, but it'd have taken some rewiring of thinking process.
- lsofzz 6y ago
  > I tried using Nomad once after being a little worn out by Kubernetes' complexity. For some reason, the Nomad abstractions didn't click on the first couple attempts. In comparison, Kubernetes' abstractions mapped 1:1 to my understanding of the service oriented architecture. <3 Nomad. However, Nomad only satisfies a really tiny part of Kubernetes ecosystem which is the ability to pack containers and schedule them efficiently in a cluster; Plus, it scales really well. Kubernetes provides a bit more than that.

rtempaccount1 6y ago

I'd agree that k8s has a lot of functionality built-in, another important thing to realise is what k8s doesn't do. In addition to the well-known integration points (Container Runtime/Network/Storage Interfaces), there's things like the lack of a good built-in user authentication mechanism with Kubernetes, which means you pretty much always need some external authentication service for your clusters. That's not too bad if your on one of the big managed providers (GKE/AKS/EKS) but can get complex for people who want to deploy on-prem.

snuxoll 6y ago
> That's not too bad if your on one of the big managed providers (GKE/AKS/EKS) but can get complex for people who want to deploy on-prem. Go spin up Keycloak, join it to your user-directory of choice (or not and just use the internal directory), configure it as your authentication provider, done.
- threeseed 6y ago
  Or use Dex: https://github.com/dexidp/dex https://github.com/dexidp/dex Which has the advantage of not needing any external databases.
  - snuxoll 6y ago
    That works too depending on your requirements. Either way, authentication is not a hard problem to solve.
    - kissgyorgy 6y ago
      LOL. You clearly not worked with SSO or anything a bit more complex. It's a pretty hard problem, there are even companies whose whole portfolio is around authentication only!
- raesene9 6y ago
  Right so in addition to the complexity of running k8s (which is the general point of the post) you now have to learn about OAuth servers and LDAP integration. In many corporates you also now have the challenges of cross-team/department work, for the k8s team to work with the AD team to get it setup. And still that won't get you away from the problem that without a first class user / group object in k8s people often end up running into problems with JML processes over time and mismatch between AuthN and AuthZ...

mpfundstein 6y ago

Nomad, Consul + Swarm is still a lovely solution and I prefer it a lot to K8s. K8s is a big monolith and often way too complex for my personal use cases. I hope Hashicorp sooner or later builds a proper replacement for Swarm so that we can have overlay networks without hassle. I know there is Weave, but never tried it.

kitotik 6y ago
Which part of Swarm are you looking to replace? Nomad already does the container part, and Consul Connect is the networking overlay / service mesh. Work is being done to get Nomad better integrated with Connect.
tie_ 6y ago
K8s is a lot of things, but a monolith it is not. Quite the opposite - the complexity comes from the large number of relatively simple components interacting in various ways.
schmichael 6y ago
Broader CNI and multi-interface networking support coming in 0.12! Would love to hear what else you miss from swarm: schmichael@hashicorp.com Update: Honestly if you haven't tried out our Consul Connect integration, please do. mTLS with just "connect { sidecar_proxy {} }" stanza and Consul running: https://www.nomadproject.io/docs/integrations/consul-connect/ https://www.nomadproject.io/docs/integrations/consul-connect...

abhorrence 6y ago

Chronos wasn't killed by k8s -- it was killed by Airflow.

corford 6y ago

FWIW, where I work we're going with Nomad over K8S. It gives us everything we want and nothing more (plus it's from a source we trust and love: HashiCorp). The nice thing with Nomad is, thanks to it's straightforward design/approach, it should be easy(ish) in the future to swap it out and go with something else if we outgrow it (or HashiCorp abandon it for some reason).

djsumdog 6y ago

I know quite a few startups going the Nomad route. It might not be as mainstream, but I don't think it's going to be phased out any time soon. Rancher was still maintaining their own scheduler for a while (not sure if they still do) because there were a lot of legacy customers still on it. Racher and RancherOS has pretty much moved to a full k8s management shop though.

gen220 6y ago

We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it all alive, and they’d be able to maintain the pieces of it that we use. Aurora configuration kinda sucks, but I think job configurations might just suck in general. That being said, it has been concerning to us to see the big institutional players move off of mesos and aurora. We wouldn’t like to be active maintainers, but we could. I think that’s the main difference between Mesos and K8s: with K8s, we wouldn’t want to maintain it, and we wouldn’t be able to (since it’s so large). Somehow mesos and aurora feel more manageable.

harrisonjackson 6y ago
5 dedicated folks = a lot of dollars. $500k-1M? Do those 5 folks keep things up reliably without burning out? Honestly curious - never used k8s or Mesos* in production and we have a much smaller team managing a much smaller infrastructure, I am sure. edit: meant to say mesos but applies to both
- gen220 6y ago
  Yea 500k-1m is probably correct. But, to be fair, they’re not dedicated to only keeping the lights on, they probably spend 20% (1 eng year per year) of their collective hours fighting fires, so I don’t think burnout is too much of an issue. The remainder of their time is spent maintaining libraries and web interfaces. They’re a pretty standard “platform engineering” team. In my perspective, it doesn’t matter what technology you go with (K8s or otherwise): a responsibly-managed your platform team requires at least one team with a full on-call rotation (i.e. at least 4 engineers), depending on how wide your golden path is.
ridv 6y ago
I'm a former Apache Aurora maintainer. Aurora has been (and continues to be) awesome for us and I'm so happy to hear other folks are still using it and it's working out for them. Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as the remedy so I think you're right on the money with your statement. When the Project Management Committee (PMC) voted to put Aurora in the attic we were all super bummed but we just ran out of interested developers :(. The PMC agreed to kick off an "official" fork but so far it's just me maintaining it: https://github.com/aurora-scheduler/aurora https://github.com/aurora-scheduler/aurora

michaeldwan 6y ago

We're using Nomad to manage a large fleet of firecracker vms at fly.io. It's not as robust as k8s, but I think that's a feature. It's well documented, extensible, and predictable. Not a big community, but hashicorp folks are responsive on GitHub.

schmichael 6y ago
We're working on community! There's an office hours tomorrow: https://mobile.twitter.com/HashiCorp/status/1270126346103132170 https://mobile.twitter.com/HashiCorp/status/1270126346103132... There's also https://discuss.hashicorp.com/ https://discuss.hashicorp.com/ and we're discussing (harhar) how else to improve our community relations.
- michaeldwan 6y ago
  That's great to hear!

whatsmyusername 6y ago

Swarm being dead is news to me, we use it for our on-prem bits without issue (though we only have a few of them). We use ECS for the majority of our container orchestration and love it.

random3 6y ago

Mesos was great. It was a much cleaner, flexible separation of concerns. Too bad Mesosphere killed it.

p_l 6y ago

I think it's more that Mesos committed suicide by DCOS - which sapped the strength from the community, something that first Google, then CNF work hard at avoiding. Another thing that might have been just me, but I really couldn't see any structure in Mesos deployments. Yes, you could run X, Y, Z on top of it to get various features, but they all had separate APIs, separate input files, etc. Having used k8s before that it was a huge blow, since at the time (late 2018) I was used to think that Mesos et al were more mature and advanced, and instead encountered k8s circa 1.2 with less cohesion :/

jatins 6y ago

I tried using Nomad once after being a little worn out by Kubernetes' complexity. For some reason, the Nomad abstractions didn't click on the first couple attempts. In comparison, Kubernetes' abstractions mapped 1:1 to my understanding of the service oriented architecture. I'd have probably gotten used to it had I spent more time using it, but it'd have taken some rewiring of thinking process.

lsofzz 6y ago
> I tried using Nomad once after being a little worn out by Kubernetes' complexity. For some reason, the Nomad abstractions didn't click on the first couple attempts. In comparison, Kubernetes' abstractions mapped 1:1 to my understanding of the service oriented architecture. <3 Nomad. However, Nomad only satisfies a really tiny part of Kubernetes ecosystem which is the ability to pack containers and schedule them efficiently in a cluster; Plus, it scales really well. Kubernetes provides a bit more than that.