4 ms·

okay so swarm is dead, but is kubernetes actually that good or is it just ubiquitous and you’re forced to use it today? what about nomad? or mrsk?

ghshephard 4y ago
Nomad is awesome and works at scale. The engineers continue to battle harden it and it’s a joy to work with. You do have to manage things like service discovery (usually with consul) and traffic routing separately - but the integration with vault is sublime. About the only real negative of Nomad is that it doesn’t have the mindshare that k8s does, so you don’t see the amount of developer engagement in extending it the way you do in the k8s SIGs. Also, being an expert in Nomad doesn’t give you the same number of career opportunities, and on the other side - there aren’t umpteen thousand nomad SREs the way there are with k8s - so getting someone up to speed can take a couple months (but this system is very well defined, well documented, and small enough that any half talented engineer can master it very quickly) Nomad does have the very important advantage that Hashicorp stands behind the product - so if anything goes awry, you’ve got a support team and escalation that will jump on and root cause/resolve any issue, usually within a matter of hours and even in the really squirrelly cases (that you are only likely to see when when you are managing many, many thousands of nodes in a cluster) within days.
- sofixa 4y ago
  > You do have to manage things like service discovery (usually with consul) Since the last couple of versions there has been native Service Discovery in Nomad which works pretty well.
- irusensei 4y ago
  My personal experience is that Consul and vault are too complicated to fiddle it. I can spin a kubernetes cluster in minutes but I gave up on trying a Consul+Vault+Nomad lab around all the setup steps for replication, RBAC and whatnot.
- eega 4y ago
  Less career opportunities need not be bad if they are better payed and the companies are more accommodating.
ngc248 4y ago
k8s is good, I don't get the hate against k8s.
- sofixa 4y ago
  k8s is good, but complex, and thus not fit for everyone and everything, but gets overused to death due to being the most popular choice, thus you have a lot of people that dislike it due to bad experience.
  - emptysongglass 4y ago
    There's an absolutely massive ecosystem of tooling around it to make a developer's life easier and abstract away the confusing parts. No one needs to write K8s manifests directly if they don't want to.
    - sofixa 4y ago
      The abstractions work very well, until they don't and you need to dive in 5 abstractions deep to debug what's happening, or an upgrade needs to be done and you have conflicting dependencies a few levels deep.
      - ngc248 4y ago
        But isn't this the case with everything? I feel there is a necessary level of complexity to everything. Things can't be made simpler beyond a level.
      - dontlaugh 4y ago
        Even during development I’ve had to learn far more than I’d like about k8s internals as things broke left and right. ECS, nomad or even autoscaling VMs are much easier to deal with when they fail.
- pjmlp 4y ago
  YAML spaghetti, plus a complexity that makes Java application servers look like toys.
emptysongglass 4y ago
Swarm isn't dead: Mirantis still puts engineering resources into it and has plenty of businesses relying on it. Bunch of new features just released https://www.mirantis.com/blog/announcing-the-23-0-major-release-for-mirantis-container-runtimeand-moby https://www.mirantis.com/blog/announcing-the-23-0-major-rele...

ghshephard 4y ago

Nomad is awesome and works at scale. The engineers continue to battle harden it and it’s a joy to work with. You do have to manage things like service discovery (usually with consul) and traffic routing separately - but the integration with vault is sublime. About the only real negative of Nomad is that it doesn’t have the mindshare that k8s does, so you don’t see the amount of developer engagement in extending it the way you do in the k8s SIGs. Also, being an expert in Nomad doesn’t give you the same number of career opportunities, and on the other side - there aren’t umpteen thousand nomad SREs the way there are with k8s - so getting someone up to speed can take a couple months (but this system is very well defined, well documented, and small enough that any half talented engineer can master it very quickly) Nomad does have the very important advantage that Hashicorp stands behind the product - so if anything goes awry, you’ve got a support team and escalation that will jump on and root cause/resolve any issue, usually within a matter of hours and even in the really squirrelly cases (that you are only likely to see when when you are managing many, many thousands of nodes in a cluster) within days.

sofixa 4y ago
> You do have to manage things like service discovery (usually with consul) Since the last couple of versions there has been native Service Discovery in Nomad which works pretty well.
irusensei 4y ago
My personal experience is that Consul and vault are too complicated to fiddle it. I can spin a kubernetes cluster in minutes but I gave up on trying a Consul+Vault+Nomad lab around all the setup steps for replication, RBAC and whatnot.
eega 4y ago
Less career opportunities need not be bad if they are better payed and the companies are more accommodating.

ngc248 4y ago

k8s is good, I don't get the hate against k8s.

sofixa 4y ago
k8s is good, but complex, and thus not fit for everyone and everything, but gets overused to death due to being the most popular choice, thus you have a lot of people that dislike it due to bad experience.
- emptysongglass 4y ago
  There's an absolutely massive ecosystem of tooling around it to make a developer's life easier and abstract away the confusing parts. No one needs to write K8s manifests directly if they don't want to.
  - sofixa 4y ago
    The abstractions work very well, until they don't and you need to dive in 5 abstractions deep to debug what's happening, or an upgrade needs to be done and you have conflicting dependencies a few levels deep.
    - ngc248 4y ago
      But isn't this the case with everything? I feel there is a necessary level of complexity to everything. Things can't be made simpler beyond a level.
    - dontlaugh 4y ago
      Even during development I’ve had to learn far more than I’d like about k8s internals as things broke left and right. ECS, nomad or even autoscaling VMs are much easier to deal with when they fail.
pjmlp 4y ago
YAML spaghetti, plus a complexity that makes Java application servers look like toys.

emptysongglass 4y ago

Swarm isn't dead: Mirantis still puts engineering resources into it and has plenty of businesses relying on it. Bunch of new features just released https://www.mirantis.com/blog/announcing-the-23-0-major-release-for-mirantis-container-runtimeand-moby https://www.mirantis.com/blog/announcing-the-23-0-major-rele...