7 ms·

End of an era... Are there any viable alternatives to Kubernetes and Nomad?

dragonsh 5y ago
For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. A good alternative to kubernetes is LXD [1] or just stick with docker compose. Kubernetes except for managed services from cloud providers is more difficult than an average application to manage and a huge cost in itself to run and maintain. [1] https://www.linuxcontainers.org/ https://www.linuxcontainers.org/
- marcinzm 5y ago
  >except for managed services from cloud providers I'd imagine most small to medium companies would be running things on a cloud service using managed kubernetes. It seems mostly larger companies that are sticking with non-cloud hardware and services. The advantage of kubernetes in that case is that there is a large ecosystem of helm charts, guides, documentation, etc. Deploying something new from scratch is fairly easy since someone else has done all the leg work and packaged it all up.
  - rantwasp 5y ago
    just because something is packaged does not mean it’s usable. YMMV but this is how security horror stories start. Someone ran a container they had no idea where it came from, happily used a helm chart. Most of the times it’s not even malicious - it’s outdated software because “it just works”
    - marcinzm 5y ago
      In my experience all the common helm charts and docker images are regularly updated. If you don't update your installation of them then you also wouldn't update a docker compose or LXD.
      - rantwasp 5y ago
        that’s a big statement. what constitutes “common” helm charts?
- mplewis 5y ago
  For a different perspective, learning Kubernetes and using it widely gives you a universal set of tools for managing all sorts of applications at all scales. https://news.ycombinator.com/item?id=26502900 https://news.ycombinator.com/item?id=26502900
  - ForHackernews 5y ago
    Sure, this is kind of true, but only in the most depressing way possible. Kubernetes is overengineered and terrible but it's also just about the only game in town if you want a vendor-agnostic container orchestration system. This is the same situation as Javascript circa 2008: "Learning this absolute dogshit language and using it widely will give you the ability to write universal applications that will run in browsers everywhere and make you very employable." You're not wrong about k8s today, and wouldn't have been wrong about JS in the past, but boy is it a sad indictment of our industry that these things are true.
    - marcinzm 5y ago
      Kubernetes is engineered to solve the five hundred different problems that different people have in a way that works for them. If you don't do that then everyone will complain about their feature or some edge case being missing and not use the system (see comments on mesos in this thread). That's not over engineering, that's the required amount of engineering for this sort of system.
      - ForHackernews 5y ago
        Yeah, k8s people keep telling me this. But also, k8s "secrets" are not, in fact, secret, you can't actually accept traffic from the web without some extra magic cloud load balancer (cf https://v0-3-0--metallb.netlify.app/ https://v0-3-0--metallb.netlify.app/ maybe eventually) or properly run anything stateful like a database (maybe soon). Forget covering "everyone's" use cases: From where I'm sitting, k8s is an insanely complicated system that does a miserable job of covering the 95% use cases that Heroku solved in 2008. It's great that k8s (maybe) solves hard problems that nobody except Google has, but it doesn't solve the easy problems that most people have.
- 0xEFF 5y ago
  > For over 90% of workloads kubernetes is an overkill. It's not. Take any simple web app and deploy it into managed GKE with Anthos and you automatically get SLI's and the ability to define SLO's for availability and latency with a nice wizard. Takes a few minutes to setup. The amount of engineering needed to achieve good SLO monitoring dwarfs the engineering needed to run a simple app so it just never happened. That's no longer the case. > Only when company is reaching google scale kubernetes make sense. Also obviously not true given the number of companies deriving value from kubernetes. Note, I'm a Google Cloud partner.
  - dragonsh 5y ago
    Your statement already support that without the blessings of engineering team of Amazon, Google, Microsoft, Digital Ocean and various managed kubernetes service it's impossible for a reasonable small team to manage and monitor k8s and all of this service comes with lock-in and additional capital outlay. Obviously for a Google Cloud Partner, more people are tied to gcp and kubernetes, higher the revenue. Its secondary if it's really necessary for an application to require k8s.
    - deleted 5y ago
      [deleted]
- orthecreedence 5y ago
  I don't know about this. I caution people against microservices architecture all the time, but at my company having a scheduler just made sense. We spin up and down queue workers by massive amounts every hour, and doing this with anything besides a scheduler would be really tricky. Granted, we use Nomad, not k8s, but we definitely need a scheduler and definitely are not reaching Google-scale.
  - p_l 5y ago
    It doesn't even have to be some microservices monster, just trying to bin-pack a bunch of different services, even monoliths, onto a fleet of servers. If I did it how the usual "k8s is only for FAANG scale" people tell me to do, I'd go bankrupt ;)
- lima 5y ago
  > For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. I hear people repeating this truism all day, but from practical experience, it doesn't seem to be the case - Kubernetes is paying dividends even in small single-node setups.
  - lovedswain 5y ago
    K8s single noder and GKE user here, can confirm I wouldn't remotely even consider going back. Deploying an app takes 5-10 minutes at most first time around, new pushes <1 minute, and when there is ops bullshit involved, it is never wasted work, and never needs to be repeated twice. I hated Kubernetes ops complexity at first, but there really isn't that much to it, and if it's too much, a service like GKE takes 70%+ of it away from you
- madhadron 5y ago
  > Only when company is reaching google scale kubernetes make sense. Long before Google scale. Kubernetes won't handle clusters anywhere near that large.
desktopninja 5y ago
Personally think AWS ECS is a 3rd place contender. Would add sprinkles to it if only they'd allow yaml files vs json configs in the aws-cli. ecs-cli and copilot are alright. Generally prefer to stay as close as possible to aws-cli
- kbar13 5y ago
  AWS CDK makes ECS a lot more palatable
- whatsmyusername 5y ago
  In AWS I’d choose ECS over any k8s variation on price alone before you get to the IAM integrations.
fastball 5y ago
Docker-swarm?
- mplewis 5y ago
  Docker Swarm is being cannibalized by k8s.
- yjftsjthsd-h 5y ago
  Having worked at a company running Docker swarm at... medium(?) scale... I have witnessed a truly shocking variety of bugs in its network stack. I always wondered if it was something specific to the company setup, but the result is that we just couldn't stay on a platform that would randomly fail to give containers a working network connection to each other.
ForHackernews 5y ago
Flockport? https://thenewstack.io/flockport-time-to-start-all-over-again-and-return-to-lxc-containers/ https://thenewstack.io/flockport-time-to-start-all-over-agai...

For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. A good alternative to kubernetes is LXD [1] or just stick with docker compose. Kubernetes except for managed services from cloud providers is more difficult than an average application to manage and a huge cost in itself to run and maintain. [1] https://www.linuxcontainers.org/ https://www.linuxcontainers.org/

marcinzm 5y ago
>except for managed services from cloud providers I'd imagine most small to medium companies would be running things on a cloud service using managed kubernetes. It seems mostly larger companies that are sticking with non-cloud hardware and services. The advantage of kubernetes in that case is that there is a large ecosystem of helm charts, guides, documentation, etc. Deploying something new from scratch is fairly easy since someone else has done all the leg work and packaged it all up.
- rantwasp 5y ago
  just because something is packaged does not mean it’s usable. YMMV but this is how security horror stories start. Someone ran a container they had no idea where it came from, happily used a helm chart. Most of the times it’s not even malicious - it’s outdated software because “it just works”
  - marcinzm 5y ago
    In my experience all the common helm charts and docker images are regularly updated. If you don't update your installation of them then you also wouldn't update a docker compose or LXD.
    - rantwasp 5y ago
      that’s a big statement. what constitutes “common” helm charts?
mplewis 5y ago
For a different perspective, learning Kubernetes and using it widely gives you a universal set of tools for managing all sorts of applications at all scales. https://news.ycombinator.com/item?id=26502900 https://news.ycombinator.com/item?id=26502900
- ForHackernews 5y ago
  Sure, this is kind of true, but only in the most depressing way possible. Kubernetes is overengineered and terrible but it's also just about the only game in town if you want a vendor-agnostic container orchestration system. This is the same situation as Javascript circa 2008: "Learning this absolute dogshit language and using it widely will give you the ability to write universal applications that will run in browsers everywhere and make you very employable." You're not wrong about k8s today, and wouldn't have been wrong about JS in the past, but boy is it a sad indictment of our industry that these things are true.
  - marcinzm 5y ago
    Kubernetes is engineered to solve the five hundred different problems that different people have in a way that works for them. If you don't do that then everyone will complain about their feature or some edge case being missing and not use the system (see comments on mesos in this thread). That's not over engineering, that's the required amount of engineering for this sort of system.
    - ForHackernews 5y ago
      Yeah, k8s people keep telling me this. But also, k8s "secrets" are not, in fact, secret, you can't actually accept traffic from the web without some extra magic cloud load balancer (cf https://v0-3-0--metallb.netlify.app/ https://v0-3-0--metallb.netlify.app/ maybe eventually) or properly run anything stateful like a database (maybe soon). Forget covering "everyone's" use cases: From where I'm sitting, k8s is an insanely complicated system that does a miserable job of covering the 95% use cases that Heroku solved in 2008. It's great that k8s (maybe) solves hard problems that nobody except Google has, but it doesn't solve the easy problems that most people have.
0xEFF 5y ago
> For over 90% of workloads kubernetes is an overkill. It's not. Take any simple web app and deploy it into managed GKE with Anthos and you automatically get SLI's and the ability to define SLO's for availability and latency with a nice wizard. Takes a few minutes to setup. The amount of engineering needed to achieve good SLO monitoring dwarfs the engineering needed to run a simple app so it just never happened. That's no longer the case. > Only when company is reaching google scale kubernetes make sense. Also obviously not true given the number of companies deriving value from kubernetes. Note, I'm a Google Cloud partner.
- dragonsh 5y ago
  Your statement already support that without the blessings of engineering team of Amazon, Google, Microsoft, Digital Ocean and various managed kubernetes service it's impossible for a reasonable small team to manage and monitor k8s and all of this service comes with lock-in and additional capital outlay. Obviously for a Google Cloud Partner, more people are tied to gcp and kubernetes, higher the revenue. Its secondary if it's really necessary for an application to require k8s.
  - deleted 5y ago
    [deleted]
orthecreedence 5y ago
I don't know about this. I caution people against microservices architecture all the time, but at my company having a scheduler just made sense. We spin up and down queue workers by massive amounts every hour, and doing this with anything besides a scheduler would be really tricky. Granted, we use Nomad, not k8s, but we definitely need a scheduler and definitely are not reaching Google-scale.
- p_l 5y ago
  It doesn't even have to be some microservices monster, just trying to bin-pack a bunch of different services, even monoliths, onto a fleet of servers. If I did it how the usual "k8s is only for FAANG scale" people tell me to do, I'd go bankrupt ;)
lima 5y ago
> For over 90% of workloads kubernetes is an overkill. Only when company is reaching google scale kubernetes make sense. I hear people repeating this truism all day, but from practical experience, it doesn't seem to be the case - Kubernetes is paying dividends even in small single-node setups.
- lovedswain 5y ago
  K8s single noder and GKE user here, can confirm I wouldn't remotely even consider going back. Deploying an app takes 5-10 minutes at most first time around, new pushes <1 minute, and when there is ops bullshit involved, it is never wasted work, and never needs to be repeated twice. I hated Kubernetes ops complexity at first, but there really isn't that much to it, and if it's too much, a service like GKE takes 70%+ of it away from you
madhadron 5y ago
> Only when company is reaching google scale kubernetes make sense. Long before Google scale. Kubernetes won't handle clusters anywhere near that large.

desktopninja 5y ago

Personally think AWS ECS is a 3rd place contender. Would add sprinkles to it if only they'd allow yaml files vs json configs in the aws-cli. ecs-cli and copilot are alright. Generally prefer to stay as close as possible to aws-cli

kbar13 5y ago
AWS CDK makes ECS a lot more palatable
whatsmyusername 5y ago
In AWS I’d choose ECS over any k8s variation on price alone before you get to the IAM integrations.

fastball 5y ago

Docker-swarm?

mplewis 5y ago
Docker Swarm is being cannibalized by k8s.
yjftsjthsd-h 5y ago
Having worked at a company running Docker swarm at... medium(?) scale... I have witnessed a truly shocking variety of bugs in its network stack. I always wondered if it was something specific to the company setup, but the result is that we just couldn't stay on a platform that would randomly fail to give containers a working network connection to each other.

ForHackernews 5y ago

Flockport? https://thenewstack.io/flockport-time-to-start-all-over-again-and-return-to-lxc-containers/ https://thenewstack.io/flockport-time-to-start-all-over-agai...