3 ms·

Ask HN: Kubernetes vs. Nomad vs. Mesos vs ...

9 points by vamitrou 10y ago

What do you guys use? What are the pros and cons regarding features, stability and production readiness?

user5994461 10y ago
I assume the topic is Docker orchestration to automatically start/stop/restart/kill/roll upgrade/canary/bluegreen the servers. If you are on AWS, use auto scaling groups. It achieves more than these software, it works and it's stable. (Bonus: you don't need docker at all). If you are on GCE, use Google Container Engine (i.e. their kubernetes). That's the only [sane] way to have kubernetes running in production. I'm planning a blog post about all that later.
- vamitrou 10y ago
  I don't get why everybody talks about "cloud" and AWS/CGE, it seems you guys ignore the fact that many companies still maintain their own infrastructure or they lease them from other companies smaller than Amazon or Google!
  - user5994461 10y ago
    Because we need it of course! Either we got GCE/AWS or we gotta recreate it on bare metal with VmWare :D If you're a [very] small company with limited resources (and limited needs?) to the point of living on a cheap featureless hosting provider, you should forget about Kubernetes and Mesos. They're hell to setup, don't waste your time with that. What's the scale of your operations? How many servers? How much budget for hardware? How many applications? How many devs? How many ops? Maybe open another question, with more details. More people will catch up if you're lucky.
- usgroup 10y ago
  That's interesting. Would you mind saying more about your kubernetes experience? I have a side-line scepticism about Kubernetes because it feels over-engineered but otherwise a container orchestration fan.
schmidtc 10y ago
I run a small 4 node mesos cluster. So far it's been great, took a day or two to setup and has been running for about a year now with little to no maintenance. (I should probably update). I launch long running services through Marathon and crons with Chronos. I use haproxy for service discovery and load balancing. It's mostly used for data processing, but does serve data to a production web environment (~40k req / day). The things I like the most about mesos is that it is light weight and removes a like of the friction in deploying new services. So I can test out a new idea without a big investment or f*ing around with the giant monolith. I found container based approaches too resource intensive for my budget.
lnalx 10y ago
I use Docker in production with Docker Compose for containers orchestration. Each service has its own Git repo with a compose file describing the orchestration. Swarm for distributed services and managing multiple servers. Everything is a container and could be scaled, destroyed, restarted in a very simple way. Machine creation is automated with Terraform and Ansible for provisioning, no matter the provider.
- alauda 10y ago
  Had you tried hyper.sh? All you need is the compose file, no more terraform and ansible.
- usgroup 10y ago
  Out of interest, how are you handling persistence in your stack? Are the cluster machines homogeneous (i.e. all the same)? Can containers be deployed to any machine or must they be deployed to where the data was left?
nivertech 10y ago
This link might help you to decide which container orchestrator/scheduler to use: https://twitter.com/nivertech/status/777556214767124481 https://twitter.com/nivertech/status/777556214767124481

9 points by vamitrou 10y ago

What do you guys use? What are the pros and cons regarding features, stability and production readiness?

I assume the topic is Docker orchestration to automatically start/stop/restart/kill/roll upgrade/canary/bluegreen the servers. If you are on AWS, use auto scaling groups. It achieves more than these software, it works and it's stable. (Bonus: you don't need docker at all). If you are on GCE, use Google Container Engine (i.e. their kubernetes). That's the only [sane] way to have kubernetes running in production. I'm planning a blog post about all that later.

vamitrou 10y ago
I don't get why everybody talks about "cloud" and AWS/CGE, it seems you guys ignore the fact that many companies still maintain their own infrastructure or they lease them from other companies smaller than Amazon or Google!
- user5994461 10y ago
  Because we need it of course! Either we got GCE/AWS or we gotta recreate it on bare metal with VmWare :D If you're a [very] small company with limited resources (and limited needs?) to the point of living on a cheap featureless hosting provider, you should forget about Kubernetes and Mesos. They're hell to setup, don't waste your time with that. What's the scale of your operations? How many servers? How much budget for hardware? How many applications? How many devs? How many ops? Maybe open another question, with more details. More people will catch up if you're lucky.
usgroup 10y ago
That's interesting. Would you mind saying more about your kubernetes experience? I have a side-line scepticism about Kubernetes because it feels over-engineered but otherwise a container orchestration fan.

schmidtc 10y ago

I run a small 4 node mesos cluster. So far it's been great, took a day or two to setup and has been running for about a year now with little to no maintenance. (I should probably update). I launch long running services through Marathon and crons with Chronos. I use haproxy for service discovery and load balancing. It's mostly used for data processing, but does serve data to a production web environment (~40k req / day). The things I like the most about mesos is that it is light weight and removes a like of the friction in deploying new services. So I can test out a new idea without a big investment or f*ing around with the giant monolith. I found container based approaches too resource intensive for my budget.

lnalx 10y ago

I use Docker in production with Docker Compose for containers orchestration. Each service has its own Git repo with a compose file describing the orchestration. Swarm for distributed services and managing multiple servers. Everything is a container and could be scaled, destroyed, restarted in a very simple way. Machine creation is automated with Terraform and Ansible for provisioning, no matter the provider.

alauda 10y ago
Had you tried hyper.sh? All you need is the compose file, no more terraform and ansible.
usgroup 10y ago
Out of interest, how are you handling persistence in your stack? Are the cluster machines homogeneous (i.e. all the same)? Can containers be deployed to any machine or must they be deployed to where the data was left?

nivertech 10y ago

This link might help you to decide which container orchestrator/scheduler to use: https://twitter.com/nivertech/status/777556214767124481 https://twitter.com/nivertech/status/777556214767124481