5 ms·

I've had a few questions about Kubernetes and production grade environments and container orchestration. 1. Why Kubernetes? (when mesosphere/dcos is available?

I've had a few questions about Kubernetes and production grade environments and container orchestration. 1. Why Kubernetes? (when mesosphere/dcos is available?) 2. Any available best practices when setting the entire environment from scratch? (right from server level security, to doing data backups for example) --- As a critic, I thought that if you were running it in production, you'd need 3 masters? (to ensure that if the master goes down, another takes over?) So with 3 masters, and 2 slaves...the production cost go up to about $25/m. I feel that's too expensive for what you do get (price vs performance) compared to others (I personally feel a production environment running 5 servers from packet.net @ ~180$/m is the way to go [0]) [0] - https://www.packet.net/bare-metal/servers/type-0/ https://www.packet.net/bare-metal/servers/type-0/

vittore 10y ago
Why DCOS if swarm is available? Not trolling, just want to put my head around it too.
- ohstopitu 10y ago
  From what I understand, Swarm is not yet in it's prime state and seems to have a lot of issues that are not around. DCOS is based of mesos which is supposedly battle tested (I know it's the same for Kubernetes) Also, at least in mesos, you can set up time scheduled tasks which I don't think is possible with Swarm.
avitzurel 10y ago
I commented about DCOS vs Kubernetes many times before here on HN. I had a production cluster running on mesos/marathon (mesosphere). Also have an open source project heavily invested in it [1]. Recently, I refactored the entire environment to work on Kubernets and the open source project is following it as well (It is basically a miror of my production ENV). After this long story I will say this. They are both hard to set up correctly, especially if you follow best-practices of closed networks and publicly accessed resources (VPN, Internet Gateway etc...). Kubernetes has a lot of "force" around it right now with the community booming. Posts like this (with it's many problems), is somewhat of a proof becuase it shows a vibrant community around a project. Kube is more stable from my experience. Like you mentioned, I would not run Kube production on anything less than 200-250$ which means multiple masters and multiple slaves for the pod runners. It's nice to run something cheap to check the capabilities but not more than that. Regarding DO, I'm a big fan and run a lot of projects (personal) with them, for production grade stacks I would use AWS/Google/Azure, those give you better capabilities around hardening a cluster (security, networking) for production workloads. [1] http://docs.the-startup-stack.com http://docs.the-startup-stack.com
- sandGorgon 10y ago
  this is very interesting! can you talk about the problems you found. Im not sure what you mean by pod runners.. are you referring to kubelet here? we are considering a k8s based stack.. but have seen very few on-metal deployments with k8s.
old-gregg 10y ago
I'll bite: > Why Kubernetes? (when mesosphere/dcos is available?) Frankly, if you're familiar and comfortable with DCOS, the core financial benefit is the same: increased hardware utilization, assuming your workload can realize that benefit. And if your application is relatively monolithic and handles the same steady load, I'd advise to stay away from advanced orchestration tools because they're only add moving parts and more complexity, unless you have [1] > Any available best practices when setting the entire environment from scratch? For education, I would recommend Kelsey's excellent "Kubernetes the Hard Way": https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way For production, I would recommend Telekube: http://gravitational.com/managed-kubernetes/ http://gravitational.com/managed-kubernetes/ It does a lot of extra heavy lifting to make sure your k8s feels like always-on black box (I work there). [1] Kubernetes also has a "side" benefit of abstracting the underlying infrastructure away, so if you have a use case of running the same complex stack on AWS, colo, vmware, etc then running it on top of a "cloud abstraction layer" lowers your ops/dev costs significantly. We talk about this here: http://gravitational.com/telekube/ http://gravitational.com/telekube/
- ohstopitu 10y ago
  Thanks for the links, I had to setup a Mesos cluster on ~200 machines for my university (which is what got me interested in DCOS in the first place). I was then trying to replicate the same in the cloud but with more constraints (security, networking etc.) and found both Kubernetes and DCOS complicated. Recently Azure seems to have DCOS (and Kubernetes) as it's orchestration tools for Azure Container Service and I wanted to learn more about each.
- gkop 10y ago
  If you're already on Google Cloud, and wanting to deploy a containerized app, then the side benefit alone is probably worthwhile because Kubernetes takes care of dealing with both Google Cloud and Docker, with little or no downside.
tracker1 10y ago
You could always get started with something simpler like dokku (don't run on the $5 DO plan, and make sure to setup a swap file). In the end, it is pretty cool. Though, I'd just assume Docker's (the company) administration tools for DO become even better, as I'd love to "self-host" with Docker mgt tooling, and deploy my apps/services. Easier setup and flow with Let's Encrypt is also essential. In the end, I think it's interesting, but overkill for a lot of scenarios... In a prior role, we went with 3 instances of dokku, with everything (but DB) deployed to all three instances, and an LB in front... was easier than all the orchestration tooling at the time.
discordianfish 10y ago
1) Simply put: Because I believe it's the best option both right now and from it's prospect. You might look into mesos(phere) if you want to run other workloads than containers for which support on mesos is more mature 2) This setup is done from scratch, so given all the constraints this is what I would do. Then again, I'd probably care little about saving a few $$ by using DigitalOcean and look into deploying it on AWS or GCP which is more mature you don't need to use terraform which could be more reliable.. The cluster I describe has 3 nodes, all running controller and worker services. But again: Please don't use this setup 1:1 for production. If you really just want to run a blog and stuff, there are better options.
eikenberry 10y ago
One thing I hear a lot when people talk about Kubernetes vs Mesos is the developer experience. Such that a developer can run a kubernetes cluster on their laptop to work with "easily" vs it being essentially impossible with Mesos.

vittore 10y ago

Why DCOS if swarm is available? Not trolling, just want to put my head around it too.

ohstopitu 10y ago
From what I understand, Swarm is not yet in it's prime state and seems to have a lot of issues that are not around. DCOS is based of mesos which is supposedly battle tested (I know it's the same for Kubernetes) Also, at least in mesos, you can set up time scheduled tasks which I don't think is possible with Swarm.

avitzurel 10y ago

I commented about DCOS vs Kubernetes many times before here on HN. I had a production cluster running on mesos/marathon (mesosphere). Also have an open source project heavily invested in it [1]. Recently, I refactored the entire environment to work on Kubernets and the open source project is following it as well (It is basically a miror of my production ENV). After this long story I will say this. They are both hard to set up correctly, especially if you follow best-practices of closed networks and publicly accessed resources (VPN, Internet Gateway etc...). Kubernetes has a lot of "force" around it right now with the community booming. Posts like this (with it's many problems), is somewhat of a proof becuase it shows a vibrant community around a project. Kube is more stable from my experience. Like you mentioned, I would not run Kube production on anything less than 200-250$ which means multiple masters and multiple slaves for the pod runners. It's nice to run something cheap to check the capabilities but not more than that. Regarding DO, I'm a big fan and run a lot of projects (personal) with them, for production grade stacks I would use AWS/Google/Azure, those give you better capabilities around hardening a cluster (security, networking) for production workloads. [1] http://docs.the-startup-stack.com http://docs.the-startup-stack.com

sandGorgon 10y ago
this is very interesting! can you talk about the problems you found. Im not sure what you mean by pod runners.. are you referring to kubelet here? we are considering a k8s based stack.. but have seen very few on-metal deployments with k8s.

old-gregg 10y ago

I'll bite: > Why Kubernetes? (when mesosphere/dcos is available?) Frankly, if you're familiar and comfortable with DCOS, the core financial benefit is the same: increased hardware utilization, assuming your workload can realize that benefit. And if your application is relatively monolithic and handles the same steady load, I'd advise to stay away from advanced orchestration tools because they're only add moving parts and more complexity, unless you have [1] > Any available best practices when setting the entire environment from scratch? For education, I would recommend Kelsey's excellent "Kubernetes the Hard Way": https://github.com/kelseyhightower/kubernetes-the-hard-way https://github.com/kelseyhightower/kubernetes-the-hard-way For production, I would recommend Telekube: http://gravitational.com/managed-kubernetes/ http://gravitational.com/managed-kubernetes/ It does a lot of extra heavy lifting to make sure your k8s feels like always-on black box (I work there). [1] Kubernetes also has a "side" benefit of abstracting the underlying infrastructure away, so if you have a use case of running the same complex stack on AWS, colo, vmware, etc then running it on top of a "cloud abstraction layer" lowers your ops/dev costs significantly. We talk about this here: http://gravitational.com/telekube/ http://gravitational.com/telekube/

ohstopitu 10y ago
Thanks for the links, I had to setup a Mesos cluster on ~200 machines for my university (which is what got me interested in DCOS in the first place). I was then trying to replicate the same in the cloud but with more constraints (security, networking etc.) and found both Kubernetes and DCOS complicated. Recently Azure seems to have DCOS (and Kubernetes) as it's orchestration tools for Azure Container Service and I wanted to learn more about each.
gkop 10y ago
If you're already on Google Cloud, and wanting to deploy a containerized app, then the side benefit alone is probably worthwhile because Kubernetes takes care of dealing with both Google Cloud and Docker, with little or no downside.

tracker1 10y ago

You could always get started with something simpler like dokku (don't run on the $5 DO plan, and make sure to setup a swap file). In the end, it is pretty cool. Though, I'd just assume Docker's (the company) administration tools for DO become even better, as I'd love to "self-host" with Docker mgt tooling, and deploy my apps/services. Easier setup and flow with Let's Encrypt is also essential. In the end, I think it's interesting, but overkill for a lot of scenarios... In a prior role, we went with 3 instances of dokku, with everything (but DB) deployed to all three instances, and an LB in front... was easier than all the orchestration tooling at the time.

discordianfish 10y ago

1) Simply put: Because I believe it's the best option both right now and from it's prospect. You might look into mesos(phere) if you want to run other workloads than containers for which support on mesos is more mature 2) This setup is done from scratch, so given all the constraints this is what I would do. Then again, I'd probably care little about saving a few $$ by using DigitalOcean and look into deploying it on AWS or GCP which is more mature you don't need to use terraform which could be more reliable.. The cluster I describe has 3 nodes, all running controller and worker services. But again: Please don't use this setup 1:1 for production. If you really just want to run a blog and stuff, there are better options.

eikenberry 10y ago

One thing I hear a lot when people talk about Kubernetes vs Mesos is the developer experience. Such that a developer can run a kubernetes cluster on their laptop to work with "easily" vs it being essentially impossible with Mesos.