3 ms·

> Kubernetes, Mesos, etc. appear to be great orchestration tools for an organization with a few [or many] engineers dedicated to operations. Their not so great

> Kubernetes, Mesos, etc. appear to be great orchestration tools for an organization with a few [or many] engineers dedicated to operations. Their not so great for a small team [or individuals] who are just trying to deploy some software. In my opinion, the opposite is true about Kubernetes (though it may be true about the Mesos stack). It's a great way to reduce the need for dedicated ops engineers. Once you have a Kubernetes cluster running, it's essentially a "platform as a service" for developers. They can deploy new pods and configure resources all with a single client. You generally no longer need to think about the servers themselves. You only need to think in terms of containers. No package management, no configuration management (in the Salt/Puppet/Ansible/Chef sense), no Unix shell access needed. It's hugely liberating to treat your cluster as a bunch of containers on top of a virtualized scheduling substrate. Kubernetes itself is very lightweight, and requires minimal administration. (If you're on GKE, K8s is managed for you.) Swarm Engine is very much knocking off Kubernetes' features. Swarm offers the convenience of building some orchestration primitives into Docker itself, whereas Kubernetes is layered on top of Docker. Other than that, they're trying to solve the same thing. I'd say Kubernetes' design is superior, however.

brudgers 10y ago
I don't disagree. I think the differences lie in reducing versus eliminating a requirement for operations engineers and the level at which abstractions are surfaced. To me, Swarm looks headed toward abstractions at a higher level than Kubernetes much like Docker surfaces higher level abstractions than LXC. Maybe that's the difference between a product and a solution. I don't know.
- lobster_johnson 10y ago
  > To me, Swarm looks headed toward abstractions at a higher level than Kubernetes Kubernetes admittedly takes a slightly different, modular, layered approach, whereas Swarm is simple to the point of naivety. This simplicity is potentially a threat to its future ability to adapt to different requirements, whereas Kubernetes offers a separation of primitives that allows it to scale from "simple" to "complex". For example, in Kubernetes, a deployment is a separate object from the pods themselves. You create a deployment, which in turn creates a replica set, which in turn creates or tears down pods as needed. But you don't really need to work directly with replica sets or know exactly how they work, but they're there, and can be used outside of a deployment. If all you care about is deploying apps, then you only need to deal with the deployment.
  - hosh 10y ago
    And, there is enough documentation where you can: (1) Write your own custom deployer https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri (2) Write your own controllers https://deis.com/blog/2016/scheduling-your-kubernetes-pods-with-elixir/ https://deis.com/blog/2016/scheduling-your-kubernetes-pods-w... Those two are just examples. My point is that if you need to dig in, you can. You can do this because those higher-level behaviors are built on a solid foundation of well-thought-out primitives. What I expect to happen is that we will have a diverse set of things built on top of Kubernetes. Some of it will get folded into Kubernetes core. Some will not. I think the center of gravity for containers has been shifting towards Kubernetes.
  - brudgers 10y ago
    Swarm is simple to the point of naivety. Exactly. To me, that approach is what allowed Docker to make containers on the laptop ubiquitous. Kubernetes is unlikely to be the tool that takes schedulers to that universe. Swarm might and I think that's the goal. I mean, I don't really want to care about pods and replica sets. They're obstacles in between me and what I want when what I want is more horsepower behind an app. For the same reasons I probably am better off with garbage collection than malloc [1]. I've only got so many brain cells. [1]: http://linux.die.net/man/3/malloc http://linux.die.net/man/3/malloc
    - mentat 10y ago
      It's not useful though. You can't take Docker "laptop containers" and use them in production. It's at best a development tool, at worst creating yet another environment to support and to have test differences in.
    - lobster_johnson 10y ago
      The nice thing about Kubernetes is that you don't really need to know much about those things, but if you do need to "level up" in terms of production complexity, those tools are there to use. I'm sure Swarm has its place, though.
      - 10y ago
- mentat 10y ago
  Kubernetes deployments are as high level as you can get while still having anything to do with actual operational environments. You don't have to use the lower level constructs if you don't want to. If you don't understand what either is doing, you're not going to be able to operate it anyway, and all of these still require operators.
djsumdog 10y ago
You still need to provision the Kubernetes/Docker servers? You still need Anisble/Puppet for that part. There aren't a lot of "Here's an API to throw up your containers and provision IPs" services out there. I see that Amazon has one (I avoid Amazon; they're the new Wal-Mart), but no one else really. I mean DigitalOcean provides CoreOS, so in theory you can provision one of those droplets and toss up some containers, but there's not a real API for "use this endpoint to deploy your container." In the corporate world, yes .. we have dev ops teams to try out and create provisioning systems to throw up Mesos or CoreOS or OpenStack clusters. Once they're up and your devs are using them, they're lower maintenance that provisioning individual servers for individual services. For home devs, it'd be nice to have a plug an forget for docker containers (other than Amazon). The other thing I still don't get about Docker: storage. MySQL containers have instructions for mounting volumes, but it doesn't feel like there's a good, standard way to handle storage. I'm sure that makes it more versatile, but like I said previously: plug and forget for home devs and startups. You want a databases as a service, get ready to shell out $80/month minimum for Amazon RDS or create a set of scripts around your docker containers that ensure you have slaves and/or backups.
- doublerebel 10y ago
  Try Joyent's Triton. It literally is > a real API for "use this endpoint to deploy your container." I've been using Triton since as early as possible and it's fantastic. You can tag/label containers and they show up in the SDN DNS by tag (called CNS), no need for Consul or other service discovery. I do still use Consul for many reasons but CNS is awesome for bootstrapping a cluster as well as public dns routing/failover.
- lugg 10y ago
  Don't try to put storage into containers at this point. You'll just be in for a bad time. Either run a seperate vps for MySQL or use a hosted solution. Containerization will get there but i think putting old some types of old school apps in containers is a bad idea. E.g. redis, beanstalk, and similar fit nicely I to clusters and restarting nodes without too much issue or downtime. MySQL, if your install is large enough, you never want it to go down. This pretty much goes against everything to do with containerization, your developers also gain no benefit of it being in a container (apart from maybe running a small db in developent only where Provisioning a vm would be too painful)
raesene9 10y ago
When you say K8s is lightweight, I presume you're purely talking about GKE deployments, and not on-prem or alternate cloud deployments? My experience of K8s so far is that GKE is happy path and most of the demonstrations/documentation is focused on that use case. When you step off that path, you can either go for something scripted which does a lot of things in the background, or what seems like quite an involved manual setup (etcd, controller node setup, Certificate setup, networking etc)

brudgers 10y ago

I don't disagree. I think the differences lie in reducing versus eliminating a requirement for operations engineers and the level at which abstractions are surfaced. To me, Swarm looks headed toward abstractions at a higher level than Kubernetes much like Docker surfaces higher level abstractions than LXC. Maybe that's the difference between a product and a solution. I don't know.

lobster_johnson 10y ago
> To me, Swarm looks headed toward abstractions at a higher level than Kubernetes Kubernetes admittedly takes a slightly different, modular, layered approach, whereas Swarm is simple to the point of naivety. This simplicity is potentially a threat to its future ability to adapt to different requirements, whereas Kubernetes offers a separation of primitives that allows it to scale from "simple" to "complex". For example, in Kubernetes, a deployment is a separate object from the pods themselves. You create a deployment, which in turn creates a replica set, which in turn creates or tears down pods as needed. But you don't really need to work directly with replica sets or know exactly how they work, but they're there, and can be used outside of a deployment. If all you care about is deploying apps, then you only need to deal with the deployment.
- hosh 10y ago
  And, there is enough documentation where you can: (1) Write your own custom deployer https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri (2) Write your own controllers https://deis.com/blog/2016/scheduling-your-kubernetes-pods-with-elixir/ https://deis.com/blog/2016/scheduling-your-kubernetes-pods-w... Those two are just examples. My point is that if you need to dig in, you can. You can do this because those higher-level behaviors are built on a solid foundation of well-thought-out primitives. What I expect to happen is that we will have a diverse set of things built on top of Kubernetes. Some of it will get folded into Kubernetes core. Some will not. I think the center of gravity for containers has been shifting towards Kubernetes.
- brudgers 10y ago
  Swarm is simple to the point of naivety. Exactly. To me, that approach is what allowed Docker to make containers on the laptop ubiquitous. Kubernetes is unlikely to be the tool that takes schedulers to that universe. Swarm might and I think that's the goal. I mean, I don't really want to care about pods and replica sets. They're obstacles in between me and what I want when what I want is more horsepower behind an app. For the same reasons I probably am better off with garbage collection than malloc [1]. I've only got so many brain cells. [1]: http://linux.die.net/man/3/malloc http://linux.die.net/man/3/malloc
  - mentat 10y ago
    It's not useful though. You can't take Docker "laptop containers" and use them in production. It's at best a development tool, at worst creating yet another environment to support and to have test differences in.
  - lobster_johnson 10y ago
    The nice thing about Kubernetes is that you don't really need to know much about those things, but if you do need to "level up" in terms of production complexity, those tools are there to use. I'm sure Swarm has its place, though.
    - 10y ago
mentat 10y ago
Kubernetes deployments are as high level as you can get while still having anything to do with actual operational environments. You don't have to use the lower level constructs if you don't want to. If you don't understand what either is doing, you're not going to be able to operate it anyway, and all of these still require operators.

djsumdog 10y ago

You still need to provision the Kubernetes/Docker servers? You still need Anisble/Puppet for that part. There aren't a lot of "Here's an API to throw up your containers and provision IPs" services out there. I see that Amazon has one (I avoid Amazon; they're the new Wal-Mart), but no one else really. I mean DigitalOcean provides CoreOS, so in theory you can provision one of those droplets and toss up some containers, but there's not a real API for "use this endpoint to deploy your container." In the corporate world, yes .. we have dev ops teams to try out and create provisioning systems to throw up Mesos or CoreOS or OpenStack clusters. Once they're up and your devs are using them, they're lower maintenance that provisioning individual servers for individual services. For home devs, it'd be nice to have a plug an forget for docker containers (other than Amazon). The other thing I still don't get about Docker: storage. MySQL containers have instructions for mounting volumes, but it doesn't feel like there's a good, standard way to handle storage. I'm sure that makes it more versatile, but like I said previously: plug and forget for home devs and startups. You want a databases as a service, get ready to shell out $80/month minimum for Amazon RDS or create a set of scripts around your docker containers that ensure you have slaves and/or backups.

doublerebel 10y ago
Try Joyent's Triton. It literally is > a real API for "use this endpoint to deploy your container." I've been using Triton since as early as possible and it's fantastic. You can tag/label containers and they show up in the SDN DNS by tag (called CNS), no need for Consul or other service discovery. I do still use Consul for many reasons but CNS is awesome for bootstrapping a cluster as well as public dns routing/failover.
lugg 10y ago
Don't try to put storage into containers at this point. You'll just be in for a bad time. Either run a seperate vps for MySQL or use a hosted solution. Containerization will get there but i think putting old some types of old school apps in containers is a bad idea. E.g. redis, beanstalk, and similar fit nicely I to clusters and restarting nodes without too much issue or downtime. MySQL, if your install is large enough, you never want it to go down. This pretty much goes against everything to do with containerization, your developers also gain no benefit of it being in a container (apart from maybe running a small db in developent only where Provisioning a vm would be too painful)

raesene9 10y ago

When you say K8s is lightweight, I presume you're purely talking about GKE deployments, and not on-prem or alternate cloud deployments? My experience of K8s so far is that GKE is happy path and most of the demonstrations/documentation is focused on that use case. When you step off that path, you can either go for something scripted which does a lot of things in the background, or what seems like quite an involved manual setup (etcd, controller node setup, Certificate setup, networking etc)