8 ms·

So confused by all the posts from people who say they run Swarm because kubernetes is too complicated or is only for huge deployments. I’ve had all sort of dif

So confused by all the posts from people who say they run Swarm because kubernetes is too complicated or is only for huge deployments. I’ve had all sort of difficulties installing Docker. By hand it’s not trivial to get a secure install. Docker machine is great except it’s often broken. The Docker machine dev team is a tired, understaffed bunch that’s always playing a sisyphean whack-a-mole against dozens of cloud providers and very needy posters on Github, myself included. Kubernetes on the other hand is trivial with GKE. It’s great for single node deployments. I run a single node on GKE and it’s awesome, easy, and very cheap. You can even run preemptible instances. The myth that kubernetes is complicated is largely perpetuated by the same kind of people who say React is complicated: the people who’ve not tried it. And like React, once you try kubernetes you never go back. Kubernetes is actually the orchestration equivalent of React. You declare what should be true, and Kubernetes takes care of the rest. And the features it provides are useful for any-sized application! If you try kubernetes you quickly discover persistent volumes and statefulsets, which take away most of the complexities out of stateful applications (ie most applications). You also discover ingress resources and controllers, which make trivial so many things that are difficult with Swarm, like TLS termination. Swam doesn’t have such features, which any non-trivial app (say, Django, wordpress, etc) benefits from tremendously.

moondev 9y ago
I agree for the most part, but kubernetes orchestrates docker (for now). It's also much easier to install than full blown k8s
dominotw 9y ago
GKE=Google cloud? I don't understand ur comment, yea using something in preinstalled in the cloud is easier than installing something on your own.
- tedmiston 9y ago
  GKE = Google Kubernetes Engine (aka Google Container Engine for Kubernetes on Google Cloud Platform) https://cloud.google.com/kubernetes-engine/ https://cloud.google.com/kubernetes-engine/
  - deleted 9y ago
    [deleted]
- linkmotif 9y ago
  Docker isn't easy to install (see my OP). Kubernetes is probably more difficult to install, but it's not more expensive to use GKE than to roll your own Kubernetes elsewhere. And once you understand Kubernetes from a user perspecitve, it's easier to set up on your own.
  - kuschku 9y ago
    > it's not more expensive to use GKE than to roll your own Kubernetes elsewhere. $13 per month per egress rule, bandwidth costs a factor of 100 more expensive than dedicated hosters, and costs for hardware 10x of what dedicated hosters offer. I’m not sure what your definition of "more expensive" is, but compared to renting dedicated hardware (e.g., from Hetzner), or colocating, GKE is significantly more expensive. Of course, if you’re on the scale of Spotify, you can get much better deals – but from what I see on https://cloud.google.com/products/calculator/ https://cloud.google.com/products/calculator/, it’s not cost-effective.
    - linkmotif 9y ago
      Maybe it’s my inexperience. I’ve not had to define egress rules yet. I run 1 preemptible n1-standard-2 node and pay for ancillary services. It’s $35-40/month.
      - kuschku 9y ago
        I have the equivalent of 2 n1-standard-4 nodes (a bit more performant, actually), rented as dedicated servers, for $30 a month. Including egress and bandwidth. That’s why I disagreed with the "GKE isn’t more expensive".
  - flylib 9y ago
    "but it's not more expensive to use GKE than to roll your own Kubernetes elsewhere." depends on what scale your at and the fact that your vendor locked, can't use GKE in other data centers
tedmiston 9y ago
Do you really find k8s useful for a single node deployment or were you just making an example? I haven't used but I have used DC/OS, Mesos, and Marathon extensively, which is not a setup I'd do for a small number of nodes personally.
- linkmotif 9y ago
  Yes. Though I should emphasize that I'm using GKE. So it's zero config, zero ops. I am running a single n1-standard-2 node on GKE for a smallish app right now. Comes to 30-40 dollars a month all in (egress traffic, cloud storage, other services). I am working hard on getting people to use this app, and it's great to know that scaling will be best-in-class if I succeed. But I stress it's not about the scaling. It's about the features even at a single node. I wouldn't be spending time writing this stuff had it not been revelatory for me. PS. If you're not able to try it on GKE, there's also https://github.com/kubernetes/kops https://github.com/kubernetes/kops. GKE is great for trying, at least, though. Just get a cluster running and get rolling with the concepts. Then you'll know what it's all about. There isn't that much to it, and what there is to it is great and well documented with a great community.
  - Axsuul 9y ago
    Yet with a Docker Swarm setup, you could get away with probably 10 dollars/month. K8s likely taking more resources than your app containers is why k8s may not be the best option for single node deployments. I've personally tried both orchestration options and much prefer Swarm to smaller deployments. Sure k8s could prepare you better for the future if your app takes off to the moon but that's just premature optimization.
    - linkmotif 9y ago
      This might all very well be true, but Swarm doesn’t let you connect cloud disks directly to your containers, to my knowledge. Also I challenge the $10 because what cloud instances plus ancillary services add up to $10? I’m talking 2 CPUs and 7.5gb RAM.
LaGrange 9y ago
"Kubernetes on the other hand is trivial with GKE." Yes, true, but that's apple vs oranges. Swarm doesn't compete against GKE, it competes against Kubeadm. Because sometimes one actually can't or won't use cloud services.
- linkmotif 9y ago
  I guess you are right, it's apple vs. oranges. If we take GKE out of the picture, in terms of setup, is it accurate to say it competes with Kops?
  - LaGrange 9y ago
    Kops seem (from quick overview) to be focused on AWS and/or other cloud providers. Several times I've been limited either to actual hardware in a room, or a contracted provider giving me something along the lines of OpenStack. Unless there's a project I've missed, at the moment that means I either have to deal with kubeadm, or just roll from scratch myself (umm... nope).
drdaeman 9y ago
> Kubernetes on the other hand is trivial with GKE How do I install GKE on my servers? ;) > By hand it’s not trivial to get a secure install. The default install (basically, adding a repo and apt-get install docker-ce on Debian and derivatives - trivial to automate with Ansible) is reasonably secure if you view Docker as a tool for packaging and task scheduling with some nice extras and don't buy the marketed isolation properties. It only listens for commands on a local socket, and permissions are sane. I haven't looked into Swarm mode protocol traffic, though but I don't think it's tweakable anyway. > The myth that kubernetes is complicated is largely perpetuated by the same kind of people who say React is complicated: the people who’ve not tried it. I've tried K8s. I've set up a test cluster, it worked, I wrote some YAML, it worked, all good. So I've worsened the conditions (explicitly going into "I want things to break" territory) and made it fail. I've researched how hard it is to diagnose the problem and fix it - it happened to be complicated. At least, for me. Just felt that "if something goes wrong here, I'll have bad time trying to fix it". Surely, this is not the case on GKE where you don't run and don't manage the cluster. I had somewhat similar experience with Docker and Docker Swarm mode, and it was significantly easier to dive into the code, find out the relevant parts and see what's going on. > difficult with Swarm, like TLS termination YMMV, but I just add some labels to the service and Traefik does the rest. ;) (But, yeah, Traefik with Swarm requires some reasonable but not exactly obvious networking logic. May take one deployment of "why I'm getting 504s?!" to figure it out. And Traefik needs access to manager nodes to work with built-in Swarm service discovery.)
- linkmotif 9y ago
  Thanks for the reply. I agree with what you say. I'm not tying to say people should all jump to k8s. Having options on the market is great. But I was trying to refute the notion that Kubernetes has no advantages unless you're running a huge cluster. My main points where: * It works great with 1 node. * It comes with many features that Swarm does not have that are useful even at 1 node (PersistentVolumes, StatefulSets are biggest for me, though there are _many_ more I wouldn't want to go without anymore). * Docker is not trivial to set up, either. > How do I install GKE on my servers? ;) Yes, of course. I was just saying there's a solid option to start exploring quickly. > It only listens for commands on a local socket. This is kind of a non-starter, isn't it? Of course it's easy to apt-get install docker, but then you want to control it remotely, right? Once you realize how nice it is to control Docker remotely, it's hard to imagine life before.
csmiller 9y ago
Have any recommended resources for learning Kubernetes? I've looked at it a few times, but it always seemed rather intimidating.
- dominotw 9y ago
  manning book Kubernetes in action was easy( albeit a bit tedious) read.
- bboreham 9y ago
  Interactive tutorials from katacoda: https://www.katacoda.com/courses/kubernetes https://www.katacoda.com/courses/kubernetes
- dankohn1 9y ago
  If you like videos, I recommend the free EdX course. If you don't I recommend the Katacoda interactive sessions. https://kubernetes.io/docs/tutorials/ https://kubernetes.io/docs/tutorials/ (Disclosure: I run CNCF, which hosts Kubernetes and funded the EdX course.)
tokenizerrr 9y ago
You're running on GKE. Of course that's easy, they're handling all the difficult parts for you! It's still not that difficult running it on your own servers, but there are many more pain points. Please get some experience with that, then re-evaluate whether Docker or Kubernetes is easier for small deployments.
- linkmotif 9y ago
  Right, I agree. But with your easier Swarm setup, how do you then attach cloud disks directly to your docker container, like a PersistentVolume affords? That one feature makes basically anything worth it, IMO. Most apps are stateful.
  - jlawer 9y ago
    NFS mounts, provisioned via the same configuration management platform the hardware is provisioned by (Saltstack in my case)
  - Bgd1 9y ago
    You use one of the Docker plugins: https://docs.docker.com/docker-for-aws/persistent-data-volumes/ https://docs.docker.com/docker-for-aws/persistent-data-volum... https://github.com/thecodeteam/rexray https://github.com/thecodeteam/rexray
  - tokenizerrr 9y ago
    You don't, because you're running on your own hardware, and there is no concept of cloud disks. There are some solutions like Ceph, but they're tough to setup. Usually I just pin containers to hosts. This is fine on a small setup. In fact, many of my small setups are just a single host.
    - raarts 9y ago
      > Ceph tough to setup Try ProxMox. They have it built-in.
jacksmith21006 9y ago
This! I wonder if people say it because it is what they heard or from actual experience? I found K8s far easier but seen people say otherwise and suspect there is a bit of an echochamber happening.
- raarts 9y ago
  I can't believe that you came to that conclusion after trying them both. Try comparing the config files for Swarm and k8s for the same system, the former are much shorter and simpler.
AlexCoventry 9y ago
I'm not proposing the following as a solution to your problem, which I'm sure is on a much larger scale, affording many opportunities for failure. I'm curious how it breaks in that context, though. I regularly use a script like this, and it's never failed me unless it runs out of disk space: https://github.com/coventry/InformationDropout/blob/master/preprocessing/docker-install.sh https://github.com/coventry/InformationDropout/blob/master/p...

moondev 9y ago

I agree for the most part, but kubernetes orchestrates docker (for now). It's also much easier to install than full blown k8s

dominotw 9y ago

GKE=Google cloud? I don't understand ur comment, yea using something in preinstalled in the cloud is easier than installing something on your own.

tedmiston 9y ago
GKE = Google Kubernetes Engine (aka Google Container Engine for Kubernetes on Google Cloud Platform) https://cloud.google.com/kubernetes-engine/ https://cloud.google.com/kubernetes-engine/
- deleted 9y ago
  [deleted]
linkmotif 9y ago
Docker isn't easy to install (see my OP). Kubernetes is probably more difficult to install, but it's not more expensive to use GKE than to roll your own Kubernetes elsewhere. And once you understand Kubernetes from a user perspecitve, it's easier to set up on your own.
- kuschku 9y ago
  > it's not more expensive to use GKE than to roll your own Kubernetes elsewhere. $13 per month per egress rule, bandwidth costs a factor of 100 more expensive than dedicated hosters, and costs for hardware 10x of what dedicated hosters offer. I’m not sure what your definition of "more expensive" is, but compared to renting dedicated hardware (e.g., from Hetzner), or colocating, GKE is significantly more expensive. Of course, if you’re on the scale of Spotify, you can get much better deals – but from what I see on https://cloud.google.com/products/calculator/ https://cloud.google.com/products/calculator/, it’s not cost-effective.
  - linkmotif 9y ago
    Maybe it’s my inexperience. I’ve not had to define egress rules yet. I run 1 preemptible n1-standard-2 node and pay for ancillary services. It’s $35-40/month.
    - kuschku 9y ago
      I have the equivalent of 2 n1-standard-4 nodes (a bit more performant, actually), rented as dedicated servers, for $30 a month. Including egress and bandwidth. That’s why I disagreed with the "GKE isn’t more expensive".
- flylib 9y ago
  "but it's not more expensive to use GKE than to roll your own Kubernetes elsewhere." depends on what scale your at and the fact that your vendor locked, can't use GKE in other data centers

tedmiston 9y ago

Do you really find k8s useful for a single node deployment or were you just making an example? I haven't used but I have used DC/OS, Mesos, and Marathon extensively, which is not a setup I'd do for a small number of nodes personally.

linkmotif 9y ago
Yes. Though I should emphasize that I'm using GKE. So it's zero config, zero ops. I am running a single n1-standard-2 node on GKE for a smallish app right now. Comes to 30-40 dollars a month all in (egress traffic, cloud storage, other services). I am working hard on getting people to use this app, and it's great to know that scaling will be best-in-class if I succeed. But I stress it's not about the scaling. It's about the features even at a single node. I wouldn't be spending time writing this stuff had it not been revelatory for me. PS. If you're not able to try it on GKE, there's also https://github.com/kubernetes/kops https://github.com/kubernetes/kops. GKE is great for trying, at least, though. Just get a cluster running and get rolling with the concepts. Then you'll know what it's all about. There isn't that much to it, and what there is to it is great and well documented with a great community.
- Axsuul 9y ago
  Yet with a Docker Swarm setup, you could get away with probably 10 dollars/month. K8s likely taking more resources than your app containers is why k8s may not be the best option for single node deployments. I've personally tried both orchestration options and much prefer Swarm to smaller deployments. Sure k8s could prepare you better for the future if your app takes off to the moon but that's just premature optimization.
  - linkmotif 9y ago
    This might all very well be true, but Swarm doesn’t let you connect cloud disks directly to your containers, to my knowledge. Also I challenge the $10 because what cloud instances plus ancillary services add up to $10? I’m talking 2 CPUs and 7.5gb RAM.

LaGrange 9y ago

"Kubernetes on the other hand is trivial with GKE." Yes, true, but that's apple vs oranges. Swarm doesn't compete against GKE, it competes against Kubeadm. Because sometimes one actually can't or won't use cloud services.

linkmotif 9y ago
I guess you are right, it's apple vs. oranges. If we take GKE out of the picture, in terms of setup, is it accurate to say it competes with Kops?
- LaGrange 9y ago
  Kops seem (from quick overview) to be focused on AWS and/or other cloud providers. Several times I've been limited either to actual hardware in a room, or a contracted provider giving me something along the lines of OpenStack. Unless there's a project I've missed, at the moment that means I either have to deal with kubeadm, or just roll from scratch myself (umm... nope).

drdaeman 9y ago

> Kubernetes on the other hand is trivial with GKE How do I install GKE on my servers? ;) > By hand it’s not trivial to get a secure install. The default install (basically, adding a repo and apt-get install docker-ce on Debian and derivatives - trivial to automate with Ansible) is reasonably secure if you view Docker as a tool for packaging and task scheduling with some nice extras and don't buy the marketed isolation properties. It only listens for commands on a local socket, and permissions are sane. I haven't looked into Swarm mode protocol traffic, though but I don't think it's tweakable anyway. > The myth that kubernetes is complicated is largely perpetuated by the same kind of people who say React is complicated: the people who’ve not tried it. I've tried K8s. I've set up a test cluster, it worked, I wrote some YAML, it worked, all good. So I've worsened the conditions (explicitly going into "I want things to break" territory) and made it fail. I've researched how hard it is to diagnose the problem and fix it - it happened to be complicated. At least, for me. Just felt that "if something goes wrong here, I'll have bad time trying to fix it". Surely, this is not the case on GKE where you don't run and don't manage the cluster. I had somewhat similar experience with Docker and Docker Swarm mode, and it was significantly easier to dive into the code, find out the relevant parts and see what's going on. > difficult with Swarm, like TLS termination YMMV, but I just add some labels to the service and Traefik does the rest. ;) (But, yeah, Traefik with Swarm requires some reasonable but not exactly obvious networking logic. May take one deployment of "why I'm getting 504s?!" to figure it out. And Traefik needs access to manager nodes to work with built-in Swarm service discovery.)

linkmotif 9y ago
Thanks for the reply. I agree with what you say. I'm not tying to say people should all jump to k8s. Having options on the market is great. But I was trying to refute the notion that Kubernetes has no advantages unless you're running a huge cluster. My main points where: * It works great with 1 node. * It comes with many features that Swarm does not have that are useful even at 1 node (PersistentVolumes, StatefulSets are biggest for me, though there are _many_ more I wouldn't want to go without anymore). * Docker is not trivial to set up, either. > How do I install GKE on my servers? ;) Yes, of course. I was just saying there's a solid option to start exploring quickly. > It only listens for commands on a local socket. This is kind of a non-starter, isn't it? Of course it's easy to apt-get install docker, but then you want to control it remotely, right? Once you realize how nice it is to control Docker remotely, it's hard to imagine life before.

csmiller 9y ago

Have any recommended resources for learning Kubernetes? I've looked at it a few times, but it always seemed rather intimidating.

dominotw 9y ago
manning book Kubernetes in action was easy( albeit a bit tedious) read.
bboreham 9y ago
Interactive tutorials from katacoda: https://www.katacoda.com/courses/kubernetes https://www.katacoda.com/courses/kubernetes
dankohn1 9y ago
If you like videos, I recommend the free EdX course. If you don't I recommend the Katacoda interactive sessions. https://kubernetes.io/docs/tutorials/ https://kubernetes.io/docs/tutorials/ (Disclosure: I run CNCF, which hosts Kubernetes and funded the EdX course.)

tokenizerrr 9y ago

You're running on GKE. Of course that's easy, they're handling all the difficult parts for you! It's still not that difficult running it on your own servers, but there are many more pain points. Please get some experience with that, then re-evaluate whether Docker or Kubernetes is easier for small deployments.

linkmotif 9y ago
Right, I agree. But with your easier Swarm setup, how do you then attach cloud disks directly to your docker container, like a PersistentVolume affords? That one feature makes basically anything worth it, IMO. Most apps are stateful.
- jlawer 9y ago
  NFS mounts, provisioned via the same configuration management platform the hardware is provisioned by (Saltstack in my case)
- Bgd1 9y ago
  You use one of the Docker plugins: https://docs.docker.com/docker-for-aws/persistent-data-volumes/ https://docs.docker.com/docker-for-aws/persistent-data-volum... https://github.com/thecodeteam/rexray https://github.com/thecodeteam/rexray
- tokenizerrr 9y ago
  You don't, because you're running on your own hardware, and there is no concept of cloud disks. There are some solutions like Ceph, but they're tough to setup. Usually I just pin containers to hosts. This is fine on a small setup. In fact, many of my small setups are just a single host.
  - raarts 9y ago
    > Ceph tough to setup Try ProxMox. They have it built-in.

jacksmith21006 9y ago

This! I wonder if people say it because it is what they heard or from actual experience? I found K8s far easier but seen people say otherwise and suspect there is a bit of an echochamber happening.

raarts 9y ago
I can't believe that you came to that conclusion after trying them both. Try comparing the config files for Swarm and k8s for the same system, the former are much shorter and simpler.

AlexCoventry 9y ago

I'm not proposing the following as a solution to your problem, which I'm sure is on a much larger scale, affording many opportunities for failure. I'm curious how it breaks in that context, though. I regularly use a script like this, and it's never failed me unless it runs out of disk space: https://github.com/coventry/InformationDropout/blob/master/preprocessing/docker-install.sh https://github.com/coventry/InformationDropout/blob/master/p...