5 ms·

I feel like their needs to be a sanity check. Realistically what does kubernetes do to "not make software fail"? Health checks? Autorestarting containers that c

I feel like their needs to be a sanity check. Realistically what does kubernetes do to "not make software fail"? Health checks? Autorestarting containers that crash? Enforce various timeouts? Relatively quick distributed configuration deployments? These can be hard problems for sure, but do you need all of kubernetes to get these benefits in a production application?

theflyinghorse 7y ago
Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).
- y4mi 7y ago
  But swarm only has overlay network iirc? Swarm is great to get simple orchestration going, but it doesn't really do the same thing. With k8s you can configure multi-point routing, storage backends and load balancing among other things. With swarm, you get ... Overlay network (which is shit if you actually need to really scale) and 'scaling', which starts n containers of the same service. Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don't need. We've used both at my current job for toy projects (self hosted). Never in production however. And I'm a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers
  - hinkley 7y ago
    Possibly edgy opinion: Load balancing should be a solved problem already. Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing. Any competitor should be able to use the same solutions. To put it another way, this shouldn't be a differentiator. The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don't take advantage of things like diurnal variability in workloads.
    - AaronFriel 7y ago
      Kubernetes does use dead simple off-the-shelf software for ingress and load balancing. That software though, unfortunately, has a lot of knobs, and what "Ingress" and "Service" resources do is make sure those knobs are turned to the right settings. The nginx ingress controller for example, under the hood, just generates and applies a big ol' nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.
      - darkarmani 7y ago
        Is it hard to integrate a (HA) hardware load-balancer in front instead?
        elsonrodriguez 7y ago
        The easy way to do this is with NodePorts, wherein you configure your LB with all the nodes in your cluster being app servers on a certain port for a certain app. However you will lose some performance as there's some iptables magic under the hood. Beyond that there's a sea of options for more native integrations that will depend on whether your LB vendor has an K8s integration, how friendly your networking team is, and how much code you're willing to write.
        merb 7y ago
        for baremetal you can use metallb or kube-router if you have bgp infra. no need for hardware ha lb
      - eeZah7Ux 7y ago
        That's the opposite of dead simple and k8s tries hard to make things more complex than needed.
      - hinkley 7y ago
        > That software though, unfortunately, has a lot of knobs Lots of people have different definitions of 'easy' which is why I didn't say 'easy'. But how did you get this far off the rails with 'dead simple'?
  - cpuguy83 7y ago
    Swarm comes with overlay networking, you can install network plugins for whatever you need.
    - pepemon 7y ago
      You can't switch networking for Swarm Mode
      - cpuguy83 7y ago
        Yes you can. The plugin just needs to be swarm capable. https://docs.docker.com/engine/extend/plugins_network/ https://docs.docker.com/engine/extend/plugins_network/
- geezerjay 7y ago
  Docker Swarm doesn't support basic features such as shared volumes (between separate hosts), though.
  - BretFisher 7y ago
    sure it does, you just need a proper volume driver like rexray AWS EFS
    - geezerjay 7y ago
      The following link states that rexray does not support shared volumes, and that it was designed so that "it forcefully restricts a single volume to only be available to one host at a time." https://github.com/rexray/rexray/issues/343 https://github.com/rexray/rexray/issues/343
- haolez 7y ago
  I made this choice as well and I regret it. I've had a nice share of bugs regarding DNS and service discovery. However, those bugs seem to be fixed after version 19.x, but I'm still bitter about all the problems that I had because of simple features failing on me.
- tepidandroid 7y ago
  The same can be said of Hashistack (Consul, Nomad, Vault). Orders of magnitude simpler with the obvious tradeoff of being less featureful.
  - chucky_z 7y ago
    Consul and Vault offer functionality that Kubernetes is completely lacking, though.
  - StavrosK 7y ago
    Is it worth going with those? I'm torn between deploying stuff on Kubernetes (on a managed control plane, of course, which you can get for free nowadays with providers as long as you use their own machines) and rolling my own with the Hashistack.
    - tepidandroid 7y ago
      Depends on your use-case, really. IMO, Hashistack is fine if you have well-defined requirements. You can pick and choose the components you need (including non Hashi stuff) and integrate them with relative ease, keeping resource footprint and operational complexity at a minimum. I’ve found that it’s quite easy to pinpoint problems when they arise. Kubernetes is more of an everything-and-the-kitchen-sink approach that you can’t really outgrow but because of its monolithic design, debugging can be quite challenging. It does, however, come with a much larger and much more active community that you can go to for help.

theflyinghorse 7y ago

Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).

y4mi 7y ago
But swarm only has overlay network iirc? Swarm is great to get simple orchestration going, but it doesn't really do the same thing. With k8s you can configure multi-point routing, storage backends and load balancing among other things. With swarm, you get ... Overlay network (which is shit if you actually need to really scale) and 'scaling', which starts n containers of the same service. Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don't need. We've used both at my current job for toy projects (self hosted). Never in production however. And I'm a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers
- hinkley 7y ago
  Possibly edgy opinion: Load balancing should be a solved problem already. Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing. Any competitor should be able to use the same solutions. To put it another way, this shouldn't be a differentiator. The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don't take advantage of things like diurnal variability in workloads.
  - AaronFriel 7y ago
    Kubernetes does use dead simple off-the-shelf software for ingress and load balancing. That software though, unfortunately, has a lot of knobs, and what "Ingress" and "Service" resources do is make sure those knobs are turned to the right settings. The nginx ingress controller for example, under the hood, just generates and applies a big ol' nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.
    - darkarmani 7y ago
      Is it hard to integrate a (HA) hardware load-balancer in front instead?
      - elsonrodriguez 7y ago
        The easy way to do this is with NodePorts, wherein you configure your LB with all the nodes in your cluster being app servers on a certain port for a certain app. However you will lose some performance as there's some iptables magic under the hood. Beyond that there's a sea of options for more native integrations that will depend on whether your LB vendor has an K8s integration, how friendly your networking team is, and how much code you're willing to write.
      - merb 7y ago
        for baremetal you can use metallb or kube-router if you have bgp infra. no need for hardware ha lb
    - eeZah7Ux 7y ago
      That's the opposite of dead simple and k8s tries hard to make things more complex than needed.
    - hinkley 7y ago
      > That software though, unfortunately, has a lot of knobs Lots of people have different definitions of 'easy' which is why I didn't say 'easy'. But how did you get this far off the rails with 'dead simple'?
- cpuguy83 7y ago
  Swarm comes with overlay networking, you can install network plugins for whatever you need.
  - pepemon 7y ago
    You can't switch networking for Swarm Mode
    - cpuguy83 7y ago
      Yes you can. The plugin just needs to be swarm capable. https://docs.docker.com/engine/extend/plugins_network/ https://docs.docker.com/engine/extend/plugins_network/
geezerjay 7y ago
Docker Swarm doesn't support basic features such as shared volumes (between separate hosts), though.
- BretFisher 7y ago
  sure it does, you just need a proper volume driver like rexray AWS EFS
  - geezerjay 7y ago
    The following link states that rexray does not support shared volumes, and that it was designed so that "it forcefully restricts a single volume to only be available to one host at a time." https://github.com/rexray/rexray/issues/343 https://github.com/rexray/rexray/issues/343
haolez 7y ago
I made this choice as well and I regret it. I've had a nice share of bugs regarding DNS and service discovery. However, those bugs seem to be fixed after version 19.x, but I'm still bitter about all the problems that I had because of simple features failing on me.
tepidandroid 7y ago
The same can be said of Hashistack (Consul, Nomad, Vault). Orders of magnitude simpler with the obvious tradeoff of being less featureful.
- chucky_z 7y ago
  Consul and Vault offer functionality that Kubernetes is completely lacking, though.
- StavrosK 7y ago
  Is it worth going with those? I'm torn between deploying stuff on Kubernetes (on a managed control plane, of course, which you can get for free nowadays with providers as long as you use their own machines) and rolling my own with the Hashistack.
  - tepidandroid 7y ago
    Depends on your use-case, really. IMO, Hashistack is fine if you have well-defined requirements. You can pick and choose the components you need (including non Hashi stuff) and integrate them with relative ease, keeping resource footprint and operational complexity at a minimum. I’ve found that it’s quite easy to pinpoint problems when they arise. Kubernetes is more of an everything-and-the-kitchen-sink approach that you can’t really outgrow but because of its monolithic design, debugging can be quite challenging. It does, however, come with a much larger and much more active community that you can go to for help.