6 ms·

I'd heavily encourage you to read the docs. If the scenarios you outlined are your top concern for your homelab, I suspect the complexity and resource overhead

by sgk284 5y ago

I'd heavily encourage you to read the docs. If the scenarios you outlined are your top concern for your homelab, I suspect the complexity and resource overhead of k8s isn't warranted. - Cert Renewal / Rotation: https://docs.docker.com/engine/reference/commandline/swarm_ca/ https://docs.docker.com/engine/reference/commandline/swarm_c... - TLS Termination: Arguably this is an application layer concern and out of scope for an orchestration layer, but in either case use Nginx (or your favorite TLS terminating proxy) as your ingress point. This is equivalent to the Nginx ingress controller in k8s. - Storage: Docker volumes (local or ceph, for distributed). - Disaster Recovery: Node failure? Cluster failure? Data loss? What kind of DR? The strategies you'd use with Swarm will be very similar to those you'd use with k8s. - Grouping resources: Labels and/or services. In a Swarm world you'll have fewer moving pieces, far lower resource requirements, and will manage it all in a docker-compose file(s). You'll also get things like service discovery, routing, and secrets management for "free". No additional configuration needed.

MrOwen 5y ago
I would argue the myriad of alternatives you've suggested as alternatives produce significantly more overhead. You linked Swarm CA renewal operations; this is maybe 1/50 of the functionality of cert-manager. Is there a product like cert-manager that I can run on Swarm that's tightly integrated? Why not have ingress intrinsically linked to the underlying reaources like k8s does? Your suggestion means that someone needs to come up with a methodology and automation to update ingress with Swarm. Yeah, it's obviously possible but imo, Traefik ingress reaources are way easier to manage with config spread across far fewer places. Storage.. I have an nfs storage provider in k8s which allows me to dynamically create storage with lifecycle management included... for free... just because I'm using k8s. DR as in something happens to my house or other technical issues which bring down my cluster. I would posit that it's much easier for me to restore a cluster because it's all yaml and imo more descriptive and capable than swarm yaml syntax. I think you get my point. Yes, you can do it all with Swarm but with things like k3s and k0s, it's just as easy to start using k8s. And way more help, docs, and tooling to boot.
- danudey 5y ago
  Honestly, this is such a common refrain in open-source communities. "I'm going to use this product/project." "You shouldn't use that, use this instead." "But this product/project has all these features I want." "Sure but you can just build and maintain those features yourself if you want. It's open source, after all!" Or alternately: "Well no one really needs those anyway, it's just people being told that they do." I've been hearing this same refrain for over twenty years now. It seems as though some people don't understand that there's actual value in having a system that just works, rather than one that you have to build and maintain yourself just to emulate the system that works.
  - saurik 5y ago
    So, I don't use either of these things, but still tend to read the arguments between people who do, and that isn't what it feels like to me (as again, someone who doesn't use either of these things). Specifically, it feels like you are missing that the people who argue against k8s are saying that it is extremely difficult to configure, and so instead of this clean "some people don't understand that it is nice to have a system that just works", it feels like the argument is actually about a tradeoff between what you think is easier to put together: a single system with a million options that seem to all need to be configured or it doesn't work or a handful of systems that are each easy to understand in separate, but where you will need to do all the glue manually.
  - KronisLV 5y ago
    I do partially agree with the point that you're making, but perhaps from a slightly different angle - i'd say that the biggest value that Kubernetes provides (or at least tries to, with varying results), is having a common set of abstractions that allow both reusing skills and also treating different vendors' clusters as very similar resources. > It seems as though some people don't understand that there's actual value in having a system that just works, rather than one that you have to build and maintain yourself just to emulate the system that works. This, however, feels like a lie. It might "just work" because of the efforts of hundreds of engineers that are in the employ of the larger cloud vendors, because they're handling all of the problems in the background, however that will also cost you. It might also "just work" in on prem deployments because you simply haven't run into any problems yet... and in my experience it's not a matter of "Whether?" but rather one of "When?" And when these problems inevitably do surface, you have to ask yourself, whether you have the capacity to debug them and mitigate them. In the case of K3s clusters, or another lightweight Kubernetes distro that might be doable. However, if you're dealing with a large distro like RKE or your own custom one, then you might have to debug this complex, fragmented and distributed system to get to the root cause and then figure out how to solve it. There's definitely a reason why so many DevOps specialists with Kubernetes credentials are so well paid right now. There's also a reason for why many of the enterprise deployments of Kubernetes that i've seen are essentially black holes of manhours, much like implementing Apache Kafka when something like RabbitMQ might have sufficed - sometimes because of chasing after hype and doing CV driven development, other times because the people didn't know enough about the other options out there. There's also a reason for why Docker Swarm is still alive and kicking despite how its maintenance and development have been mismanaged, as well as why HashiCorp Nomad is liked by many. In short: The people who will try to build their own complex systems will most likely fail and will build something undocumented, unmaintainable, buggy and unstable. The people who try to adapt the technologies that are better suited for large scale deployments for their own needs, might find that the complexity grinds their velocity to a halt and makes them waste huge amounts of effort in maintaining this solution. The people who want to make someone else deal with these complexities and just want the benefits of the tech, will probably have to pay appropriately for this, which sometimes is a good deal, but other times is a bad one and also ensures vendor lock. The people who reevaluate what they're trying to do and how much they actually need to do, might sometimes settle for a lightweight solution which does what's necessary, not much more and leads to vaguely positive outcomes.
t_sawyer 5y ago
Yes, ingress like Traefik can auto renew certs. My problem with swarm is that the overlay networking was super flaky. Dead containers still being registered on the network cause every X requests to fail (since it’s round robin) because it’s routing to an overlay IP that doesn’t exist. Further, I could never find storage adapters for cloud storage that continued to be maintained. Rexray was the longest standing but hasn’t had a commit since Jan 2019.

by sgk284 5y ago

MrOwen 5y ago

I would argue the myriad of alternatives you've suggested as alternatives produce significantly more overhead. You linked Swarm CA renewal operations; this is maybe 1/50 of the functionality of cert-manager. Is there a product like cert-manager that I can run on Swarm that's tightly integrated? Why not have ingress intrinsically linked to the underlying reaources like k8s does? Your suggestion means that someone needs to come up with a methodology and automation to update ingress with Swarm. Yeah, it's obviously possible but imo, Traefik ingress reaources are way easier to manage with config spread across far fewer places. Storage.. I have an nfs storage provider in k8s which allows me to dynamically create storage with lifecycle management included... for free... just because I'm using k8s. DR as in something happens to my house or other technical issues which bring down my cluster. I would posit that it's much easier for me to restore a cluster because it's all yaml and imo more descriptive and capable than swarm yaml syntax. I think you get my point. Yes, you can do it all with Swarm but with things like k3s and k0s, it's just as easy to start using k8s. And way more help, docs, and tooling to boot.

danudey 5y ago
Honestly, this is such a common refrain in open-source communities. "I'm going to use this product/project." "You shouldn't use that, use this instead." "But this product/project has all these features I want." "Sure but you can just build and maintain those features yourself if you want. It's open source, after all!" Or alternately: "Well no one really needs those anyway, it's just people being told that they do." I've been hearing this same refrain for over twenty years now. It seems as though some people don't understand that there's actual value in having a system that just works, rather than one that you have to build and maintain yourself just to emulate the system that works.
- saurik 5y ago
  So, I don't use either of these things, but still tend to read the arguments between people who do, and that isn't what it feels like to me (as again, someone who doesn't use either of these things). Specifically, it feels like you are missing that the people who argue against k8s are saying that it is extremely difficult to configure, and so instead of this clean "some people don't understand that it is nice to have a system that just works", it feels like the argument is actually about a tradeoff between what you think is easier to put together: a single system with a million options that seem to all need to be configured or it doesn't work or a handful of systems that are each easy to understand in separate, but where you will need to do all the glue manually.
- KronisLV 5y ago
  I do partially agree with the point that you're making, but perhaps from a slightly different angle - i'd say that the biggest value that Kubernetes provides (or at least tries to, with varying results), is having a common set of abstractions that allow both reusing skills and also treating different vendors' clusters as very similar resources. > It seems as though some people don't understand that there's actual value in having a system that just works, rather than one that you have to build and maintain yourself just to emulate the system that works. This, however, feels like a lie. It might "just work" because of the efforts of hundreds of engineers that are in the employ of the larger cloud vendors, because they're handling all of the problems in the background, however that will also cost you. It might also "just work" in on prem deployments because you simply haven't run into any problems yet... and in my experience it's not a matter of "Whether?" but rather one of "When?" And when these problems inevitably do surface, you have to ask yourself, whether you have the capacity to debug them and mitigate them. In the case of K3s clusters, or another lightweight Kubernetes distro that might be doable. However, if you're dealing with a large distro like RKE or your own custom one, then you might have to debug this complex, fragmented and distributed system to get to the root cause and then figure out how to solve it. There's definitely a reason why so many DevOps specialists with Kubernetes credentials are so well paid right now. There's also a reason for why many of the enterprise deployments of Kubernetes that i've seen are essentially black holes of manhours, much like implementing Apache Kafka when something like RabbitMQ might have sufficed - sometimes because of chasing after hype and doing CV driven development, other times because the people didn't know enough about the other options out there. There's also a reason for why Docker Swarm is still alive and kicking despite how its maintenance and development have been mismanaged, as well as why HashiCorp Nomad is liked by many. In short: The people who will try to build their own complex systems will most likely fail and will build something undocumented, unmaintainable, buggy and unstable. The people who try to adapt the technologies that are better suited for large scale deployments for their own needs, might find that the complexity grinds their velocity to a halt and makes them waste huge amounts of effort in maintaining this solution. The people who want to make someone else deal with these complexities and just want the benefits of the tech, will probably have to pay appropriately for this, which sometimes is a good deal, but other times is a bad one and also ensures vendor lock. The people who reevaluate what they're trying to do and how much they actually need to do, might sometimes settle for a lightweight solution which does what's necessary, not much more and leads to vaguely positive outcomes.

t_sawyer 5y ago

Yes, ingress like Traefik can auto renew certs. My problem with swarm is that the overlay networking was super flaky. Dead containers still being registered on the network cause every X requests to fail (since it’s round robin) because it’s routing to an overlay IP that doesn’t exist. Further, I could never find storage adapters for cloud storage that continued to be maintained. Rexray was the longest standing but hasn’t had a commit since Jan 2019.