10 ms·

K8S is great if you have Google problems, but most people don't and I think much of hype around it is much like the hype that existed a decade ago around "web s

by sgk284 5y ago

K8S is great if you have Google problems, but most people don't and I think much of hype around it is much like the hype that existed a decade ago around "web scale" and "big data" and "NoSQL for everything". Docker Swarm is (with some use-case specific exceptions) more than sufficient for any shop running a few hundred nodes with a few thousand containers – and it is two orders of magnitude less complex than k8s. You can read Docker Swarm's docs end-to-end in < 30 minutes, and have a cluster running in roughly the amount of time it takes to install Docker on three boxes. Docker's failing was a failure to market Docker Swarm. Most people don't even know it exists, yet every single machine with Docker has it already! The last time I checked, they only had a single dev that was part-time allocated to supporting it. So it's not so much that Docker failed to jump on the k8s bandwagon, so much as they built a wonderful thing that they never marketed and semi-abandoned, to focus on Docker Hub and other core Docker things they thought they'd be able to more directly monetize. Fortunately, Docker Swarm remains a wonderful piece of software, even without particularly active investment, because it is a fairly complete offering for most use cases. And it seems to be growing in adoption.

bovermyer 5y ago
I was dismayed when Docker effectively (and quietly) declared that they were no longer supporting Docker Swarm. Docker Swarm remains far simpler to set up, run, and troubleshoot than Kubernetes. I just can't choose it over K8s, though, knowing that it's unsupported.
- sgk284 5y ago
  I've hemmed and hawed over this as well, but I believe that sometimes software can be complete and stable and not need constant iteration. As long as they add security patches and don't remove it from Docker, I'll still default to Swarm. My fallback, if Swarm is ever removed, is Hashicorp's Nomad. I've only used it in a hobby capacity, but is (at face value, at least) another wonderful piece of software. I've used k8s a bunch in production (including right now at my current employer) and just can't recommend it – it's a technical distraction for most teams.
  - bovermyer 5y ago
    I've never looked into Nomad. Your comment's gotten me interested now.
    - zenlikethat 5y ago
      It’s awesome. Works just as well on one node as in a cluster, and has Docker drivers and everything. Give it a whirl.
MrOwen 5y ago
Why only Google problems? Why not homelab problems where I'm managing upwards of 20 apps and I don't want to come up with custom automation to do things like cert renewal, tls termiantion, dynamic storage, and disaster recovery? What if I don't want to come up with my own system for organizing and grouping resources? Lots of homelab people (I assume) chain together hacky scripts and probably many things are not repeatable. With k8s, all my configs are in a single repo and I can spin back up my entire cluster on new hardware anywhere provided I've backed up my storage. How does Swarm handle all of these problems in a more unified or complete way that k8s does?
- sgk284 5y ago
  I'd heavily encourage you to read the docs. If the scenarios you outlined are your top concern for your homelab, I suspect the complexity and resource overhead of k8s isn't warranted. - Cert Renewal / Rotation: https://docs.docker.com/engine/reference/commandline/swarm_ca/ https://docs.docker.com/engine/reference/commandline/swarm_c... - TLS Termination: Arguably this is an application layer concern and out of scope for an orchestration layer, but in either case use Nginx (or your favorite TLS terminating proxy) as your ingress point. This is equivalent to the Nginx ingress controller in k8s. - Storage: Docker volumes (local or ceph, for distributed). - Disaster Recovery: Node failure? Cluster failure? Data loss? What kind of DR? The strategies you'd use with Swarm will be very similar to those you'd use with k8s. - Grouping resources: Labels and/or services. In a Swarm world you'll have fewer moving pieces, far lower resource requirements, and will manage it all in a docker-compose file(s). You'll also get things like service discovery, routing, and secrets management for "free". No additional configuration needed.
  - MrOwen 5y ago
    I would argue the myriad of alternatives you've suggested as alternatives produce significantly more overhead. You linked Swarm CA renewal operations; this is maybe 1/50 of the functionality of cert-manager. Is there a product like cert-manager that I can run on Swarm that's tightly integrated? Why not have ingress intrinsically linked to the underlying reaources like k8s does? Your suggestion means that someone needs to come up with a methodology and automation to update ingress with Swarm. Yeah, it's obviously possible but imo, Traefik ingress reaources are way easier to manage with config spread across far fewer places. Storage.. I have an nfs storage provider in k8s which allows me to dynamically create storage with lifecycle management included... for free... just because I'm using k8s. DR as in something happens to my house or other technical issues which bring down my cluster. I would posit that it's much easier for me to restore a cluster because it's all yaml and imo more descriptive and capable than swarm yaml syntax. I think you get my point. Yes, you can do it all with Swarm but with things like k3s and k0s, it's just as easy to start using k8s. And way more help, docs, and tooling to boot.
ritchiea 5y ago
One of the tough lessons of my years as a developer is many developers love to adopt Google scale solutions and Google practices because they’re from Google and they come with prestige and a feeling of proximity to the most visible technology company. Whether their problems are a good fit for Google scale solutions or not. It’s inevitable and it takes a lot of soft skills to argue against even if it’s dead obvious you’re going to be running a few instances of an application maximum. The simple solution is far less sexy than the Google solution.
manojlds 5y ago
>K8S is great if you have Google problems Borg is great if you have Google problems. K8s is not Borg.
danudey 5y ago
The dream is to be able to take a software and yeet it into the cloud with minimal effort. That's not kubernetes at all. That said, if your tooling or products or software supports kubernetes, which a lot more are these days, then it becomes a lot easier. A lot of companies these days are saying "Here's how to install it, or just grab this helm chart"; Gitlab, for example. We even had one piece of software we were testing which only had two options: cloud-hosted, or self-hosted on k8s. Nothing else was supported. k8s doesn't seem to be any easier to configure, with the possible exception of microk8s (which has its own issues), but the amount of things you can do with it once it's up and running is compelling.
ljm 5y ago
Unfortunately for Docker, I don't trust or want to use Swarm because of their attempts to monetize and because of the frequency that they push breaking bugs or breaking changes into production releases. To the extent that they made downgrading a paid option, not upgrading. And I'm in the process of switching everything I've got over to Podman now. Swarm might be great, but the ecosystem around k8s is massive. So my two choices are: ignore Docker entirely and do it like I've always done (Heroku, chef, puppet, etc. just managing my servers myself), or buy into containers and go with K8S. To that extent, I only ever deal with K8S at work, and for my personal stuff I don't go near Docker at all. Why would I when I can get an 8GB/4CPU VPS for $9/mo and serve everything I care about from that? I've got fail2ban, systemd, caddy, and plenty of resources. I can host mysql on the same machine because there isn't a god-given law about hosting your DB on a separate box. Same with anything else. No need for Docker or Swarm or K8s.
- da39a3ee 5y ago
  > because of their attempts to monetize Did you read the article? It was about a company struggling to monetize an open source project. It hardly seems time for teenagerish M$-Windows asides on how you don't like people monetizing the software they are giving you for free.
  - p1necone 5y ago
    There's an implicit "in a way that was hostile to their customers/the docker community" in there.
    - da39a3ee 5y ago
      What was implicit was the view that any "attempt to monetize" is hostile. Which, as I said, is something one associates with Linux hobbyists from the early 2000s.
- BackBlast 5y ago
  Who do you use for your VPS provider? Those specs for that price are much better than what I typically see price wise on DigitalOcean, Linode, Vultr, AWS, etc. I ignore docker myself. It's been too unstable for me and historically just costs me time, I'd rather just use systemd on a VPS. Occasionally when someone else has setup a complex system in docker I want to play with it has been convenient. Still, I avoid installing it at all.
orthoxerox 5y ago
The problem with solutions bigger than `docker-compose up -d` and smaller than k8s is that they have a growth ceiling, real or imaginary. Sooner or later you will hit some limitation and will either thrash against it or have to upgrade to k8s. As a decision maker in an IT org, you can either: - pick up Swarm or Nomad now, and then potentially be blamed for the expensive and costly migration to k8s - pick up k8s now, waste the time of your (voiceless) devs and ops on an overengineered solution, but retain access to the larger pool of technologies and a smooth upgrade path should the need arise
- jen20 5y ago
  > smaller than k8s is that they have a growth ceiling Kubernetes has a serious growth ceiling - way below what Nomad has. So if you actually have a problem of interesting scale, it's almost certainly "pick Nomad now, or get blamed for a costly migration later when you realise that Kubernetes is not for you".
- tekromancr 5y ago
  And that is a sound solution if you have limitless VC bux to burn while you implement your over-engineered solution, but most shops have to be a little bit scrappier. Not to mention that you end up scaling to a size where k8s is a sound solution, that "costly" migration is going to be a rounding error on your balance sheet
  - popinman322 5y ago
    > that "costly" migration is going to be a rounding error on your balance sheet This. Just because your services are containerized doesn't mean they'll scale well with each order of magnitude. Likely you'll need to re-architect other parts of the system to work at your new scale.
- sgk284 5y ago
  You've posed a false dichotomoy. I've worked at multiple billion dollar startups that got by just fine without k8s. And I've worked at startups that started on completely different stacks and migrated to k8s. If you're already containerized, you've reserved a good amount of optionality. The kind of reasoning you pose is similar to "SQL doesn't scale to infinity, so start with an infinitely scalable eventually consistent document store". This line of thinking is dangerous to most companies, and often the death of startups. Assume YAGNI. There's some truth to "Nobody ever got fired for choosing IBM", but IBM was never a good choice for anyone except consultants. k8s isn't too far from that.
ghaff 5y ago
Pretty much. Fundamentally Kubernetes “won.” Try to pitch your company on something else and you’re asking them to bet on something else “without particularly active investment” to use your words. As well as much less of a complementary software ecosystem.
dirkg 5y ago
IMO the complexity of k8s tends to be overstated. Of course its a vast and complex system, but its also incredibly well documented and based on simple design principles. The reason people have such a hard time with it is - - deploying k8s on bare metal is not easy. It was never really designed for this and to be fair Google has a vested interest in making you use GCP. EKS is nothing more than cobbled together scripts and you still need to do a lot of manual work - k8s introduces a ton of terminology and new concepts (pods/service/deployment/ingress) not to mention the whole host of addons built on top like Istio. It shouldn't take any decent dev more than a couple days to install minikube/k3s or use a cloud provider, play around and get comfortable with k8s.
- bashinator 5y ago
  Coming from a long career in ops and systems administration, k8s was very straightforward to learn. All the complexity is in automating the things that I’ve already had to know how to do - scaling, failover, resource planning, etc etc.
  - fiddlerwoaroof 5y ago
    Yeah, this stuff is complicated any way you go: puppet, chef, ansible, Cloudformation/CDK, Terraform, K8s all have non-zero learning curves.
    - pknomad 5y ago
      You're trading for structured environment at the cost of complexity for sure, but I think there's something to be said about the degree of complexity. Ansible for example only requires python + some handful library + ssh connection whereas Chef/Puppet requires agent install + SSL for correct functionality.
- sqqqqrly 5y ago
  See k3s.io. Cannot get much easier.
zaat 5y ago
K8S based on distros like k3s or microk8s is a great and simple solution for manufacturing and managing edge devices, IoT solutions and similar products that are setup once on a single box that runs half a dozen or more things and are shipped to business clients. It is very simple to spin a minimal distro based cluster. It is relatively simple to have it up and running your solution. Sure getting Kubernetes from the source and tailoring all the required components to run cluster with full HA etc' is a complex stuff, but building a private cloud based on vanilla Kubernetes isn't the only or the most common use case.
- brazzledazzle 5y ago
  I've also seen more of it being used for deployments of on-prem versions of SaaS services. I think the idea is that if you're already hosting your SaaS services on k8s distributing a machine image to your Enterprise customers running a scaled down deployment of your services on a single node kube cluster makes some of the packaging and updating challenges easier.
ajmurmann 5y ago
I suspect that there is a tendency to pick more powerful and complex tools and infrastructure than needed. For us as engineers it's easy to foresee some of the scaling-related, technical problems of the far future. It's so tempting to prepare for those, especially if you need to learn a new technology anyways and it looks great on your resume. In addition everyone knows intellectually that their own startup is unlikely to reach Google scale, but we do badly want to believed it will. So maybe the right bet is to always put your money on the more powerful and complex technology as long as the complexity is inherent and not incidental. Of course this is also limited to infrastructure. Consumer products and other markets would be totally different

by sgk284 5y ago

bovermyer 5y ago

I was dismayed when Docker effectively (and quietly) declared that they were no longer supporting Docker Swarm. Docker Swarm remains far simpler to set up, run, and troubleshoot than Kubernetes. I just can't choose it over K8s, though, knowing that it's unsupported.

sgk284 5y ago
I've hemmed and hawed over this as well, but I believe that sometimes software can be complete and stable and not need constant iteration. As long as they add security patches and don't remove it from Docker, I'll still default to Swarm. My fallback, if Swarm is ever removed, is Hashicorp's Nomad. I've only used it in a hobby capacity, but is (at face value, at least) another wonderful piece of software. I've used k8s a bunch in production (including right now at my current employer) and just can't recommend it – it's a technical distraction for most teams.
- bovermyer 5y ago
  I've never looked into Nomad. Your comment's gotten me interested now.
  - zenlikethat 5y ago
    It’s awesome. Works just as well on one node as in a cluster, and has Docker drivers and everything. Give it a whirl.

MrOwen 5y ago

Why only Google problems? Why not homelab problems where I'm managing upwards of 20 apps and I don't want to come up with custom automation to do things like cert renewal, tls termiantion, dynamic storage, and disaster recovery? What if I don't want to come up with my own system for organizing and grouping resources? Lots of homelab people (I assume) chain together hacky scripts and probably many things are not repeatable. With k8s, all my configs are in a single repo and I can spin back up my entire cluster on new hardware anywhere provided I've backed up my storage. How does Swarm handle all of these problems in a more unified or complete way that k8s does?

sgk284 5y ago
I'd heavily encourage you to read the docs. If the scenarios you outlined are your top concern for your homelab, I suspect the complexity and resource overhead of k8s isn't warranted. - Cert Renewal / Rotation: https://docs.docker.com/engine/reference/commandline/swarm_ca/ https://docs.docker.com/engine/reference/commandline/swarm_c... - TLS Termination: Arguably this is an application layer concern and out of scope for an orchestration layer, but in either case use Nginx (or your favorite TLS terminating proxy) as your ingress point. This is equivalent to the Nginx ingress controller in k8s. - Storage: Docker volumes (local or ceph, for distributed). - Disaster Recovery: Node failure? Cluster failure? Data loss? What kind of DR? The strategies you'd use with Swarm will be very similar to those you'd use with k8s. - Grouping resources: Labels and/or services. In a Swarm world you'll have fewer moving pieces, far lower resource requirements, and will manage it all in a docker-compose file(s). You'll also get things like service discovery, routing, and secrets management for "free". No additional configuration needed.
- MrOwen 5y ago
  I would argue the myriad of alternatives you've suggested as alternatives produce significantly more overhead. You linked Swarm CA renewal operations; this is maybe 1/50 of the functionality of cert-manager. Is there a product like cert-manager that I can run on Swarm that's tightly integrated? Why not have ingress intrinsically linked to the underlying reaources like k8s does? Your suggestion means that someone needs to come up with a methodology and automation to update ingress with Swarm. Yeah, it's obviously possible but imo, Traefik ingress reaources are way easier to manage with config spread across far fewer places. Storage.. I have an nfs storage provider in k8s which allows me to dynamically create storage with lifecycle management included... for free... just because I'm using k8s. DR as in something happens to my house or other technical issues which bring down my cluster. I would posit that it's much easier for me to restore a cluster because it's all yaml and imo more descriptive and capable than swarm yaml syntax. I think you get my point. Yes, you can do it all with Swarm but with things like k3s and k0s, it's just as easy to start using k8s. And way more help, docs, and tooling to boot.

ritchiea 5y ago

One of the tough lessons of my years as a developer is many developers love to adopt Google scale solutions and Google practices because they’re from Google and they come with prestige and a feeling of proximity to the most visible technology company. Whether their problems are a good fit for Google scale solutions or not. It’s inevitable and it takes a lot of soft skills to argue against even if it’s dead obvious you’re going to be running a few instances of an application maximum. The simple solution is far less sexy than the Google solution.

manojlds 5y ago

>K8S is great if you have Google problems Borg is great if you have Google problems. K8s is not Borg.

danudey 5y ago

The dream is to be able to take a software and yeet it into the cloud with minimal effort. That's not kubernetes at all. That said, if your tooling or products or software supports kubernetes, which a lot more are these days, then it becomes a lot easier. A lot of companies these days are saying "Here's how to install it, or just grab this helm chart"; Gitlab, for example. We even had one piece of software we were testing which only had two options: cloud-hosted, or self-hosted on k8s. Nothing else was supported. k8s doesn't seem to be any easier to configure, with the possible exception of microk8s (which has its own issues), but the amount of things you can do with it once it's up and running is compelling.

ljm 5y ago

Unfortunately for Docker, I don't trust or want to use Swarm because of their attempts to monetize and because of the frequency that they push breaking bugs or breaking changes into production releases. To the extent that they made downgrading a paid option, not upgrading. And I'm in the process of switching everything I've got over to Podman now. Swarm might be great, but the ecosystem around k8s is massive. So my two choices are: ignore Docker entirely and do it like I've always done (Heroku, chef, puppet, etc. just managing my servers myself), or buy into containers and go with K8S. To that extent, I only ever deal with K8S at work, and for my personal stuff I don't go near Docker at all. Why would I when I can get an 8GB/4CPU VPS for $9/mo and serve everything I care about from that? I've got fail2ban, systemd, caddy, and plenty of resources. I can host mysql on the same machine because there isn't a god-given law about hosting your DB on a separate box. Same with anything else. No need for Docker or Swarm or K8s.

da39a3ee 5y ago
> because of their attempts to monetize Did you read the article? It was about a company struggling to monetize an open source project. It hardly seems time for teenagerish M$-Windows asides on how you don't like people monetizing the software they are giving you for free.
- p1necone 5y ago
  There's an implicit "in a way that was hostile to their customers/the docker community" in there.
  - da39a3ee 5y ago
    What was implicit was the view that any "attempt to monetize" is hostile. Which, as I said, is something one associates with Linux hobbyists from the early 2000s.
BackBlast 5y ago
Who do you use for your VPS provider? Those specs for that price are much better than what I typically see price wise on DigitalOcean, Linode, Vultr, AWS, etc. I ignore docker myself. It's been too unstable for me and historically just costs me time, I'd rather just use systemd on a VPS. Occasionally when someone else has setup a complex system in docker I want to play with it has been convenient. Still, I avoid installing it at all.

orthoxerox 5y ago

The problem with solutions bigger than `docker-compose up -d` and smaller than k8s is that they have a growth ceiling, real or imaginary. Sooner or later you will hit some limitation and will either thrash against it or have to upgrade to k8s. As a decision maker in an IT org, you can either: - pick up Swarm or Nomad now, and then potentially be blamed for the expensive and costly migration to k8s - pick up k8s now, waste the time of your (voiceless) devs and ops on an overengineered solution, but retain access to the larger pool of technologies and a smooth upgrade path should the need arise

jen20 5y ago
> smaller than k8s is that they have a growth ceiling Kubernetes has a serious growth ceiling - way below what Nomad has. So if you actually have a problem of interesting scale, it's almost certainly "pick Nomad now, or get blamed for a costly migration later when you realise that Kubernetes is not for you".
tekromancr 5y ago
And that is a sound solution if you have limitless VC bux to burn while you implement your over-engineered solution, but most shops have to be a little bit scrappier. Not to mention that you end up scaling to a size where k8s is a sound solution, that "costly" migration is going to be a rounding error on your balance sheet
- popinman322 5y ago
  > that "costly" migration is going to be a rounding error on your balance sheet This. Just because your services are containerized doesn't mean they'll scale well with each order of magnitude. Likely you'll need to re-architect other parts of the system to work at your new scale.
sgk284 5y ago
You've posed a false dichotomoy. I've worked at multiple billion dollar startups that got by just fine without k8s. And I've worked at startups that started on completely different stacks and migrated to k8s. If you're already containerized, you've reserved a good amount of optionality. The kind of reasoning you pose is similar to "SQL doesn't scale to infinity, so start with an infinitely scalable eventually consistent document store". This line of thinking is dangerous to most companies, and often the death of startups. Assume YAGNI. There's some truth to "Nobody ever got fired for choosing IBM", but IBM was never a good choice for anyone except consultants. k8s isn't too far from that.

ghaff 5y ago

Pretty much. Fundamentally Kubernetes “won.” Try to pitch your company on something else and you’re asking them to bet on something else “without particularly active investment” to use your words. As well as much less of a complementary software ecosystem.

dirkg 5y ago

IMO the complexity of k8s tends to be overstated. Of course its a vast and complex system, but its also incredibly well documented and based on simple design principles. The reason people have such a hard time with it is - - deploying k8s on bare metal is not easy. It was never really designed for this and to be fair Google has a vested interest in making you use GCP. EKS is nothing more than cobbled together scripts and you still need to do a lot of manual work - k8s introduces a ton of terminology and new concepts (pods/service/deployment/ingress) not to mention the whole host of addons built on top like Istio. It shouldn't take any decent dev more than a couple days to install minikube/k3s or use a cloud provider, play around and get comfortable with k8s.

bashinator 5y ago
Coming from a long career in ops and systems administration, k8s was very straightforward to learn. All the complexity is in automating the things that I’ve already had to know how to do - scaling, failover, resource planning, etc etc.
- fiddlerwoaroof 5y ago
  Yeah, this stuff is complicated any way you go: puppet, chef, ansible, Cloudformation/CDK, Terraform, K8s all have non-zero learning curves.
  - pknomad 5y ago
    You're trading for structured environment at the cost of complexity for sure, but I think there's something to be said about the degree of complexity. Ansible for example only requires python + some handful library + ssh connection whereas Chef/Puppet requires agent install + SSL for correct functionality.
sqqqqrly 5y ago
See k3s.io. Cannot get much easier.

zaat 5y ago

K8S based on distros like k3s or microk8s is a great and simple solution for manufacturing and managing edge devices, IoT solutions and similar products that are setup once on a single box that runs half a dozen or more things and are shipped to business clients. It is very simple to spin a minimal distro based cluster. It is relatively simple to have it up and running your solution. Sure getting Kubernetes from the source and tailoring all the required components to run cluster with full HA etc' is a complex stuff, but building a private cloud based on vanilla Kubernetes isn't the only or the most common use case.

brazzledazzle 5y ago
I've also seen more of it being used for deployments of on-prem versions of SaaS services. I think the idea is that if you're already hosting your SaaS services on k8s distributing a machine image to your Enterprise customers running a scaled down deployment of your services on a single node kube cluster makes some of the packaging and updating challenges easier.

ajmurmann 5y ago

I suspect that there is a tendency to pick more powerful and complex tools and infrastructure than needed. For us as engineers it's easy to foresee some of the scaling-related, technical problems of the far future. It's so tempting to prepare for those, especially if you need to learn a new technology anyways and it looks great on your resume. In addition everyone knows intellectually that their own startup is unlikely to reach Google scale, but we do badly want to believed it will. So maybe the right bet is to always put your money on the more powerful and complex technology as long as the complexity is inherent and not incidental. Of course this is also limited to infrastructure. Consumer products and other markets would be totally different