8 ms·

Right tool for the right job. Is K8s too complicated? For some use cases it is. They probably should do a better job of discouraging certain use cases, but cal

Right tool for the right job. Is K8s too complicated? For some use cases it is. They probably should do a better job of discouraging certain use cases, but calling their elevator pitch “bullshit” is hyperbolic. There are exceptions to every rule, but a good rule of thumb is cluster size. If you’re managing less than 25 servers than K8s is probably over kill. As you start to creep north of 40 servers K8s really starts to shine. The other place K8s really shines is dynamic load. I manage anywhere from 600-1000 16Gb VMs, I can’t imagine doing it without K8s. If cluster size isn’t a good rule of thumb then application architectur probably is. If no one person in you company has a complete mental model of the application architecture or it is impossible because it is so complex, again container orchestration might be a good way to go. Final point: If you’re struggling with K8s swallow your pride and buy a managed solution, then learn as you go.

pstadler 8y ago
Even though I fully agree with you, running a little 3 node cluster just for fun is amazing. Thanks to Rook and an Nginx ingress controller with kube-lego, I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session. This, in my point of view, is absolutely powerful. Shameless plug, I‘ve been working on a project explaining how to run small scale clusters in the cloud for more than a year: https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide
- ad_hominem 8y ago
  You may want to update your tutorial to use cert-manager because kube-lego is in maintenance-only mode
  - Filligree 8y ago
    You mean, it's stable. That sounds like a good reason to use it, not to avoid it.
    - raesene9 8y ago
      From the Kube-lego github page, it looks like the last version of k8s is supports is 1.8, which will only be supported until 1.11 comes out. So sounds more like it's deprecated than stable.
    - kuschku 8y ago
      It doesn’t support ACMEv2, Wildcard certs, and more. And Certmanager is alpha. The alternatives are all horrible.
    - ad_hominem 8y ago
      Not really; I had an issue that silently broke renewals. When I found the open issue that corresponded to it the maintainers were herding people to cert-manager. There are a lot of issues where they are doing that (besides very obviously at the top of the README with warning symbols). At the very least it's eventually going to break on newer versions of Kubernetes. Maintenance-only != LTS.
      - eric_h 8y ago
        Maintenance-only usually means deprecated. There are some areas in the software world where maintenance-only can last a long time and can be de facto LTS. The Kubernetes ecosystem is not one of them.
    - megaman22 8y ago
      Depends on where you come from. In JS world, stable means there's a fork in it, and it's grayer than last weeks' meat. In server-side world, that means you're good to build a business on it for 15 years
  - vorpalhex 8y ago
    cert-manager is marked as alpha/non-production currently.
    - s_kilk 8y ago
      (Off Topic) interesting that we're seeing two radically different philosophies of software development in this thread, in stark contrast.
    - ad_hominem 8y ago
      So is kube-lego.
  - pstadler 8y ago
    Thanks for the hint, I didn‘t know that. Related issue: https://github.com/hobby-kube/guide/issues/47 https://github.com/hobby-kube/guide/issues/47
- wpietri 8y ago
  Ooh, thanks, this is just what I've been looking for.
- inteleng 8y ago
  > I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session Is it grammatically possible to use more buzzwords in a single sentence? Reading that made me want to vomit.
  - vectorpush 8y ago
    The words have meaning. If you don't understand them you could just ask for them to be explained rather than throwing out insults. I suspect you actually do understand the meaning of what was written though so I suppose that means you're just trying to start a flamewar.
    - StavrosK 8y ago
      The problem is that "leveraging" can be replaced with "using" every time, with the added benefit that it won't leave a bad taste in the reader's mouth.
      - inteleng 8y ago
        This is what I mean.
- lowbloodsugar 8y ago
  You: Look what I did with k8s Top voted reply: You had to use a thing to do what you want, and that thing isn't even supported any more. [Either I was hallucinating at the time and imagined it, or there was a reply that got deleted] I think you've just quite nicely demonstrated the original authors point.
  - pstadler 8y ago
    You sound like you‘re new to this business.
    - lowbloodsugar 8y ago
      Sometimes the people "new to this business" have the best ideas. It's more likely that I'm a grumpy old man worried about his lawn.
- Pneumaticat 8y ago
  How did I not find this one month ago, as I set up a two node hobby cluster?!
- bogomipz 8y ago
  >"Rook and an Nginx ingress controller with kube-lego" Can you elaborate on this configuration, specifically Root and Kube-lego? I am not familiar with those.
- bruinjoe 8y ago
  Thanks for sharing about hobby-kube.
cube2222 8y ago
I think it's actually good as soon as you're doing microservices and don't want to use any managed vendor lock in service like app engine. The setup is easy if you use a managed kubernetes offering and it provides so many things out of the box you'd otherwise have to take care of yourself. Service discovery, service lifecycles, updates, storage management, logs, load balancing (inter-service), all those things you should automate even if you have 5 servers, and k8s makes this darn easy in my opinion. The declarative abstraction is also very easy to use and intuitive in my opinion.
- nickbauman 8y ago
  I routinely take app engine apps and run them on the FOSS AppScale platform unchanged. For the things you can't port, like using Google as your auth, there are 3rd party solutions for that already. Choosing Google's App Engine is more than just choosing a managed solution, too. You choose it because of Google's reach: consider, for a moment, how you send a push notification to an Android device. When you do it from Google's cloud, your metal is co-located with their push infra. Also caching, logging, tracing and even a global CDN is integrated into the platform. Think of the cost of having to sort this yourself.
ownagefool 8y ago
I'd say it's actually just a convention vs configuration argument. Kubernetes isn't giving you anything you couldn't have already built with configuration management. It just happens to a standard written by a bunch of people with a background in the problem domain. Personally, I think the abstractions are thoughtful and the system isn't really inherently more complicated than what you'll eventually build anyways, but as the article said, we all have a bias towards that crap we ourselves invented, because we already know how that works, and we find learning something like kubernetes a chore. That said, I wouldn't say it's about how many servers you manage. If you have a single monolith, written and ran by a single team, it doesn't neccessairly get more complex as you throw more servers at it. But when you have several teams, writing and hosting several systems, it's nicer to have a convention framework instead of an undocumented snowflake platform. I have to agre on your final point though. If you find running kubernetes complicated, go to GKE and treat it like you do your IaaS provider, which is more complicated but isn't something you typically deal with.
- majewsky 8y ago
  > Kubernetes isn't giving you anything you couldn't have already built with configuration management. Configuration management does not give you loadbalancing. Configuration management does not give you rolling upgrades. I mean, sure, you can do this stuff with CM as well, but with k8s, there's nothing to build. It's already there
  - deleted 8y ago
    [deleted]
  - toomuchtodo 8y ago
    You're trading operational complexity in other areas for arguable operational functionality that CM of haproxy (or perhaps Microsoft's load balancer if you need UDP LB services) already provides for, in software that is still under active development and has sharp edges. Your systems should be as simple as possible, but no more simple than that. Kubernetes is not turnkey and requires a significant operational time and resource commitment. Prepare accordingly.
  - 8y ago
hinkley 8y ago
Docker compose can handle simple container situations and many PoCs just fine, and part of the work will be analogous to the Kubernetes solution. If all you have is a handful of load balanced web servers and a database you can hold off on kube for a while.
- pm90 8y ago
  The feature gap between compose and k8s is starting to add up though. Compose is nice for tutorials and POC within a containerized environment but IMO not that good for full-fledged production.
znpy 8y ago
Interesting, what do you think should be used to manage a cluster of less than 25 machines?
- hacknat 8y ago
  It wasn’t a hard rule. It depend on the app and the use case, but probably just docker on all the machines with some static cloud loadbalancers in between everything. Or just straight config management.
- akvadrako 8y ago
  Hosted k8s. all the benefits with almost no extra maintenance.
  - Filligree 8y ago
    Okay, how about for a tiny budget?
    - zie 8y ago
      Nomad (www.nomadproject.io) is what I would recommend. Nomad is ONLY a task scheduler across a cluster of machines. Which is why it’s not rocket science, nor is it operationally complex. You say I need X cpu and X memory and I need these files out on disk(or this docker image) and run this command. It will enforce your task gets exactly X memory, X cpu and X disk, so you can’t over-provision at the task level, Docker and friends don't do this. It handles batch(i.e. cron) and spark workloads, system jobs(run on every node) and services (any long-running task). For instance with nomad batch jobs you can almost entirely replace Celery and other distributed task queues, in a platform and language agnostic way! You can do load balancing and all the other things k8s does, but you use specialized tools for these things: * For HTTPS traffic you can use Fabio, Traefik, HAProxy, Nginx, etc. * For TCP traffic you can use Fabio, Relayd, etc. These are outside of Nomad’s scope, except that you can run those jobs inside of Nomad just fine. and it’s all declarative, a total win.
      - 8y ago
sidlls 8y ago
If no one person in you company has a complete mental model of the application architecture or it is impossible because it is so complex, One caveat to this: architectures that are complex because of poor design should probably be redone rather than hidden behind yet another layer of complexity.
yeukhon 8y ago
Less than 25 servers is not overall when you take into account people generally choose a larger instance type.
MrBuddyCasino 8y ago
K8s is not the only game in town. The question is not k8s vs. old-school ssh and apt-get, but rather k8s vs. Hashicorp Nomad or AWS Fargate. For a few microservices, Nomad is really nice and simple.
- tannhaeuser 8y ago
  Or mesos/marathon, though I'm in no way qualified to make comparisons vs k8s.
  - mdekkers 8y ago
    Or mesos/marathon, though I'm in no way qualified to make comparisons vs k8s. I am qualified to make that comparison, and Mesosphere all the way. Rapid, no bullshit deploy, not having to deal with key stack components being alpha, simple management, wide support (including running k8s on mesos if you are so inclined). There is a very long list of why mesosphere can be considered to be a stable and mature product. Every time I work with k8s I think "this is cool and all, but absolutely not ready for production use" - and in my book production use means not needing a small army of sysadmins (sorry, SME's) to keep it running, not changing core components every change of the moonphase, the majority of core components being stable and battle-tested, and a learning curve that doesn't look like a altitude-vs-time plot of the latest SpaceX launch.
oso2k 8y ago
Or find someone to train you. The Kubernetes stack is deep. There's no shame in being taught this stuff. Nobody is born already knowing all the nuances. As an OpenShift consultant for Red Hat, in a single day, I've run a tcpdump trace to figure out why a particular NFS vendor had trouble with traffic over a new network vlan, then, spent an hour or two with a Dev in ChromeDevTools to help them get session management cookies straightened out. Some people call that a day spent as a "Full Stack Engineer", I call that Tuesday. And I can't tell you how many obscure corner case scenarios I've found bugs in Kubernetes, OpenShift or a client's App.
akavel 8y ago
What good alternative can you suggest for less than 25 servers?
- Axsuul 8y ago
  Docker Swarm
- mdekkers 8y ago
  Mesosphere
sneak 8y ago
> Final point: If you’re struggling with K8s swallow your pride and buy a managed solution, then learn as you go. Can you make vendor recommendations, please?

pstadler 8y ago

Even though I fully agree with you, running a little 3 node cluster just for fun is amazing. Thanks to Rook and an Nginx ingress controller with kube-lego, I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session. This, in my point of view, is absolutely powerful. Shameless plug, I‘ve been working on a project explaining how to run small scale clusters in the cloud for more than a year: https://github.com/hobby-kube/guide https://github.com/hobby-kube/guide

ad_hominem 8y ago
You may want to update your tutorial to use cert-manager because kube-lego is in maintenance-only mode
- Filligree 8y ago
  You mean, it's stable. That sounds like a good reason to use it, not to avoid it.
  - raesene9 8y ago
    From the Kube-lego github page, it looks like the last version of k8s is supports is 1.8, which will only be supported until 1.11 comes out. So sounds more like it's deprecated than stable.
  - kuschku 8y ago
    It doesn’t support ACMEv2, Wildcard certs, and more. And Certmanager is alpha. The alternatives are all horrible.
  - ad_hominem 8y ago
    Not really; I had an issue that silently broke renewals. When I found the open issue that corresponded to it the maintainers were herding people to cert-manager. There are a lot of issues where they are doing that (besides very obviously at the top of the README with warning symbols). At the very least it's eventually going to break on newer versions of Kubernetes. Maintenance-only != LTS.
    - eric_h 8y ago
      Maintenance-only usually means deprecated. There are some areas in the software world where maintenance-only can last a long time and can be de facto LTS. The Kubernetes ecosystem is not one of them.
  - megaman22 8y ago
    Depends on where you come from. In JS world, stable means there's a fork in it, and it's grayer than last weeks' meat. In server-side world, that means you're good to build a business on it for 15 years
- vorpalhex 8y ago
  cert-manager is marked as alpha/non-production currently.
  - s_kilk 8y ago
    (Off Topic) interesting that we're seeing two radically different philosophies of software development in this thread, in stark contrast.
  - ad_hominem 8y ago
    So is kube-lego.
- pstadler 8y ago
  Thanks for the hint, I didn‘t know that. Related issue: https://github.com/hobby-kube/guide/issues/47 https://github.com/hobby-kube/guide/issues/47
wpietri 8y ago
Ooh, thanks, this is just what I've been looking for.
inteleng 8y ago
> I’m able to deploy applications leveraging distributed storage and getting tls secured endpoints without a single ssh session Is it grammatically possible to use more buzzwords in a single sentence? Reading that made me want to vomit.
- vectorpush 8y ago
  The words have meaning. If you don't understand them you could just ask for them to be explained rather than throwing out insults. I suspect you actually do understand the meaning of what was written though so I suppose that means you're just trying to start a flamewar.
  - StavrosK 8y ago
    The problem is that "leveraging" can be replaced with "using" every time, with the added benefit that it won't leave a bad taste in the reader's mouth.
    - inteleng 8y ago
      This is what I mean.
lowbloodsugar 8y ago
You: Look what I did with k8s Top voted reply: You had to use a thing to do what you want, and that thing isn't even supported any more. [Either I was hallucinating at the time and imagined it, or there was a reply that got deleted] I think you've just quite nicely demonstrated the original authors point.
- pstadler 8y ago
  You sound like you‘re new to this business.
  - lowbloodsugar 8y ago
    Sometimes the people "new to this business" have the best ideas. It's more likely that I'm a grumpy old man worried about his lawn.
Pneumaticat 8y ago
How did I not find this one month ago, as I set up a two node hobby cluster?!
bogomipz 8y ago
>"Rook and an Nginx ingress controller with kube-lego" Can you elaborate on this configuration, specifically Root and Kube-lego? I am not familiar with those.
bruinjoe 8y ago
Thanks for sharing about hobby-kube.

cube2222 8y ago

I think it's actually good as soon as you're doing microservices and don't want to use any managed vendor lock in service like app engine. The setup is easy if you use a managed kubernetes offering and it provides so many things out of the box you'd otherwise have to take care of yourself. Service discovery, service lifecycles, updates, storage management, logs, load balancing (inter-service), all those things you should automate even if you have 5 servers, and k8s makes this darn easy in my opinion. The declarative abstraction is also very easy to use and intuitive in my opinion.

nickbauman 8y ago
I routinely take app engine apps and run them on the FOSS AppScale platform unchanged. For the things you can't port, like using Google as your auth, there are 3rd party solutions for that already. Choosing Google's App Engine is more than just choosing a managed solution, too. You choose it because of Google's reach: consider, for a moment, how you send a push notification to an Android device. When you do it from Google's cloud, your metal is co-located with their push infra. Also caching, logging, tracing and even a global CDN is integrated into the platform. Think of the cost of having to sort this yourself.

ownagefool 8y ago

I'd say it's actually just a convention vs configuration argument. Kubernetes isn't giving you anything you couldn't have already built with configuration management. It just happens to a standard written by a bunch of people with a background in the problem domain. Personally, I think the abstractions are thoughtful and the system isn't really inherently more complicated than what you'll eventually build anyways, but as the article said, we all have a bias towards that crap we ourselves invented, because we already know how that works, and we find learning something like kubernetes a chore. That said, I wouldn't say it's about how many servers you manage. If you have a single monolith, written and ran by a single team, it doesn't neccessairly get more complex as you throw more servers at it. But when you have several teams, writing and hosting several systems, it's nicer to have a convention framework instead of an undocumented snowflake platform. I have to agre on your final point though. If you find running kubernetes complicated, go to GKE and treat it like you do your IaaS provider, which is more complicated but isn't something you typically deal with.

majewsky 8y ago
> Kubernetes isn't giving you anything you couldn't have already built with configuration management. Configuration management does not give you loadbalancing. Configuration management does not give you rolling upgrades. I mean, sure, you can do this stuff with CM as well, but with k8s, there's nothing to build. It's already there
- deleted 8y ago
  [deleted]
- toomuchtodo 8y ago
  You're trading operational complexity in other areas for arguable operational functionality that CM of haproxy (or perhaps Microsoft's load balancer if you need UDP LB services) already provides for, in software that is still under active development and has sharp edges. Your systems should be as simple as possible, but no more simple than that. Kubernetes is not turnkey and requires a significant operational time and resource commitment. Prepare accordingly.
- 8y ago

hinkley 8y ago

Docker compose can handle simple container situations and many PoCs just fine, and part of the work will be analogous to the Kubernetes solution. If all you have is a handful of load balanced web servers and a database you can hold off on kube for a while.

pm90 8y ago
The feature gap between compose and k8s is starting to add up though. Compose is nice for tutorials and POC within a containerized environment but IMO not that good for full-fledged production.

znpy 8y ago

Interesting, what do you think should be used to manage a cluster of less than 25 machines?

hacknat 8y ago
It wasn’t a hard rule. It depend on the app and the use case, but probably just docker on all the machines with some static cloud loadbalancers in between everything. Or just straight config management.
akvadrako 8y ago
Hosted k8s. all the benefits with almost no extra maintenance.
- Filligree 8y ago
  Okay, how about for a tiny budget?
  - zie 8y ago
    Nomad (www.nomadproject.io) is what I would recommend. Nomad is ONLY a task scheduler across a cluster of machines. Which is why it’s not rocket science, nor is it operationally complex. You say I need X cpu and X memory and I need these files out on disk(or this docker image) and run this command. It will enforce your task gets exactly X memory, X cpu and X disk, so you can’t over-provision at the task level, Docker and friends don't do this. It handles batch(i.e. cron) and spark workloads, system jobs(run on every node) and services (any long-running task). For instance with nomad batch jobs you can almost entirely replace Celery and other distributed task queues, in a platform and language agnostic way! You can do load balancing and all the other things k8s does, but you use specialized tools for these things: * For HTTPS traffic you can use Fabio, Traefik, HAProxy, Nginx, etc. * For TCP traffic you can use Fabio, Relayd, etc. These are outside of Nomad’s scope, except that you can run those jobs inside of Nomad just fine. and it’s all declarative, a total win.
    - 8y ago

sidlls 8y ago

If no one person in you company has a complete mental model of the application architecture or it is impossible because it is so complex, One caveat to this: architectures that are complex because of poor design should probably be redone rather than hidden behind yet another layer of complexity.

yeukhon 8y ago

Less than 25 servers is not overall when you take into account people generally choose a larger instance type.

MrBuddyCasino 8y ago

K8s is not the only game in town. The question is not k8s vs. old-school ssh and apt-get, but rather k8s vs. Hashicorp Nomad or AWS Fargate. For a few microservices, Nomad is really nice and simple.

tannhaeuser 8y ago
Or mesos/marathon, though I'm in no way qualified to make comparisons vs k8s.
- mdekkers 8y ago
  Or mesos/marathon, though I'm in no way qualified to make comparisons vs k8s. I am qualified to make that comparison, and Mesosphere all the way. Rapid, no bullshit deploy, not having to deal with key stack components being alpha, simple management, wide support (including running k8s on mesos if you are so inclined). There is a very long list of why mesosphere can be considered to be a stable and mature product. Every time I work with k8s I think "this is cool and all, but absolutely not ready for production use" - and in my book production use means not needing a small army of sysadmins (sorry, SME's) to keep it running, not changing core components every change of the moonphase, the majority of core components being stable and battle-tested, and a learning curve that doesn't look like a altitude-vs-time plot of the latest SpaceX launch.

oso2k 8y ago

Or find someone to train you. The Kubernetes stack is deep. There's no shame in being taught this stuff. Nobody is born already knowing all the nuances. As an OpenShift consultant for Red Hat, in a single day, I've run a tcpdump trace to figure out why a particular NFS vendor had trouble with traffic over a new network vlan, then, spent an hour or two with a Dev in ChromeDevTools to help them get session management cookies straightened out. Some people call that a day spent as a "Full Stack Engineer", I call that Tuesday. And I can't tell you how many obscure corner case scenarios I've found bugs in Kubernetes, OpenShift or a client's App.

akavel 8y ago

What good alternative can you suggest for less than 25 servers?

Axsuul 8y ago
Docker Swarm
mdekkers 8y ago
Mesosphere

sneak 8y ago

> Final point: If you’re struggling with K8s swallow your pride and buy a managed solution, then learn as you go. Can you make vendor recommendations, please?