6 ms·

The problem with solutions bigger than `docker-compose up -d` and smaller than k8s is that they have a growth ceiling, real or imaginary. Sooner or later you wi

The problem with solutions bigger than `docker-compose up -d` and smaller than k8s is that they have a growth ceiling, real or imaginary. Sooner or later you will hit some limitation and will either thrash against it or have to upgrade to k8s. As a decision maker in an IT org, you can either: - pick up Swarm or Nomad now, and then potentially be blamed for the expensive and costly migration to k8s - pick up k8s now, waste the time of your (voiceless) devs and ops on an overengineered solution, but retain access to the larger pool of technologies and a smooth upgrade path should the need arise

jen20 5y ago
> smaller than k8s is that they have a growth ceiling Kubernetes has a serious growth ceiling - way below what Nomad has. So if you actually have a problem of interesting scale, it's almost certainly "pick Nomad now, or get blamed for a costly migration later when you realise that Kubernetes is not for you".
tekromancr 5y ago
And that is a sound solution if you have limitless VC bux to burn while you implement your over-engineered solution, but most shops have to be a little bit scrappier. Not to mention that you end up scaling to a size where k8s is a sound solution, that "costly" migration is going to be a rounding error on your balance sheet
- popinman322 5y ago
  > that "costly" migration is going to be a rounding error on your balance sheet This. Just because your services are containerized doesn't mean they'll scale well with each order of magnitude. Likely you'll need to re-architect other parts of the system to work at your new scale.
sgk284 5y ago
You've posed a false dichotomoy. I've worked at multiple billion dollar startups that got by just fine without k8s. And I've worked at startups that started on completely different stacks and migrated to k8s. If you're already containerized, you've reserved a good amount of optionality. The kind of reasoning you pose is similar to "SQL doesn't scale to infinity, so start with an infinitely scalable eventually consistent document store". This line of thinking is dangerous to most companies, and often the death of startups. Assume YAGNI. There's some truth to "Nobody ever got fired for choosing IBM", but IBM was never a good choice for anyone except consultants. k8s isn't too far from that.
- dirkg 5y ago
  But you are posing a similar false dichotomy. k8s isn't some massive intractable beast that needs an army of consultants to run and maintain. It can run small loads just fine as well. Anyone can leann k8s basic principles in a day or 2. And managing a cluster is arguably easier than managing a typical bunch of EX2 instances+ALB etc that almost everyone uses. k8s also has the advantage of ubiquity. If I had to manage a bunch of containers on a cluster with > 10 nodes, I'd pick k8s over swarm/nomad etc. Your argument is 'k8s isn't a good choice for anyone except Google-scale' and I don't think thats true at all.
  - GiorgioG 5y ago
    > k8s isn't some massive intractable beast that needs an army of consultants to run and maintain. As someone who has actively avoided k8s (despite enjoying devops), my observation has been quite the opposite. At my current (small) org we have one devops guy who spends all day working on k8s related work (configuration, troubleshooting, automation, etc.) And we're about to start looking for another devops engineer because he's swamped. We only have a handful of clusters up and running so far. k8s is the poster child for overcomplicating matters for folks that don't have Google-scale problems.
    - jrockway 5y ago
      You might be doing it wrong. I use k8s for my personal projects and the maintenance overhead is lighter than what it cost me to log into my VPS every 3 months and renew the Let's Encrypt certificate. (Which was literally just pressing the up arrow and enter, because it was always the last thing in my shell history.) At work, a team of 5 people maintain our production Kubernetes clusters (that host our cloud product and app), and develop the app itself. Which by the way, creates Kubernetes clusters for individual customers, of which we have created thousands of. Overall, it's not that hard and we don't spend much time managing computers. (We do have all of the good stuff, like all K8s changes in Git, one-click deploys, etc. I did spend a little time making Shipit run in Kubernetes, but it certainly wasn't my full time job for any extended period of time.) There are problems that people run into. One is not needing more than one computer -- if everything you need to run fits on one computer, you might as well use systemd as your orchestrator. (Which people also say is overly complicated, but really it's just extremely misunderstood.) The other is not understanding Kubernetes enough. A lot of people just explode random helm charts into their cluster and are mad that it's hard to maintain. Basically, adding another abstraction layer only helps if you can debug the actual thing on the surface -- otherwise, you're just setting a time bomb for yourself. (And, helm charts require you beg the maintainers for every feature you could possibly need, and once enough people have done that you just have raw k8s manifests where everything is called some random word instead of its actual thing. Really bad. Use Kustomize or jsonnet!)

jen20 5y ago

> smaller than k8s is that they have a growth ceiling Kubernetes has a serious growth ceiling - way below what Nomad has. So if you actually have a problem of interesting scale, it's almost certainly "pick Nomad now, or get blamed for a costly migration later when you realise that Kubernetes is not for you".

tekromancr 5y ago

And that is a sound solution if you have limitless VC bux to burn while you implement your over-engineered solution, but most shops have to be a little bit scrappier. Not to mention that you end up scaling to a size where k8s is a sound solution, that "costly" migration is going to be a rounding error on your balance sheet

popinman322 5y ago
> that "costly" migration is going to be a rounding error on your balance sheet This. Just because your services are containerized doesn't mean they'll scale well with each order of magnitude. Likely you'll need to re-architect other parts of the system to work at your new scale.

sgk284 5y ago

You've posed a false dichotomoy. I've worked at multiple billion dollar startups that got by just fine without k8s. And I've worked at startups that started on completely different stacks and migrated to k8s. If you're already containerized, you've reserved a good amount of optionality. The kind of reasoning you pose is similar to "SQL doesn't scale to infinity, so start with an infinitely scalable eventually consistent document store". This line of thinking is dangerous to most companies, and often the death of startups. Assume YAGNI. There's some truth to "Nobody ever got fired for choosing IBM", but IBM was never a good choice for anyone except consultants. k8s isn't too far from that.

dirkg 5y ago
But you are posing a similar false dichotomy. k8s isn't some massive intractable beast that needs an army of consultants to run and maintain. It can run small loads just fine as well. Anyone can leann k8s basic principles in a day or 2. And managing a cluster is arguably easier than managing a typical bunch of EX2 instances+ALB etc that almost everyone uses. k8s also has the advantage of ubiquity. If I had to manage a bunch of containers on a cluster with > 10 nodes, I'd pick k8s over swarm/nomad etc. Your argument is 'k8s isn't a good choice for anyone except Google-scale' and I don't think thats true at all.
- GiorgioG 5y ago
  > k8s isn't some massive intractable beast that needs an army of consultants to run and maintain. As someone who has actively avoided k8s (despite enjoying devops), my observation has been quite the opposite. At my current (small) org we have one devops guy who spends all day working on k8s related work (configuration, troubleshooting, automation, etc.) And we're about to start looking for another devops engineer because he's swamped. We only have a handful of clusters up and running so far. k8s is the poster child for overcomplicating matters for folks that don't have Google-scale problems.
  - jrockway 5y ago
    You might be doing it wrong. I use k8s for my personal projects and the maintenance overhead is lighter than what it cost me to log into my VPS every 3 months and renew the Let's Encrypt certificate. (Which was literally just pressing the up arrow and enter, because it was always the last thing in my shell history.) At work, a team of 5 people maintain our production Kubernetes clusters (that host our cloud product and app), and develop the app itself. Which by the way, creates Kubernetes clusters for individual customers, of which we have created thousands of. Overall, it's not that hard and we don't spend much time managing computers. (We do have all of the good stuff, like all K8s changes in Git, one-click deploys, etc. I did spend a little time making Shipit run in Kubernetes, but it certainly wasn't my full time job for any extended period of time.) There are problems that people run into. One is not needing more than one computer -- if everything you need to run fits on one computer, you might as well use systemd as your orchestrator. (Which people also say is overly complicated, but really it's just extremely misunderstood.) The other is not understanding Kubernetes enough. A lot of people just explode random helm charts into their cluster and are mad that it's hard to maintain. Basically, adding another abstraction layer only helps if you can debug the actual thing on the surface -- otherwise, you're just setting a time bomb for yourself. (And, helm charts require you beg the maintainers for every feature you could possibly need, and once enough people have done that you just have raw k8s manifests where everything is called some random word instead of its actual thing. Really bad. Use Kustomize or jsonnet!)