6 ms·

Just curious - what's wrong with buying a large instance (24 cores) and running it for < 10,000 users? Kubernetes feels like an insane complexity that doesn't n

Just curious - what's wrong with buying a large instance (24 cores) and running it for < 10,000 users? Kubernetes feels like an insane complexity that doesn't need to be taken on and managed. You're gonna spend more time managing Kubernetes than writing actual software. Also, it feels like if something goes wrong in prod with your cluster - you're gonna need external help to get you back on the feet. If you're not going to build the next Facebook, why would you need so much complexity?

halfmatthalfcat 7y ago
Because it removes both provisioning and deployment concerns when you can build a container and then just tell Kubernetes to deploy it onto the cluster. Theres not much that goes into spinning up these managed Kuberentes clusters. Most of it is telling them what classes of instances you want created. When you buy a large instance, you still need to set up the instance and tweak it to your application's needs. You then need to babysit this node.
- warrenm 7y ago
  >When you buy a large instance, you still need to set up the instance and tweak it to your application's needs. You then need to babysit this node. It's called "use kickstart"
eknkc 7y ago
It is a single point of failure. I want multiple smaller instances even if I’m running something trivial. It is not about scale but about reliability for us. And the moment I go with multiple instances I need something to manage that mess. Kube handles it well.
- spectramax 7y ago
  I've used DigitalOcean's load balancer + multiple instances that run app containers. It works fine and without Kubernetes. I wasn't saying that just use 24 core single instance. Perhaps I should have worded it better.
  - eknkc 7y ago
    Well a managed kubernetes cluster is not any more complex than the other solutions. You can use GCP control panel to launch clusters, add nodes, launch containers expose and autoscale them without touching a single line of config file or a terminal if thats what you like. If you launch / manage your own cluster though.. It’s a pain.
  - closeparen 7y ago
    How do you obtain the right docker images and launch them with the right settings? How do you coordinate rolling deployments? (I'm not saying you need Kubernetes to do these things. But if you've written something to handle them, it is very probably Kubernetes-shaped).
mjfisher 7y ago
> Kubernetes feels like an insane complexity There's two things there - the complexity of engineering the cluster itself, and the complexity of using it. The former is where the pain is. If you can remove that pain by using a reliable managed offering, it changes the perspective a bit. The complexity of using it is also non-trivial - you have to learn it's terminology and primitives, understand the deployment and scheduling model, know how volumes are provisioned, aggregate logs, etc. However, the ROI for learning that complexity can be pretty good. If you get comfortable with it (maybe a month or so of learning and hacking?) you get sane rolling deployment processes by default, process management and auto healing, secrets and config management, scheduled tasks, health checks, and much, much easier infrastructure as code. Which means if things do go really sideways, it's usually not that hard just to stand up a replacement cluster from scratch. With a bit more reading and some additional open source services, you get fully automatic SSL management with let's encrypt too. All that said, I absolutely agree with you in principle. No one should overcomplicate their deployments for no benefit. It's just worth reflecting on what those benefits are. Kubernetes had a bunch of them - and whether they're worth the effort of getting to know it depends very much on the system(s) you're running.
- spectramax 7y ago
  I used to work for a company that had less than 130 employees and none of the software services were exposed to public - the total user count was 130 users. And we had a Kubernetes cluster. Thanks for an insightful response - Don't you think that a lot of wizbang software developers just want to use latest and greatest buzzword thing, whatever that might be, to get some cool points? As I grow older, I see increasing lack of objectivity and decreasing attention to KISS principles.
  - mjfisher 7y ago
    > Don't you think that a lot of wizbang software developers just want to use latest and greatest buzzword thing Yes, absolutely, and it drives me up the wall. I've seen some incredibly unsuitable technology choices made, the complexity of which have absolutely sunk productivity on projects. That said, it's easy to become cynical and associate anything that's become recently popular with hype-driven garbage. But that can blind you to some really great new stuff too. I tend to hang behind the very early adopters and wait to see how useful new tech becomes in the wild - the "a year with X" style blog posts tend to be really informative.
p_l 7y ago
It all depends on how many applications you really have to run. When you have one application, can depend on vendored services for some of the infra, or have sufficiently small requirements for extra services, it all works pretty well. Helps if you also don't do due diligence on various things (like actually caring about logging and monitoring), so you scratch out those concerns. This is not even half sarcastic, people went pretty far with that. You might probably still spend more time than you think on actual deployment and management due to lack of automation & standardization, but if you're sufficiently small on the backend it works. When the number of concerns you have to manage rises and you want to optimize the infrastructure costs, things get weirder. Kubernetes' main offering is NOT scaling. It's providing a standardized control plane you can use to reduce the operational burden of managing often wildly different things that are components in your total infrastructure build - Things like infrastructural metrics and logging, your business metrics and logging, various extra software you might run to take care of stuff, making it easier to do HA, abstract away the underlying servers so you don't have to remember which server hosted which files or how the volumes were mounted (it can be as easy as classic old NFS/CIFS server with a rock-solid disk array, you just don't have to care about mounting the volumes on individual servers). It makes it easier to manage HTTPS routing in my experience (plop an nginx-ingress-controller with a host port, do a basic routing setup to push external HTTP/HTTPS traffic to those nodes that run it, get the bliss to forget about configuring individual Nginx configs anymore --- or use your cloud's Ingress controller, with little configuration difference!). In my experience, k8s was a force multiplier to any kind of infrastructure work, because what used to be much more complex, even with automation tools like Puppet or Chef, now had a way to be nicely forced into packages that even self-arranged themselves on the servers, without me having to care about which server and where. Except for setting up a node on-prem, or very rare debug tasks (that usually can be later rebuilt using k8s apis to get another force-multiplier), I don't have to SSH to servers except maybe for fun. Sometimes I login to specific containers, but that falls under debugging and tweaking the applications I run, not the underlying infrastructure. That's the offering - whether it is right for you, is another matter. Especially if you're in cash-rich SV startup, things like Heroku might be better. For me, the costs of running on PaaS are higher than getting another full-time engineer... and we manage to run k8s part-time.
saber6 7y ago
While I see a lot of derision about Kubernetes these days, if I am starting a Greenfield platform/design/product, why wouldn't I use it? There are tremendous benefits to K8S. It isn't just hype. On the flip side, starting out as a monolithic (all in one VM) app will take significant effort to transition to micro services / K8S. If I think I might end up at microservices/K8S, I think I might as well plan for it (abstractly) initially.
- allset_ 7y ago
  I believe this is the right way to think about it. You can start off with a relatively monolithic architecture, and then break that out into smaller microservices as needed with a much easier transition.
the_other_b 7y ago
If you're not going to build the next Facebook, why would you need so much complexity? You don't. I think this is a recent point people are trying to make. Kubernetes makes sense at a certain scale, but for smaller startups it maybe shouldn't be the go to.
- spectramax 7y ago
  Can you educate on at what point Kubernetes actually makes sense? Is there a rule of thumb?
  - p_l 7y ago
    When you need to run more than 1-2 applications, especially in HA, and you're cost-conscious so just throwing tons of machines in autoscaling groups doesn't work for you. Said applications don't have to be applications that you're writing as part of your startup. It can be ElasticSearch cluster, Redis, tooling to run some kind of ETL that happens to be able to reuse your k8s cluster for task execution, CI/CD, etc. etc.
  - the_other_b 7y ago
    I don't think I'm in a position to answer this (heavily experienced outside the control plane, but not in), but if I had to answer I'd say: Once you don't personally have your hand in every application your company is running (in addition to the points the other comments have brought up).
- DelightOne 7y ago
  So if Kubernetes is too complex, then terraform is a Nono too? I don't find them complex at all. You just tell the tools to be in a specific state and the tool applies the necessary changes. Server templates. Provisioning. Orchestration. etc.
  - the_other_b 7y ago
    I don't think theres a comparison there (or I'm just unsure of the point you're making with that statement). I agree, they aren't conceptually complex, but Kubernetes is a large scheduler that _definitely_ benefits from having a dedicated team managing it. That being said, I always recommend using a tool like Terraform to back up infrastructure and the likes.
crypt1d 7y ago
I don't know why you are being down-voted, you are not wrong - Kubernetes is not something I would roll out at a start of a project. I think people are just excited to try it out so they often overlook the operational side of things.
- p_l 7y ago
  Considering that once you get through possibly high upfront cost, it greatly simplifies operational side, I often get the feeling that the "you are not Google" crowd misses the operational side completely, or looks at it through rose-tinted glasses.
  - petilon 7y ago
    Absolutely agree. The point of Kubernetes is to simplify operational side. I use it for my hobby projects. There is some learning investment needed, but after that it simplifies things so much. You can use Kubernetes for simple projects that don't need to scale to the size of Google.
    - p_l 7y ago
      In another recent thread, I mentioned running ~62 apps on kubernetes, and people asking if it could be simplified to less. They are mostly plain old PHP webapps, non-trivial amount of them Wordpress (shudduer), some done in random frameworks, some with ancient dependencies, some in node.js, one was ruby, etc. They are the equivalent of good old shared hosting stuff. With kubernetes, we generally manage them in much simplified form, and definitely much cheaper than hosting multiple VMs to work around issues in conflicting dependencies at distro level or keeping track of our own builds of PHP. We also run CI/CD on the same cluster. Ingress mechanics mean we have vastly simplified routing. Yes, we cheat a lot by using GKE instead of our own deployment of k8s, but we can manage that too, it's just cheaper that way. Pretty much smooth sailing with little operational worries except the fact that Google's stackdriver IMO sucks :)
neop1x 7y ago
Single point of failure. What if your machine needs reboot to apply kernel patches. What if your network connection breaks. Additionally, new crazy things are emerging like service mesh (Istio, Linkerd) which make observability and security easier. Of course, it all is getting quite complex. But it brings value in the end.

halfmatthalfcat 7y ago

Because it removes both provisioning and deployment concerns when you can build a container and then just tell Kubernetes to deploy it onto the cluster. Theres not much that goes into spinning up these managed Kuberentes clusters. Most of it is telling them what classes of instances you want created. When you buy a large instance, you still need to set up the instance and tweak it to your application's needs. You then need to babysit this node.

warrenm 7y ago
>When you buy a large instance, you still need to set up the instance and tweak it to your application's needs. You then need to babysit this node. It's called "use kickstart"

eknkc 7y ago

It is a single point of failure. I want multiple smaller instances even if I’m running something trivial. It is not about scale but about reliability for us. And the moment I go with multiple instances I need something to manage that mess. Kube handles it well.

spectramax 7y ago
I've used DigitalOcean's load balancer + multiple instances that run app containers. It works fine and without Kubernetes. I wasn't saying that just use 24 core single instance. Perhaps I should have worded it better.
- eknkc 7y ago
  Well a managed kubernetes cluster is not any more complex than the other solutions. You can use GCP control panel to launch clusters, add nodes, launch containers expose and autoscale them without touching a single line of config file or a terminal if thats what you like. If you launch / manage your own cluster though.. It’s a pain.
- closeparen 7y ago
  How do you obtain the right docker images and launch them with the right settings? How do you coordinate rolling deployments? (I'm not saying you need Kubernetes to do these things. But if you've written something to handle them, it is very probably Kubernetes-shaped).

mjfisher 7y ago

> Kubernetes feels like an insane complexity There's two things there - the complexity of engineering the cluster itself, and the complexity of using it. The former is where the pain is. If you can remove that pain by using a reliable managed offering, it changes the perspective a bit. The complexity of using it is also non-trivial - you have to learn it's terminology and primitives, understand the deployment and scheduling model, know how volumes are provisioned, aggregate logs, etc. However, the ROI for learning that complexity can be pretty good. If you get comfortable with it (maybe a month or so of learning and hacking?) you get sane rolling deployment processes by default, process management and auto healing, secrets and config management, scheduled tasks, health checks, and much, much easier infrastructure as code. Which means if things do go really sideways, it's usually not that hard just to stand up a replacement cluster from scratch. With a bit more reading and some additional open source services, you get fully automatic SSL management with let's encrypt too. All that said, I absolutely agree with you in principle. No one should overcomplicate their deployments for no benefit. It's just worth reflecting on what those benefits are. Kubernetes had a bunch of them - and whether they're worth the effort of getting to know it depends very much on the system(s) you're running.

spectramax 7y ago
I used to work for a company that had less than 130 employees and none of the software services were exposed to public - the total user count was 130 users. And we had a Kubernetes cluster. Thanks for an insightful response - Don't you think that a lot of wizbang software developers just want to use latest and greatest buzzword thing, whatever that might be, to get some cool points? As I grow older, I see increasing lack of objectivity and decreasing attention to KISS principles.
- mjfisher 7y ago
  > Don't you think that a lot of wizbang software developers just want to use latest and greatest buzzword thing Yes, absolutely, and it drives me up the wall. I've seen some incredibly unsuitable technology choices made, the complexity of which have absolutely sunk productivity on projects. That said, it's easy to become cynical and associate anything that's become recently popular with hype-driven garbage. But that can blind you to some really great new stuff too. I tend to hang behind the very early adopters and wait to see how useful new tech becomes in the wild - the "a year with X" style blog posts tend to be really informative.

p_l 7y ago

It all depends on how many applications you really have to run. When you have one application, can depend on vendored services for some of the infra, or have sufficiently small requirements for extra services, it all works pretty well. Helps if you also don't do due diligence on various things (like actually caring about logging and monitoring), so you scratch out those concerns. This is not even half sarcastic, people went pretty far with that. You might probably still spend more time than you think on actual deployment and management due to lack of automation & standardization, but if you're sufficiently small on the backend it works. When the number of concerns you have to manage rises and you want to optimize the infrastructure costs, things get weirder. Kubernetes' main offering is NOT scaling. It's providing a standardized control plane you can use to reduce the operational burden of managing often wildly different things that are components in your total infrastructure build - Things like infrastructural metrics and logging, your business metrics and logging, various extra software you might run to take care of stuff, making it easier to do HA, abstract away the underlying servers so you don't have to remember which server hosted which files or how the volumes were mounted (it can be as easy as classic old NFS/CIFS server with a rock-solid disk array, you just don't have to care about mounting the volumes on individual servers). It makes it easier to manage HTTPS routing in my experience (plop an nginx-ingress-controller with a host port, do a basic routing setup to push external HTTP/HTTPS traffic to those nodes that run it, get the bliss to forget about configuring individual Nginx configs anymore --- or use your cloud's Ingress controller, with little configuration difference!). In my experience, k8s was a force multiplier to any kind of infrastructure work, because what used to be much more complex, even with automation tools like Puppet or Chef, now had a way to be nicely forced into packages that even self-arranged themselves on the servers, without me having to care about which server and where. Except for setting up a node on-prem, or very rare debug tasks (that usually can be later rebuilt using k8s apis to get another force-multiplier), I don't have to SSH to servers except maybe for fun. Sometimes I login to specific containers, but that falls under debugging and tweaking the applications I run, not the underlying infrastructure. That's the offering - whether it is right for you, is another matter. Especially if you're in cash-rich SV startup, things like Heroku might be better. For me, the costs of running on PaaS are higher than getting another full-time engineer... and we manage to run k8s part-time.

saber6 7y ago

While I see a lot of derision about Kubernetes these days, if I am starting a Greenfield platform/design/product, why wouldn't I use it? There are tremendous benefits to K8S. It isn't just hype. On the flip side, starting out as a monolithic (all in one VM) app will take significant effort to transition to micro services / K8S. If I think I might end up at microservices/K8S, I think I might as well plan for it (abstractly) initially.

allset_ 7y ago
I believe this is the right way to think about it. You can start off with a relatively monolithic architecture, and then break that out into smaller microservices as needed with a much easier transition.

the_other_b 7y ago

If you're not going to build the next Facebook, why would you need so much complexity? You don't. I think this is a recent point people are trying to make. Kubernetes makes sense at a certain scale, but for smaller startups it maybe shouldn't be the go to.

spectramax 7y ago
Can you educate on at what point Kubernetes actually makes sense? Is there a rule of thumb?
- p_l 7y ago
  When you need to run more than 1-2 applications, especially in HA, and you're cost-conscious so just throwing tons of machines in autoscaling groups doesn't work for you. Said applications don't have to be applications that you're writing as part of your startup. It can be ElasticSearch cluster, Redis, tooling to run some kind of ETL that happens to be able to reuse your k8s cluster for task execution, CI/CD, etc. etc.
- the_other_b 7y ago
  I don't think I'm in a position to answer this (heavily experienced outside the control plane, but not in), but if I had to answer I'd say: Once you don't personally have your hand in every application your company is running (in addition to the points the other comments have brought up).
DelightOne 7y ago
So if Kubernetes is too complex, then terraform is a Nono too? I don't find them complex at all. You just tell the tools to be in a specific state and the tool applies the necessary changes. Server templates. Provisioning. Orchestration. etc.
- the_other_b 7y ago
  I don't think theres a comparison there (or I'm just unsure of the point you're making with that statement). I agree, they aren't conceptually complex, but Kubernetes is a large scheduler that _definitely_ benefits from having a dedicated team managing it. That being said, I always recommend using a tool like Terraform to back up infrastructure and the likes.

crypt1d 7y ago

I don't know why you are being down-voted, you are not wrong - Kubernetes is not something I would roll out at a start of a project. I think people are just excited to try it out so they often overlook the operational side of things.

p_l 7y ago
Considering that once you get through possibly high upfront cost, it greatly simplifies operational side, I often get the feeling that the "you are not Google" crowd misses the operational side completely, or looks at it through rose-tinted glasses.
- petilon 7y ago
  Absolutely agree. The point of Kubernetes is to simplify operational side. I use it for my hobby projects. There is some learning investment needed, but after that it simplifies things so much. You can use Kubernetes for simple projects that don't need to scale to the size of Google.
  - p_l 7y ago
    In another recent thread, I mentioned running ~62 apps on kubernetes, and people asking if it could be simplified to less. They are mostly plain old PHP webapps, non-trivial amount of them Wordpress (shudduer), some done in random frameworks, some with ancient dependencies, some in node.js, one was ruby, etc. They are the equivalent of good old shared hosting stuff. With kubernetes, we generally manage them in much simplified form, and definitely much cheaper than hosting multiple VMs to work around issues in conflicting dependencies at distro level or keeping track of our own builds of PHP. We also run CI/CD on the same cluster. Ingress mechanics mean we have vastly simplified routing. Yes, we cheat a lot by using GKE instead of our own deployment of k8s, but we can manage that too, it's just cheaper that way. Pretty much smooth sailing with little operational worries except the fact that Google's stackdriver IMO sucks :)

neop1x 7y ago

Single point of failure. What if your machine needs reboot to apply kernel patches. What if your network connection breaks. Additionally, new crazy things are emerging like service mesh (Istio, Linkerd) which make observability and security easier. Of course, it all is getting quite complex. But it brings value in the end.