4 ms·

That's why k8s is not a good fit for early startups. It will consume too much of your time that is better spent developing a product that the public wants.

AlphaSite 6y ago
In that case use a managed cluster and it’s less work.
- speedgoose 6y ago
  It has about the same issues but because it's managed and because you are a small company, nobody cares about your issues and you can do nothing about it.
- zimbatm 6y ago
  You still have to do upgrades every 3 months, sometimes with breaking changes. Because the underlying infrastructure gets replaced, it means spinning up a new cluster, deploying the application on it to make sure that everything is working. Then switch the traffic to it. If the cluster hold any state this can be quite a bit of work to replicate all of the data.
  - q3k 6y ago
    Kubernetes clusters can be upgraded in place, though? GKE definitely allows you to do that.
    - zimbatm 6y ago
      It works most of the time. I have also seen multiple in-place upgrades ending up in weird places. Or some resource having changed API and we had to rush to upgrade the code. What will you do if that happens? It's a bit like upgrading Postgres. It's a good idea to at least make a backup before changing the underlying storage.
    - dharmab 6y ago
      Yes, if you’re careful and do your research. Once you start extending Kubernetes you can back yourself into corners. We’ve also found breaking bugs in Kubernetes that prevented in-place upgrades in some cases (which we reported upstream and were patched).
- capableweb 6y ago
  zimbatm mentioned "early startups", I take that to assume you've just started the company and you're working on getting the product market fit right. There is absolutely no need for any clusters for that, unless you're anticipating Google-level traffic on launch day. Most companies today are better off NOT running distributed anything, as everything will get harder to do, from logging to deployments. People start with clusters when they really should start with centralized services and only go distributed when they need to.
  - zimbatm 6y ago
    Indeed. A single machine is often more than enough for early startups. It's easy for developers to understand and debug. What's important is to put the tooling in place to make the deployment reproducible. Deploy using Terraform and Docker to the single VM. Avoid modifying things manually. That's what provides the agility to then scale the infrastructure into bigger shapes as needed. That's why we developed https://github.com/numtide/terraform-provider-linuxbox https://github.com/numtide/terraform-provider-linuxbox , to make that use-case even easier.
- deleted 6y ago
  [deleted]
znpy 6y ago
imho if you're an early stage startup and you don't have an operations or devops person, you're better off either running everything off a single machine or doing devops the old way (as in baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). Once you get more revenue, hiring an operations/devops person might be a good idea. A good operations/devops person might make you save his own salary in infrastructure optimization (depending on the scale of course). another good pattern that i've seen in action is contracting the operations to an external company that's specialized in doing this. the approach would be to let the external company design and build the cluster for you, define responsibility boundaries for your company vs that company, and let such company take care of daily operations and cluster upgrades, possibly with on-call availability. A small support/consulting package (say, 15-20 hours/month) can get you very far. We've been doing something similar with SigHup (sighup.io) and their Kubernetes Fury distribution, it's been working very well so far. (disclaimer: i'm not employed by sighup, just pleased to work with them)
- zimbatm 6y ago
  Agreed. That's exactly what I have done with a number of startups. In one or two weeks, they get a full CI/CD pipeline that allows them to be productive. All of them are very happy with the result. It's so important to be deploying early on. Get the kinks out of the system and the code aligned for production. If you hire a full time devops, they will most likely build too much infrastructure to keep themselves busy. And if you don't do any devops, the transition can be quite painful.
- mwcampbell 6y ago
  > baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). I tried this approach, using Packer with Ansible to build the AMIs. It took 15-20 minutes to build an AMI. Would you suggest a different way of building the AMIs?
  - znpy 6y ago
    That was meant to be an example really, to aim at something less automatic and complex like kubernetes but still automated. You might also consider using containers without kubernetes. Keep the same AMI, package your app as a docker container, make VMs download and run the latest release from a private docker registry on startup. Publish a new docker image tagged as latest. Now you can rollout your autoscaling groups without running packer and ansible.

AlphaSite 6y ago

In that case use a managed cluster and it’s less work.

speedgoose 6y ago
It has about the same issues but because it's managed and because you are a small company, nobody cares about your issues and you can do nothing about it.
zimbatm 6y ago
You still have to do upgrades every 3 months, sometimes with breaking changes. Because the underlying infrastructure gets replaced, it means spinning up a new cluster, deploying the application on it to make sure that everything is working. Then switch the traffic to it. If the cluster hold any state this can be quite a bit of work to replicate all of the data.
- q3k 6y ago
  Kubernetes clusters can be upgraded in place, though? GKE definitely allows you to do that.
  - zimbatm 6y ago
    It works most of the time. I have also seen multiple in-place upgrades ending up in weird places. Or some resource having changed API and we had to rush to upgrade the code. What will you do if that happens? It's a bit like upgrading Postgres. It's a good idea to at least make a backup before changing the underlying storage.
  - dharmab 6y ago
    Yes, if you’re careful and do your research. Once you start extending Kubernetes you can back yourself into corners. We’ve also found breaking bugs in Kubernetes that prevented in-place upgrades in some cases (which we reported upstream and were patched).
capableweb 6y ago
zimbatm mentioned "early startups", I take that to assume you've just started the company and you're working on getting the product market fit right. There is absolutely no need for any clusters for that, unless you're anticipating Google-level traffic on launch day. Most companies today are better off NOT running distributed anything, as everything will get harder to do, from logging to deployments. People start with clusters when they really should start with centralized services and only go distributed when they need to.
- zimbatm 6y ago
  Indeed. A single machine is often more than enough for early startups. It's easy for developers to understand and debug. What's important is to put the tooling in place to make the deployment reproducible. Deploy using Terraform and Docker to the single VM. Avoid modifying things manually. That's what provides the agility to then scale the infrastructure into bigger shapes as needed. That's why we developed https://github.com/numtide/terraform-provider-linuxbox https://github.com/numtide/terraform-provider-linuxbox , to make that use-case even easier.
deleted 6y ago
[deleted]

znpy 6y ago

imho if you're an early stage startup and you don't have an operations or devops person, you're better off either running everything off a single machine or doing devops the old way (as in baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). Once you get more revenue, hiring an operations/devops person might be a good idea. A good operations/devops person might make you save his own salary in infrastructure optimization (depending on the scale of course). another good pattern that i've seen in action is contracting the operations to an external company that's specialized in doing this. the approach would be to let the external company design and build the cluster for you, define responsibility boundaries for your company vs that company, and let such company take care of daily operations and cluster upgrades, possibly with on-call availability. A small support/consulting package (say, 15-20 hours/month) can get you very far. We've been doing something similar with SigHup (sighup.io) and their Kubernetes Fury distribution, it's been working very well so far. (disclaimer: i'm not employed by sighup, just pleased to work with them)

zimbatm 6y ago
Agreed. That's exactly what I have done with a number of startups. In one or two weeks, they get a full CI/CD pipeline that allows them to be productive. All of them are very happy with the result. It's so important to be deploying early on. Get the kinks out of the system and the code aligned for production. If you hire a full time devops, they will most likely build too much infrastructure to keep themselves busy. And if you don't do any devops, the transition can be quite painful.
mwcampbell 6y ago
> baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups). I tried this approach, using Packer with Ansible to build the AMIs. It took 15-20 minutes to build an AMI. Would you suggest a different way of building the AMIs?
- znpy 6y ago
  That was meant to be an example really, to aim at something less automatic and complex like kubernetes but still automated. You might also consider using containers without kubernetes. Keep the same AMI, package your app as a docker container, make VMs download and run the latest release from a private docker registry on startup. Publish a new docker image tagged as latest. Now you can rollout your autoscaling groups without running packer and ansible.