9 ms·

Slightly off topic, but one of my most popular essays from last year was, in part, a criticism of Kubernetes. Kubernetes is best understood as an advanced optim

Slightly off topic, but one of my most popular essays from last year was, in part, a criticism of Kubernetes. Kubernetes is best understood as an advanced optimization that offers benefits to very large firms with thousands or tens of thousands of servers. Yet I've seen the idea spreading that every company, even a tiny startup with 3 servers, should Dockerize everything and run it all through Kubernetes. I mostly consult with small startups, so I've seen the damage these ideas can do. Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed. If anyone is interested in a case study of excess devops development, please see "High Availability is not compatible with an MVP, because an MVP is about fast iteration" http://www.smashcompany.com/technology/high-availability-is-not-compatible-with-mvp http://www.smashcompany.com/technology/high-availability-is-...

AYBABTME 6y ago
What do you recommend to these people instead of k8s, so that they can have all the convenience of orchestration and containerization?
- cogman10 6y ago
  Why containerize to start with? Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with. The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
  - p_l 6y ago
    PaaS tend to be quite expensive, especially when you're outside of USA. It's quite easy for example for Heroku to add a cost of fully loaded senior "devops" engineer to your bill, compared with k8s. Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
    - cogman10 6y ago
      While PaaS is expensive, the point really is that the cost of PaaS early on will likely be far less than the cost of properly setting up and building out a k8s cluster/deployment pipeline. Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
      - p_l 6y ago
        That sounds nice and dandy... except when PaaS increase your costs 1.5x compared to steady state (that has a lot of optimization opportunities left over), and about 6x when comparing initial conditions. And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
  - bronson 6y ago
    Lambdas get pricey quick. PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on). Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting. Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers. That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
  - AYBABTME 6y ago
    I mean, containerizing isn't harder than anything else. It makes distributing your artifacts much easier, and building across people's computers easier too. It takes O(minute) to containerize something, so I don't see why not do it.
- tlarkworthy 6y ago
  I do Terraform + VMs + Serverless. It's the same thing with a layer of indirection cut out and platform native cloud UIs.
- kristianpaul 6y ago
  You can always use systemd and pick an orchestration tool like ansible,cdk or terraform
  - eeZah7Ux 6y ago
    You don't even need the "orchestration tool"
- jadams3 6y ago
  Started small with k8s / gke, suffered a lot, coming to appreciate it as we grow. I think over the last 1-2 years there weren't a lot of other choices. Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible. For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.
- zenlot 6y ago
  Docker Swarm. It's not dead and you can set it up in no time.
- deleted 6y ago
  [deleted]
- cwyers 6y ago
  ECS and Cloud Run are options to get containers without k8s.
  - harpratap 6y ago
    Cloud Run is basically Custom Resource on top of Kubernetes API. The yaml definition is pretty much the same - https://cloud.google.com/run/docs/deploying#yaml https://cloud.google.com/run/docs/deploying#yaml What you mean is probably without vanilla k8s
    - cwyers 6y ago
      Well, sorta. I'm not surprised it's k8s under the hood. But it's not a managed k8s product like GKE -- you don't have to _care_ that it's k8s under the hood. To the consumer, it presents like ECS or Lambda.
- dolni 6y ago
  Run EC2 instances in an autoscaling group. Provision them with whatever you're comfortable with. Ansible is nice because you can point it at a node to test the deploy out without having to always respin the node to try again. Have ansible fire up a docker container on your host. That's it, done. You don't need Kubernetes. Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors. While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in. If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity. Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.
  - lkrubner 6y ago
    This can go a very long way. Using Ansible and Packer to provision AMIs, and then Terraform to spin up what's needed, goes a long way, without introducing a new layer of technology.
  - AYBABTME 6y ago
    But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
    - dolni 6y ago
      > But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-refresh-for-ec2-auto-scaling/ https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one. Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes? > I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1. The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently. There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through. None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.
cogman10 6y ago
The problem with Kubernetes seems to be one of knowing when to make the switch. My company is in a place where I think k8s would be really beneficial. We are at the 40k VMs running 1000 different types of apps. We suffer from tremendous wasted resources as a result. The company started before k8s (and even docker) were things. So getting us there seems like a real daunting (yet worth it) task. So, we are left with sort of a chicken/egg problem. It would have been unreasonable to start with k8s, yet now it is a huge cost to switch to it (even though it will save us millions).
- brown9-2 6y ago
  It’s easy to overestimate that you’ll save money here - if you have waste, poor cost attribution, etc., you can easily do the same with Kubernetes managing the nodes and scheduling the processes. To actually save the money you have to fix those things first.
  - cogman10 6y ago
    The issue we have is that we are a very peeky computation requirements (not a steady stream of calculations going on). Further, we've got a fairly deep data processing pipeline. It's not unusual for us to see high loads on early parts of the system 1 hour and 1 hour later seeing high loads on the later parts of the system. Unfortunately, we don't currently have a way to scale out and reassign resources. That means we've built out for our peaks even though the majority of the time that hardware is sleeping. We could spend the time building out a scaling system (right now, our resources are all on prem). However, why do that when we can instead take advantage of k8s already built in scaling capabilities? So, technically not k8s that will be saving us money, but rather implementing autoscaling. However, the solution with k8s is really nice.
    - brown9-2 6y ago
      Cluster autoscaling with nodes on-prem, or in the cloud?
- exhilaration 6y ago
p_l 6y ago
That has nothing to do with kubernetes, and a lot to do with management of the company. I might be biased, because my first commercial use of kubernetes was to reduce infrastructure spend and optimize the hell of what we have. From my PoV, the "big company" narrative is just what happens when the big company notices the cost of spinning separate VMs for everything.
scoopertrooper 6y ago
I'll have a read through your article later as I'm quite interested in hearing your point of view. However in my experience, it's pretty easy to get a reasonable Kubernetes cluster setup with stateless apps on a cloud platform managed Kubernetes service.
- ramraj07 6y ago
  Can you share in your opinion a good single guide to do this in AWS?
  - scoopertrooper 6y ago
    I haven't worked with EKS; I've mainly worked with Azure's AKS, but the principals are largely the same. I'm not sure how familiar you are with AWS, EKS, Kubernetes, or Docker in general and I don't know of a guide that encompasses all these technologies. My best advice would be: 1) Get a little toy Docker container running locally with a http server that responds with "Hello world" 2) Try and deploy it to as a pod to a local Kubernetes cluster (Docker desktop has one built in or you can try deploying Minikube if you're feeling adventurous) 3) Add an ingress and service to your cluster, so you can access your server from a browser 4) Expand your cluster by installing something like Postgres on it and have your toy app query it and print out a query result in response to requests 5) Try passing your toy app the password to login to Postgres as a Kubernetes Secret via an environmental variable 6) Start exploring EKS specific deployment documentation such as the links below. 7) Start learning about helm https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html https://docs.aws.amazon.com/eks/latest/userguide/create-clus... https://docs.aws.amazon.com/eks/latest/userguide/sample-deployment.html https://docs.aws.amazon.com/eks/latest/userguide/sample-depl... https://learn.hashicorp.com/tutorials/terraform/eks https://learn.hashicorp.com/tutorials/terraform/eks (Once your comfortable with everything else)
    - ramraj07 6y ago
      Honestly that sounds like not simple at all. I just want a webapp up and running, with a database and cache server, and I have again and again launched Elastic Beanstalk + Elasticache + RDS (or the GCP version) in half a day following basic tutorials, and this system runs very well (and scales very well) with zero need to "dive into documentation" for various things. I can then focus on my app instead of infra. It's still not clear why such simple architectures need kubernetes.
arderary 6y ago
In a lot of cases it's worth considering whether orchestration is necessary at all — pure docker can take you quite far, with just a couple shell scripts called from the last stage of your CI (or manually)
nrmitchi 6y ago
I 100% see the point you are trying to make. However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial. One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to. > Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed. This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.
- blandflakes 6y ago
  Helm was something that I ultimately decided to avoid at my last job, and I was happier for it. Upgrades that get in dirty states, often as a result of bogus post-install hooks. Having to fork charts to customize the deployment minorly for our mundane requirements. Reading doubly or triply templated files and dealing with bullshit indentation mismatches. I hated things that expected me to install with Helm.
  - nrmitchi 6y ago
    I can agree with you that some of uses of it get out of hand. Like most things, there is a balance. For the most part I don't use hooks, precisely because they muddy the waters of my expected end state. Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges. What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
    - blandflakes 6y ago
      A lot of the time, writing (or forking and customizing) my own manifest was at least more deterministic and predictable. Obviously you get lost at that point in time if you aren't diligent about upgrading. Helm felt like this middleground, and I really felt that at least for managed clusters, it would be better if the addon managers provided stuff that Just Worked (tm). We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language. Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).
jayd16 6y ago
Whats wrong with running small deploys? It takes less than half a day to set up. Its honestly way easier to get a reproducible k8s setup than some hand rolled thing even if you run a single pod. Or do you mean a small team should not use unmanaged k8s?
derefr 6y ago
Certainly, all the current implementations of Kubernetes are built out of enterprise-scale components — even if they're then squeezed into the girdle of a developer tool, the way microk8s is. But the design (functional architecture, APIs, etc.) of Kubernetes makes good sense for small scale as well. There's nothing fundamentally stopping someone from creating an "embedded Kubernetes" that exposes all the same resource APIs, but which is just a management interface for e.g. a JVM, or an Erlang node, where all those APIs are just controlling in-memory abstractions existing inside one userland process. Presenting itself as a "cluster" with one fixed node pool, with one fixed node, with one fixed type of container runtime which is just "spawn an actor-thread in this process." I don't disagree that people are pursuing the enterprise k8s stuff when they aren't at the right scale to need it — but I think it's important to realize that they're often doing that because they see the design of k8s as fundamentally sound, providing sensible abstractions and management interfaces that neither plain OS process APIs, nor Docker's APIs, nor IaaS-level control-plane APIs have great equivalents for. It's that design they're after; the HA stuff is just a "nice-to-have" that they end up adding on, because all the guides they read (that are written for enterprise integrators) insist it's a great idea. (Tangent: if Docker Inc. would get off their collective butts and do Docker Swarm over again, but this time with k8s-compatible APIs, I'm sure it'd become the most-used "Kubernetes" overnight. Most devs just want a regular Docker host, that they can `kubectl apply -f` against.)
merb 6y ago
> they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed not sure, but with 1$ million on budget, you can run a pretty pretty big gke cluster. btw. we run on e2-standard-2 and pay ~500 per month (would be a little bit more without commited usage). 6 nodes + 2 managed cloud sql databases+several load balancers. really no idea how one can pay a million even in a year. that would not be a small startup... you can probably have like 9x e2-standard-32 (32vcpu+128gb) + postgres ha 16vcpu/256gb mem/500gb ssd for like 10,867.33 per month without traffic (which might be another 5000) so with traffic you would only pay 190407.96 per year?! Edit: even without gke, k3s is way simplier than most "deployment scripts" or whatever. it's always a matter if you want to have ops people or if gke's self managing is enough. Edit2: so many lols at the blog post. I mean who the fuck spends 7000 at a colo for 40.0000 users. my macbook could probably serve that. also "and then script those older Virtual Machines with some mix of Ansible and Terraform (assuming you want to be in the cloud). " is a fucking terrible advice, been there, done that and it soon is a clusterfuck of stuff cobbeled together.
- cwyers 6y ago
  Yeah, from the blog post: > I’ve a good friend at a startup who found a co-lo data center in a small college town in Virginia, where they were able to build out a massive MongoDB database, sharded across 4 massive servers, each server holding 256 gigabytes of RAM, 1 terabyte of RAM in total. They also had a Cassandra database and several servers for their front-end. They were paying $7,000 a month for this. Like, if you're paying AWS to host MongoDB on four EC2 machines with 256gb of RAM each and no autoscaling/load balancing, you can spend a lot of money. But... why would you do that? Porting terrible architectures to cloud VMs with no rearchitecting costs a boatload, but... don't do that?
- lkrubner 6y ago
  "I mean who the fuck spends 7000 at a colo for 40.0000 users." You misread the article. The startup with the $1 million devops ended up with 40,000 users. The startup with the co-lo was a different startup, which in fact had a very large user base.
ncrmro 6y ago
You can actually set up k3os on a single node and get all the benefits eg release a new deployment and have it keep old pods up until new pods are healthy . I have it hosted on my NAS and use ansible to set up deployments and in ci. Tooling has come along way as well. But the learning curve is knarly and I picked up a few tricks from companies I worked at.
ablekh 6y ago
While there is some truth to what you are saying in your linked post, the real truth is more nuanced and, as usual, lies between any extremes. Firstly, when talking about extreme flexibility and experimentation, you're not talking about MVP in my book, but rather about a prototype. You can experiment with a prototype all you want, but no so much with an MVP, which, be definition, is a product, even if a minimal viable one. Secondly, your argument that MVP-stage early startups don't need HA breaks due to the corollary of MVP being a product. If your early-stage startup has, say, 5-10 important enterprise customers (those whose logos you so desperately need for initial sales as well as further marketing) using your MVP as product or even a pilot, guess how many of them will stay with you after a first 5-hour outage of your platform? Thirdly, in order to prevent disastrous exodus situations like mentioned above, an early startup does not need to have a globally scalable geographically distributed HA system (or, as you put it, "multiple servers, multiple load balancers, multiple failover databases, multiple data centers, multiple regions"). It would be quite enough to have a single level of redundancy for each major single point of failure. For a typical cloud architecture, it just means architecting, deploying and maintaining your system for/to/in two availability zones. This IMO represents a pretty good compromise between costs (both monetary and effort-wise) and risks (reputational, current and future sales, etc.). Fourthly, your suggested preference of VMs to containers will most likely lead to a) lower agility [due to spinning a VM being a much slower process versus spinning a container; that would be especially noticeable across development cycles: CI/CD etc.] and b) overspending on infrastructure [due to overhead stemming from the fact that you simply cannot pack VMs as efficient as containers into a given computational asset]. Fifthly, some/many operational aspects that have to be managed - manually or via automation such as Terraform - in non-HA scenarios could be partially or completely eliminated in cloud-based HA scenarios thanks to those aspects being embedded into relevant PaaS products, hence potentially significant cost savings and, equally or more importantly, time savings, thus, enabling a potentially much higher agility, which is so important to early-stage startups.
sieabahlpark 6y ago
I've managed to have really good success on using Kubernetes to actually lower my costs rather than raise them. The issue with Kubernetes is that it's a development mindset. Rather than developing persistence you develop mutations to something persistent.
pyuser583 6y ago
I see Kubernetes as a way to save money on web services. It allows for platform independent deployment. Moving from AWS to Google Cloud to bare metal can save massive amount of money.
pc86 6y ago
Serious question - what's the alternative? We have Docker containers, we want to run in swarm mode, and we want to enable service-to-service communication via gRPC. Kube seems to be the only thing that can support that unless we want to pay Traefik $20k+ which is a hard pass.
raffraffraff 6y ago
I agree that a startup shouldn't (yet) put a lot of time into redundancy if they don't have a product or customers. But a code pipeline that can be used to quickly develop, test and deploy is essential. There is absolutely nothing wrong with using containers. In fact, in 20 years in tech, it's the best packaging solution I've seen. As for Kubernetes, as long as you're not managing the backend, it's fine. I joined a tiny startup less than a year ago, set up EKS with Terraform and created CI automation to build and push containers when tests pass. It's not hard. And it does encourage good practices. If we had to move everything to Azure or GCE, it would be much easier to do so because we haven't painted ourselves into the AWS corner.

AYBABTME 6y ago

What do you recommend to these people instead of k8s, so that they can have all the convenience of orchestration and containerization?

cogman10 6y ago
Why containerize to start with? Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with. The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
- p_l 6y ago
  PaaS tend to be quite expensive, especially when you're outside of USA. It's quite easy for example for Heroku to add a cost of fully loaded senior "devops" engineer to your bill, compared with k8s. Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
  - cogman10 6y ago
    While PaaS is expensive, the point really is that the cost of PaaS early on will likely be far less than the cost of properly setting up and building out a k8s cluster/deployment pipeline. Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
    - p_l 6y ago
      That sounds nice and dandy... except when PaaS increase your costs 1.5x compared to steady state (that has a lot of optimization opportunities left over), and about 6x when comparing initial conditions. And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
- bronson 6y ago
  Lambdas get pricey quick. PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on). Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting. Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers. That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
- AYBABTME 6y ago
  I mean, containerizing isn't harder than anything else. It makes distributing your artifacts much easier, and building across people's computers easier too. It takes O(minute) to containerize something, so I don't see why not do it.
tlarkworthy 6y ago
I do Terraform + VMs + Serverless. It's the same thing with a layer of indirection cut out and platform native cloud UIs.
kristianpaul 6y ago
You can always use systemd and pick an orchestration tool like ansible,cdk or terraform
- eeZah7Ux 6y ago
  You don't even need the "orchestration tool"
jadams3 6y ago
Started small with k8s / gke, suffered a lot, coming to appreciate it as we grow. I think over the last 1-2 years there weren't a lot of other choices. Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible. For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.
zenlot 6y ago
Docker Swarm. It's not dead and you can set it up in no time.
deleted 6y ago
[deleted]
cwyers 6y ago
ECS and Cloud Run are options to get containers without k8s.
- harpratap 6y ago
  Cloud Run is basically Custom Resource on top of Kubernetes API. The yaml definition is pretty much the same - https://cloud.google.com/run/docs/deploying#yaml https://cloud.google.com/run/docs/deploying#yaml What you mean is probably without vanilla k8s
  - cwyers 6y ago
    Well, sorta. I'm not surprised it's k8s under the hood. But it's not a managed k8s product like GKE -- you don't have to _care_ that it's k8s under the hood. To the consumer, it presents like ECS or Lambda.
dolni 6y ago
Run EC2 instances in an autoscaling group. Provision them with whatever you're comfortable with. Ansible is nice because you can point it at a node to test the deploy out without having to always respin the node to try again. Have ansible fire up a docker container on your host. That's it, done. You don't need Kubernetes. Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors. While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in. If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity. Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.
- lkrubner 6y ago
  This can go a very long way. Using Ansible and Packer to provision AMIs, and then Terraform to spin up what's needed, goes a long way, without introducing a new layer of technology.
- AYBABTME 6y ago
  But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
  - dolni 6y ago
    > But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you. This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-refresh-for-ec2-auto-scaling/ https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one. Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes? > I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1. The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently. There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through. None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.

cogman10 6y ago

The problem with Kubernetes seems to be one of knowing when to make the switch. My company is in a place where I think k8s would be really beneficial. We are at the 40k VMs running 1000 different types of apps. We suffer from tremendous wasted resources as a result. The company started before k8s (and even docker) were things. So getting us there seems like a real daunting (yet worth it) task. So, we are left with sort of a chicken/egg problem. It would have been unreasonable to start with k8s, yet now it is a huge cost to switch to it (even though it will save us millions).

brown9-2 6y ago
It’s easy to overestimate that you’ll save money here - if you have waste, poor cost attribution, etc., you can easily do the same with Kubernetes managing the nodes and scheduling the processes. To actually save the money you have to fix those things first.
- cogman10 6y ago
  The issue we have is that we are a very peeky computation requirements (not a steady stream of calculations going on). Further, we've got a fairly deep data processing pipeline. It's not unusual for us to see high loads on early parts of the system 1 hour and 1 hour later seeing high loads on the later parts of the system. Unfortunately, we don't currently have a way to scale out and reassign resources. That means we've built out for our peaks even though the majority of the time that hardware is sleeping. We could spend the time building out a scaling system (right now, our resources are all on prem). However, why do that when we can instead take advantage of k8s already built in scaling capabilities? So, technically not k8s that will be saving us money, but rather implementing autoscaling. However, the solution with k8s is really nice.
  - brown9-2 6y ago
    Cluster autoscaling with nodes on-prem, or in the cloud?
exhilaration 6y ago

p_l 6y ago

That has nothing to do with kubernetes, and a lot to do with management of the company. I might be biased, because my first commercial use of kubernetes was to reduce infrastructure spend and optimize the hell of what we have. From my PoV, the "big company" narrative is just what happens when the big company notices the cost of spinning separate VMs for everything.

scoopertrooper 6y ago

I'll have a read through your article later as I'm quite interested in hearing your point of view. However in my experience, it's pretty easy to get a reasonable Kubernetes cluster setup with stateless apps on a cloud platform managed Kubernetes service.

ramraj07 6y ago
Can you share in your opinion a good single guide to do this in AWS?
- scoopertrooper 6y ago
  I haven't worked with EKS; I've mainly worked with Azure's AKS, but the principals are largely the same. I'm not sure how familiar you are with AWS, EKS, Kubernetes, or Docker in general and I don't know of a guide that encompasses all these technologies. My best advice would be: 1) Get a little toy Docker container running locally with a http server that responds with "Hello world" 2) Try and deploy it to as a pod to a local Kubernetes cluster (Docker desktop has one built in or you can try deploying Minikube if you're feeling adventurous) 3) Add an ingress and service to your cluster, so you can access your server from a browser 4) Expand your cluster by installing something like Postgres on it and have your toy app query it and print out a query result in response to requests 5) Try passing your toy app the password to login to Postgres as a Kubernetes Secret via an environmental variable 6) Start exploring EKS specific deployment documentation such as the links below. 7) Start learning about helm https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html https://docs.aws.amazon.com/eks/latest/userguide/create-clus... https://docs.aws.amazon.com/eks/latest/userguide/sample-deployment.html https://docs.aws.amazon.com/eks/latest/userguide/sample-depl... https://learn.hashicorp.com/tutorials/terraform/eks https://learn.hashicorp.com/tutorials/terraform/eks (Once your comfortable with everything else)
  - ramraj07 6y ago
    Honestly that sounds like not simple at all. I just want a webapp up and running, with a database and cache server, and I have again and again launched Elastic Beanstalk + Elasticache + RDS (or the GCP version) in half a day following basic tutorials, and this system runs very well (and scales very well) with zero need to "dive into documentation" for various things. I can then focus on my app instead of infra. It's still not clear why such simple architectures need kubernetes.

arderary 6y ago

In a lot of cases it's worth considering whether orchestration is necessary at all — pure docker can take you quite far, with just a couple shell scripts called from the last stage of your CI (or manually)

nrmitchi 6y ago

I 100% see the point you are trying to make. However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial. One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to. > Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed. This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.

blandflakes 6y ago
Helm was something that I ultimately decided to avoid at my last job, and I was happier for it. Upgrades that get in dirty states, often as a result of bogus post-install hooks. Having to fork charts to customize the deployment minorly for our mundane requirements. Reading doubly or triply templated files and dealing with bullshit indentation mismatches. I hated things that expected me to install with Helm.
- nrmitchi 6y ago
  I can agree with you that some of uses of it get out of hand. Like most things, there is a balance. For the most part I don't use hooks, precisely because they muddy the waters of my expected end state. Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges. What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
  - blandflakes 6y ago
    A lot of the time, writing (or forking and customizing) my own manifest was at least more deterministic and predictable. Obviously you get lost at that point in time if you aren't diligent about upgrading. Helm felt like this middleground, and I really felt that at least for managed clusters, it would be better if the addon managers provided stuff that Just Worked (tm). We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language. Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).

jayd16 6y ago

Whats wrong with running small deploys? It takes less than half a day to set up. Its honestly way easier to get a reproducible k8s setup than some hand rolled thing even if you run a single pod. Or do you mean a small team should not use unmanaged k8s?

derefr 6y ago

Certainly, all the current implementations of Kubernetes are built out of enterprise-scale components — even if they're then squeezed into the girdle of a developer tool, the way microk8s is. But the design (functional architecture, APIs, etc.) of Kubernetes makes good sense for small scale as well. There's nothing fundamentally stopping someone from creating an "embedded Kubernetes" that exposes all the same resource APIs, but which is just a management interface for e.g. a JVM, or an Erlang node, where all those APIs are just controlling in-memory abstractions existing inside one userland process. Presenting itself as a "cluster" with one fixed node pool, with one fixed node, with one fixed type of container runtime which is just "spawn an actor-thread in this process." I don't disagree that people are pursuing the enterprise k8s stuff when they aren't at the right scale to need it — but I think it's important to realize that they're often doing that because they see the design of k8s as fundamentally sound, providing sensible abstractions and management interfaces that neither plain OS process APIs, nor Docker's APIs, nor IaaS-level control-plane APIs have great equivalents for. It's that design they're after; the HA stuff is just a "nice-to-have" that they end up adding on, because all the guides they read (that are written for enterprise integrators) insist it's a great idea. (Tangent: if Docker Inc. would get off their collective butts and do Docker Swarm over again, but this time with k8s-compatible APIs, I'm sure it'd become the most-used "Kubernetes" overnight. Most devs just want a regular Docker host, that they can `kubectl apply -f` against.)

merb 6y ago

> they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed not sure, but with 1$ million on budget, you can run a pretty pretty big gke cluster. btw. we run on e2-standard-2 and pay ~500 per month (would be a little bit more without commited usage). 6 nodes + 2 managed cloud sql databases+several load balancers. really no idea how one can pay a million even in a year. that would not be a small startup... you can probably have like 9x e2-standard-32 (32vcpu+128gb) + postgres ha 16vcpu/256gb mem/500gb ssd for like 10,867.33 per month without traffic (which might be another 5000) so with traffic you would only pay 190407.96 per year?! Edit: even without gke, k3s is way simplier than most "deployment scripts" or whatever. it's always a matter if you want to have ops people or if gke's self managing is enough. Edit2: so many lols at the blog post. I mean who the fuck spends 7000 at a colo for 40.0000 users. my macbook could probably serve that. also "and then script those older Virtual Machines with some mix of Ansible and Terraform (assuming you want to be in the cloud). " is a fucking terrible advice, been there, done that and it soon is a clusterfuck of stuff cobbeled together.

cwyers 6y ago
Yeah, from the blog post: > I’ve a good friend at a startup who found a co-lo data center in a small college town in Virginia, where they were able to build out a massive MongoDB database, sharded across 4 massive servers, each server holding 256 gigabytes of RAM, 1 terabyte of RAM in total. They also had a Cassandra database and several servers for their front-end. They were paying $7,000 a month for this. Like, if you're paying AWS to host MongoDB on four EC2 machines with 256gb of RAM each and no autoscaling/load balancing, you can spend a lot of money. But... why would you do that? Porting terrible architectures to cloud VMs with no rearchitecting costs a boatload, but... don't do that?
lkrubner 6y ago
"I mean who the fuck spends 7000 at a colo for 40.0000 users." You misread the article. The startup with the $1 million devops ended up with 40,000 users. The startup with the co-lo was a different startup, which in fact had a very large user base.

ncrmro 6y ago

You can actually set up k3os on a single node and get all the benefits eg release a new deployment and have it keep old pods up until new pods are healthy . I have it hosted on my NAS and use ansible to set up deployments and in ci. Tooling has come along way as well. But the learning curve is knarly and I picked up a few tricks from companies I worked at.

ablekh 6y ago

While there is some truth to what you are saying in your linked post, the real truth is more nuanced and, as usual, lies between any extremes. Firstly, when talking about extreme flexibility and experimentation, you're not talking about MVP in my book, but rather about a prototype. You can experiment with a prototype all you want, but no so much with an MVP, which, be definition, is a product, even if a minimal viable one. Secondly, your argument that MVP-stage early startups don't need HA breaks due to the corollary of MVP being a product. If your early-stage startup has, say, 5-10 important enterprise customers (those whose logos you so desperately need for initial sales as well as further marketing) using your MVP as product or even a pilot, guess how many of them will stay with you after a first 5-hour outage of your platform? Thirdly, in order to prevent disastrous exodus situations like mentioned above, an early startup does not need to have a globally scalable geographically distributed HA system (or, as you put it, "multiple servers, multiple load balancers, multiple failover databases, multiple data centers, multiple regions"). It would be quite enough to have a single level of redundancy for each major single point of failure. For a typical cloud architecture, it just means architecting, deploying and maintaining your system for/to/in two availability zones. This IMO represents a pretty good compromise between costs (both monetary and effort-wise) and risks (reputational, current and future sales, etc.). Fourthly, your suggested preference of VMs to containers will most likely lead to a) lower agility [due to spinning a VM being a much slower process versus spinning a container; that would be especially noticeable across development cycles: CI/CD etc.] and b) overspending on infrastructure [due to overhead stemming from the fact that you simply cannot pack VMs as efficient as containers into a given computational asset]. Fifthly, some/many operational aspects that have to be managed - manually or via automation such as Terraform - in non-HA scenarios could be partially or completely eliminated in cloud-based HA scenarios thanks to those aspects being embedded into relevant PaaS products, hence potentially significant cost savings and, equally or more importantly, time savings, thus, enabling a potentially much higher agility, which is so important to early-stage startups.

sieabahlpark 6y ago

I've managed to have really good success on using Kubernetes to actually lower my costs rather than raise them. The issue with Kubernetes is that it's a development mindset. Rather than developing persistence you develop mutations to something persistent.

pyuser583 6y ago

I see Kubernetes as a way to save money on web services. It allows for platform independent deployment. Moving from AWS to Google Cloud to bare metal can save massive amount of money.

pc86 6y ago

Serious question - what's the alternative? We have Docker containers, we want to run in swarm mode, and we want to enable service-to-service communication via gRPC. Kube seems to be the only thing that can support that unless we want to pay Traefik $20k+ which is a hard pass.

raffraffraff 6y ago

I agree that a startup shouldn't (yet) put a lot of time into redundancy if they don't have a product or customers. But a code pipeline that can be used to quickly develop, test and deploy is essential. There is absolutely nothing wrong with using containers. In fact, in 20 years in tech, it's the best packaging solution I've seen. As for Kubernetes, as long as you're not managing the backend, it's fine. I joined a tiny startup less than a year ago, set up EKS with Terraform and created CI automation to build and push containers when tests pass. It's not hard. And it does encourage good practices. If we had to move everything to Azure or GCE, it would be much easier to do so because we haven't painted ourselves into the AWS corner.