8 ms·

Kubernetes 1.20

119 points by gtirloni 6y ago

tristor 6y ago
Volume Snapshots are going to be a game-changer for stateful workloads. I'm glad to see this feature in the wild. Great job Kubernetes team!
- tedk-42 6y ago
  Agreed! It's an abstraction on top of many existing cloud provider storage abstractions which will be a welcome feature to many places wanting a quick backup solution of their volumes.
lkrubner 6y ago
Slightly off topic, but one of my most popular essays from last year was, in part, a criticism of Kubernetes. Kubernetes is best understood as an advanced optimization that offers benefits to very large firms with thousands or tens of thousands of servers. Yet I've seen the idea spreading that every company, even a tiny startup with 3 servers, should Dockerize everything and run it all through Kubernetes. I mostly consult with small startups, so I've seen the damage these ideas can do. Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed. If anyone is interested in a case study of excess devops development, please see "High Availability is not compatible with an MVP, because an MVP is about fast iteration" http://www.smashcompany.com/technology/high-availability-is-not-compatible-with-mvp http://www.smashcompany.com/technology/high-availability-is-...
- AYBABTME 6y ago
  What do you recommend to these people instead of k8s, so that they can have all the convenience of orchestration and containerization?
  - cogman10 6y ago
    Why containerize to start with? Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with. The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
    - p_l 6y ago
      PaaS tend to be quite expensive, especially when you're outside of USA. It's quite easy for example for Heroku to add a cost of fully loaded senior "devops" engineer to your bill, compared with k8s. Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
      - cogman10 6y ago
        While PaaS is expensive, the point really is that the cost of PaaS early on will likely be far less than the cost of properly setting up and building out a k8s cluster/deployment pipeline. Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
        p_l 6y ago
        That sounds nice and dandy... except when PaaS increase your costs 1.5x compared to steady state (that has a lot of optimization opportunities left over), and about 6x when comparing initial conditions. And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
    - bronson 6y ago
      Lambdas get pricey quick. PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on). Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting. Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers. That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
- cogman10 6y ago
  The problem with Kubernetes seems to be one of knowing when to make the switch. My company is in a place where I think k8s would be really beneficial. We are at the 40k VMs running 1000 different types of apps. We suffer from tremendous wasted resources as a result. The company started before k8s (and even docker) were things. So getting us there seems like a real daunting (yet worth it) task. So, we are left with sort of a chicken/egg problem. It would have been unreasonable to start with k8s, yet now it is a huge cost to switch to it (even though it will save us millions).
  - brown9-2 6y ago
    It’s easy to overestimate that you’ll save money here - if you have waste, poor cost attribution, etc., you can easily do the same with Kubernetes managing the nodes and scheduling the processes. To actually save the money you have to fix those things first.
    - cogman10 6y ago
      The issue we have is that we are a very peeky computation requirements (not a steady stream of calculations going on). Further, we've got a fairly deep data processing pipeline. It's not unusual for us to see high loads on early parts of the system 1 hour and 1 hour later seeing high loads on the later parts of the system. Unfortunately, we don't currently have a way to scale out and reassign resources. That means we've built out for our peaks even though the majority of the time that hardware is sleeping. We could spend the time building out a scaling system (right now, our resources are all on prem). However, why do that when we can instead take advantage of k8s already built in scaling capabilities? So, technically not k8s that will be saving us money, but rather implementing autoscaling. However, the solution with k8s is really nice.
      - brown9-2 6y ago
        Cluster autoscaling with nodes on-prem, or in the cloud?
  - exhilaration 6y ago
- p_l 6y ago
  That has nothing to do with kubernetes, and a lot to do with management of the company. I might be biased, because my first commercial use of kubernetes was to reduce infrastructure spend and optimize the hell of what we have. From my PoV, the "big company" narrative is just what happens when the big company notices the cost of spinning separate VMs for everything.
- scoopertrooper 6y ago
  I'll have a read through your article later as I'm quite interested in hearing your point of view. However in my experience, it's pretty easy to get a reasonable Kubernetes cluster setup with stateless apps on a cloud platform managed Kubernetes service.
  - ramraj07 6y ago
    Can you share in your opinion a good single guide to do this in AWS?
    - scoopertrooper 6y ago
      I haven't worked with EKS; I've mainly worked with Azure's AKS, but the principals are largely the same. I'm not sure how familiar you are with AWS, EKS, Kubernetes, or Docker in general and I don't know of a guide that encompasses all these technologies. My best advice would be: 1) Get a little toy Docker container running locally with a http server that responds with "Hello world" 2) Try and deploy it to as a pod to a local Kubernetes cluster (Docker desktop has one built in or you can try deploying Minikube if you're feeling adventurous) 3) Add an ingress and service to your cluster, so you can access your server from a browser 4) Expand your cluster by installing something like Postgres on it and have your toy app query it and print out a query result in response to requests 5) Try passing your toy app the password to login to Postgres as a Kubernetes Secret via an environmental variable 6) Start exploring EKS specific deployment documentation such as the links below. 7) Start learning about helm https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html https://docs.aws.amazon.com/eks/latest/userguide/create-clus... https://docs.aws.amazon.com/eks/latest/userguide/sample-deployment.html https://docs.aws.amazon.com/eks/latest/userguide/sample-depl... https://learn.hashicorp.com/tutorials/terraform/eks https://learn.hashicorp.com/tutorials/terraform/eks (Once your comfortable with everything else)
      - ramraj07 6y ago
        Honestly that sounds like not simple at all. I just want a webapp up and running, with a database and cache server, and I have again and again launched Elastic Beanstalk + Elasticache + RDS (or the GCP version) in half a day following basic tutorials, and this system runs very well (and scales very well) with zero need to "dive into documentation" for various things. I can then focus on my app instead of infra. It's still not clear why such simple architectures need kubernetes.
- arderary 6y ago
  In a lot of cases it's worth considering whether orchestration is necessary at all — pure docker can take you quite far, with just a couple shell scripts called from the last stage of your CI (or manually)
- nrmitchi 6y ago
  I 100% see the point you are trying to make. However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial. One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to. > Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed. This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.
  - blandflakes 6y ago
    Helm was something that I ultimately decided to avoid at my last job, and I was happier for it. Upgrades that get in dirty states, often as a result of bogus post-install hooks. Having to fork charts to customize the deployment minorly for our mundane requirements. Reading doubly or triply templated files and dealing with bullshit indentation mismatches. I hated things that expected me to install with Helm.
    - nrmitchi 6y ago
      I can agree with you that some of uses of it get out of hand. Like most things, there is a balance. For the most part I don't use hooks, precisely because they muddy the waters of my expected end state. Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges. What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
      - blandflakes 6y ago
        A lot of the time, writing (or forking and customizing) my own manifest was at least more deterministic and predictable. Obviously you get lost at that point in time if you aren't diligent about upgrading. Helm felt like this middleground, and I really felt that at least for managed clusters, it would be better if the addon managers provided stuff that Just Worked (tm). We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language. Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).
- jayd16 6y ago
  Whats wrong with running small deploys? It takes less than half a day to set up. Its honestly way easier to get a reproducible k8s setup than some hand rolled thing even if you run a single pod. Or do you mean a small team should not use unmanaged k8s?
- derefr 6y ago
  Certainly, all the current implementations of Kubernetes are built out of enterprise-scale components — even if they're then squeezed into the girdle of a developer tool, the way microk8s is. But the design (functional architecture, APIs, etc.) of Kubernetes makes good sense for small scale as well. There's nothing fundamentally stopping someone from creating an "embedded Kubernetes" that exposes all the same resource APIs, but which is just a management interface for e.g. a JVM, or an Erlang node, where all those APIs are just controlling in-memory abstractions existing inside one userland process. Presenting itself as a "cluster" with one fixed node pool, with one fixed node, with one fixed type of container runtime which is just "spawn an actor-thread in this process." I don't disagree that people are pursuing the enterprise k8s stuff when they aren't at the right scale to need it — but I think it's important to realize that they're often doing that because they see the design of k8s as fundamentally sound, providing sensible abstractions and management interfaces that neither plain OS process APIs, nor Docker's APIs, nor IaaS-level control-plane APIs have great equivalents for. It's that design they're after; the HA stuff is just a "nice-to-have" that they end up adding on, because all the guides they read (that are written for enterprise integrators) insist it's a great idea. (Tangent: if Docker Inc. would get off their collective butts and do Docker Swarm over again, but this time with k8s-compatible APIs, I'm sure it'd become the most-used "Kubernetes" overnight. Most devs just want a regular Docker host, that they can `kubectl apply -f` against.)
- merb 6y ago
  > they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed not sure, but with 1$ million on budget, you can run a pretty pretty big gke cluster. btw. we run on e2-standard-2 and pay ~500 per month (would be a little bit more without commited usage). 6 nodes + 2 managed cloud sql databases+several load balancers. really no idea how one can pay a million even in a year. that would not be a small startup... you can probably have like 9x e2-standard-32 (32vcpu+128gb) + postgres ha 16vcpu/256gb mem/500gb ssd for like 10,867.33 per month without traffic (which might be another 5000) so with traffic you would only pay 190407.96 per year?! Edit: even without gke, k3s is way simplier than most "deployment scripts" or whatever. it's always a matter if you want to have ops people or if gke's self managing is enough. Edit2: so many lols at the blog post. I mean who the fuck spends 7000 at a colo for 40.0000 users. my macbook could probably serve that. also "and then script those older Virtual Machines with some mix of Ansible and Terraform (assuming you want to be in the cloud). " is a fucking terrible advice, been there, done that and it soon is a clusterfuck of stuff cobbeled together.
  - cwyers 6y ago
    Yeah, from the blog post: > I’ve a good friend at a startup who found a co-lo data center in a small college town in Virginia, where they were able to build out a massive MongoDB database, sharded across 4 massive servers, each server holding 256 gigabytes of RAM, 1 terabyte of RAM in total. They also had a Cassandra database and several servers for their front-end. They were paying $7,000 a month for this. Like, if you're paying AWS to host MongoDB on four EC2 machines with 256gb of RAM each and no autoscaling/load balancing, you can spend a lot of money. But... why would you do that? Porting terrible architectures to cloud VMs with no rearchitecting costs a boatload, but... don't do that?
  - lkrubner 6y ago
    "I mean who the fuck spends 7000 at a colo for 40.0000 users." You misread the article. The startup with the $1 million devops ended up with 40,000 users. The startup with the co-lo was a different startup, which in fact had a very large user base.
- ncrmro 6y ago
  You can actually set up k3os on a single node and get all the benefits eg release a new deployment and have it keep old pods up until new pods are healthy . I have it hosted on my NAS and use ansible to set up deployments and in ci. Tooling has come along way as well. But the learning curve is knarly and I picked up a few tricks from companies I worked at.
- ablekh 6y ago
  While there is some truth to what you are saying in your linked post, the real truth is more nuanced and, as usual, lies between any extremes. Firstly, when talking about extreme flexibility and experimentation, you're not talking about MVP in my book, but rather about a prototype. You can experiment with a prototype all you want, but no so much with an MVP, which, be definition, is a product, even if a minimal viable one. Secondly, your argument that MVP-stage early startups don't need HA breaks due to the corollary of MVP being a product. If your early-stage startup has, say, 5-10 important enterprise customers (those whose logos you so desperately need for initial sales as well as further marketing) using your MVP as product or even a pilot, guess how many of them will stay with you after a first 5-hour outage of your platform? Thirdly, in order to prevent disastrous exodus situations like mentioned above, an early startup does not need to have a globally scalable geographically distributed HA system (or, as you put it, "multiple servers, multiple load balancers, multiple failover databases, multiple data centers, multiple regions"). It would be quite enough to have a single level of redundancy for each major single point of failure. For a typical cloud architecture, it just means architecting, deploying and maintaining your system for/to/in two availability zones. This IMO represents a pretty good compromise between costs (both monetary and effort-wise) and risks (reputational, current and future sales, etc.). Fourthly, your suggested preference of VMs to containers will most likely lead to a) lower agility [due to spinning a VM being a much slower process versus spinning a container; that would be especially noticeable across development cycles: CI/CD etc.] and b) overspending on infrastructure [due to overhead stemming from the fact that you simply cannot pack VMs as efficient as containers into a given computational asset]. Fifthly, some/many operational aspects that have to be managed - manually or via automation such as Terraform - in non-HA scenarios could be partially or completely eliminated in cloud-based HA scenarios thanks to those aspects being embedded into relevant PaaS products, hence potentially significant cost savings and, equally or more importantly, time savings, thus, enabling a potentially much higher agility, which is so important to early-stage startups.
- sieabahlpark 6y ago
  I've managed to have really good success on using Kubernetes to actually lower my costs rather than raise them. The issue with Kubernetes is that it's a development mindset. Rather than developing persistence you develop mutations to something persistent.
- pyuser583 6y ago
  I see Kubernetes as a way to save money on web services. It allows for platform independent deployment. Moving from AWS to Google Cloud to bare metal can save massive amount of money.
- pc86 6y ago
  Serious question - what's the alternative? We have Docker containers, we want to run in swarm mode, and we want to enable service-to-service communication via gRPC. Kube seems to be the only thing that can support that unless we want to pay Traefik $20k+ which is a hard pass.
- raffraffraff 6y ago
  I agree that a startup shouldn't (yet) put a lot of time into redundancy if they don't have a product or customers. But a code pipeline that can be used to quickly develop, test and deploy is essential. There is absolutely nothing wrong with using containers. In fact, in 20 years in tech, it's the best packaging solution I've seen. As for Kubernetes, as long as you're not managing the backend, it's fine. I joined a tiny startup less than a year ago, set up EKS with Terraform and created CI automation to build and push containers when tests pass. It's not hard. And it does encourage good practices. If we had to move everything to Azure or GCE, it would be much easier to do so because we haven't painted ourselves into the AWS corner.
barnabask 6y ago
Going by blog posts, 1.17 was released 2019-12-09, 1.18 was released 2020-03-25, 1.19 was released 2020-08-26, and 1.20 was released 2020-12-08 (today). Great cadence. AWS EKS seems to support major versions six or seven months later. Support for 1.17 was announced 2020-07-13, 1.18 was announced 2020-10-13. I'm guessing we can plan for 1.20 support on EKS around June of 2021. No judgement, just an observation.
- AYBABTME 6y ago
  That cadence is also exhausting to follow, I wish they made an LTS sort of release so people can catch a break.
  - znpy 6y ago
    I feel you. Keeping up with kubernetes feels like a ratrace.
  - dolni 6y ago
    This is one of Kubernetes' biggest downfalls. On a small team, time is king. You likely don't have a team dedicated to keeping up with Kube releases, troubleshooting issues, etc. When you build something, you need it to work until _you_ decide to futz with it again.
  - rafaelturk 6y ago
    This is our pain.. the main reason why we are seeking other alternatives to K8S.
    - toomuchtodo 6y ago
      Hashicorp’s Nomad? Just hit 1.0.
      - AYBABTME 6y ago
        That's something I'd be curious about. How do you feel a later migration from Nomad to k8s would go?
ossusermivami 6y ago
I wonder when kubernetes will become commodity like docker became
- stunt 6y ago
  "mature"
- nojvek 6y ago
  It is already a commodity right ? The big cloud providers already provide managed k8s. GKE, EKS, AKS, DKS; Google, Amazon, Azure, Digital Ocean. You can even turn your devbox into k8s via minikube. It’s a great abstraction

119 points by gtirloni 6y ago

Volume Snapshots are going to be a game-changer for stateful workloads. I'm glad to see this feature in the wild. Great job Kubernetes team!

tedk-42 6y ago
Agreed! It's an abstraction on top of many existing cloud provider storage abstractions which will be a welcome feature to many places wanting a quick backup solution of their volumes.

lkrubner 6y ago

Slightly off topic, but one of my most popular essays from last year was, in part, a criticism of Kubernetes. Kubernetes is best understood as an advanced optimization that offers benefits to very large firms with thousands or tens of thousands of servers. Yet I've seen the idea spreading that every company, even a tiny startup with 3 servers, should Dockerize everything and run it all through Kubernetes. I mostly consult with small startups, so I've seen the damage these ideas can do. Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed. If anyone is interested in a case study of excess devops development, please see "High Availability is not compatible with an MVP, because an MVP is about fast iteration" http://www.smashcompany.com/technology/high-availability-is-not-compatible-with-mvp http://www.smashcompany.com/technology/high-availability-is-...

AYBABTME 6y ago
What do you recommend to these people instead of k8s, so that they can have all the convenience of orchestration and containerization?
- cogman10 6y ago
  Why containerize to start with? Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with. The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
  - p_l 6y ago
    PaaS tend to be quite expensive, especially when you're outside of USA. It's quite easy for example for Heroku to add a cost of fully loaded senior "devops" engineer to your bill, compared with k8s. Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
    - cogman10 6y ago
      While PaaS is expensive, the point really is that the cost of PaaS early on will likely be far less than the cost of properly setting up and building out a k8s cluster/deployment pipeline. Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
      - p_l 6y ago
        That sounds nice and dandy... except when PaaS increase your costs 1.5x compared to steady state (that has a lot of optimization opportunities left over), and about 6x when comparing initial conditions. And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
  - bronson 6y ago
    Lambdas get pricey quick. PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on). Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting. Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers. That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
cogman10 6y ago
The problem with Kubernetes seems to be one of knowing when to make the switch. My company is in a place where I think k8s would be really beneficial. We are at the 40k VMs running 1000 different types of apps. We suffer from tremendous wasted resources as a result. The company started before k8s (and even docker) were things. So getting us there seems like a real daunting (yet worth it) task. So, we are left with sort of a chicken/egg problem. It would have been unreasonable to start with k8s, yet now it is a huge cost to switch to it (even though it will save us millions).
- brown9-2 6y ago
  It’s easy to overestimate that you’ll save money here - if you have waste, poor cost attribution, etc., you can easily do the same with Kubernetes managing the nodes and scheduling the processes. To actually save the money you have to fix those things first.
  - cogman10 6y ago
    The issue we have is that we are a very peeky computation requirements (not a steady stream of calculations going on). Further, we've got a fairly deep data processing pipeline. It's not unusual for us to see high loads on early parts of the system 1 hour and 1 hour later seeing high loads on the later parts of the system. Unfortunately, we don't currently have a way to scale out and reassign resources. That means we've built out for our peaks even though the majority of the time that hardware is sleeping. We could spend the time building out a scaling system (right now, our resources are all on prem). However, why do that when we can instead take advantage of k8s already built in scaling capabilities? So, technically not k8s that will be saving us money, but rather implementing autoscaling. However, the solution with k8s is really nice.
    - brown9-2 6y ago
      Cluster autoscaling with nodes on-prem, or in the cloud?
- exhilaration 6y ago
p_l 6y ago
That has nothing to do with kubernetes, and a lot to do with management of the company. I might be biased, because my first commercial use of kubernetes was to reduce infrastructure spend and optimize the hell of what we have. From my PoV, the "big company" narrative is just what happens when the big company notices the cost of spinning separate VMs for everything.
scoopertrooper 6y ago
I'll have a read through your article later as I'm quite interested in hearing your point of view. However in my experience, it's pretty easy to get a reasonable Kubernetes cluster setup with stateless apps on a cloud platform managed Kubernetes service.
- ramraj07 6y ago
  Can you share in your opinion a good single guide to do this in AWS?
  - scoopertrooper 6y ago
    I haven't worked with EKS; I've mainly worked with Azure's AKS, but the principals are largely the same. I'm not sure how familiar you are with AWS, EKS, Kubernetes, or Docker in general and I don't know of a guide that encompasses all these technologies. My best advice would be: 1) Get a little toy Docker container running locally with a http server that responds with "Hello world" 2) Try and deploy it to as a pod to a local Kubernetes cluster (Docker desktop has one built in or you can try deploying Minikube if you're feeling adventurous) 3) Add an ingress and service to your cluster, so you can access your server from a browser 4) Expand your cluster by installing something like Postgres on it and have your toy app query it and print out a query result in response to requests 5) Try passing your toy app the password to login to Postgres as a Kubernetes Secret via an environmental variable 6) Start exploring EKS specific deployment documentation such as the links below. 7) Start learning about helm https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html https://docs.aws.amazon.com/eks/latest/userguide/create-clus... https://docs.aws.amazon.com/eks/latest/userguide/sample-deployment.html https://docs.aws.amazon.com/eks/latest/userguide/sample-depl... https://learn.hashicorp.com/tutorials/terraform/eks https://learn.hashicorp.com/tutorials/terraform/eks (Once your comfortable with everything else)
    - ramraj07 6y ago
      Honestly that sounds like not simple at all. I just want a webapp up and running, with a database and cache server, and I have again and again launched Elastic Beanstalk + Elasticache + RDS (or the GCP version) in half a day following basic tutorials, and this system runs very well (and scales very well) with zero need to "dive into documentation" for various things. I can then focus on my app instead of infra. It's still not clear why such simple architectures need kubernetes.
arderary 6y ago
In a lot of cases it's worth considering whether orchestration is necessary at all — pure docker can take you quite far, with just a couple shell scripts called from the last stage of your CI (or manually)
nrmitchi 6y ago
I 100% see the point you are trying to make. However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial. One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to. > Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed. This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.
- blandflakes 6y ago
  Helm was something that I ultimately decided to avoid at my last job, and I was happier for it. Upgrades that get in dirty states, often as a result of bogus post-install hooks. Having to fork charts to customize the deployment minorly for our mundane requirements. Reading doubly or triply templated files and dealing with bullshit indentation mismatches. I hated things that expected me to install with Helm.
  - nrmitchi 6y ago
    I can agree with you that some of uses of it get out of hand. Like most things, there is a balance. For the most part I don't use hooks, precisely because they muddy the waters of my expected end state. Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges. What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
    - blandflakes 6y ago
      A lot of the time, writing (or forking and customizing) my own manifest was at least more deterministic and predictable. Obviously you get lost at that point in time if you aren't diligent about upgrading. Helm felt like this middleground, and I really felt that at least for managed clusters, it would be better if the addon managers provided stuff that Just Worked (tm). We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language. Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).
jayd16 6y ago
Whats wrong with running small deploys? It takes less than half a day to set up. Its honestly way easier to get a reproducible k8s setup than some hand rolled thing even if you run a single pod. Or do you mean a small team should not use unmanaged k8s?
derefr 6y ago
Certainly, all the current implementations of Kubernetes are built out of enterprise-scale components — even if they're then squeezed into the girdle of a developer tool, the way microk8s is. But the design (functional architecture, APIs, etc.) of Kubernetes makes good sense for small scale as well. There's nothing fundamentally stopping someone from creating an "embedded Kubernetes" that exposes all the same resource APIs, but which is just a management interface for e.g. a JVM, or an Erlang node, where all those APIs are just controlling in-memory abstractions existing inside one userland process. Presenting itself as a "cluster" with one fixed node pool, with one fixed node, with one fixed type of container runtime which is just "spawn an actor-thread in this process." I don't disagree that people are pursuing the enterprise k8s stuff when they aren't at the right scale to need it — but I think it's important to realize that they're often doing that because they see the design of k8s as fundamentally sound, providing sensible abstractions and management interfaces that neither plain OS process APIs, nor Docker's APIs, nor IaaS-level control-plane APIs have great equivalents for. It's that design they're after; the HA stuff is just a "nice-to-have" that they end up adding on, because all the guides they read (that are written for enterprise integrators) insist it's a great idea. (Tangent: if Docker Inc. would get off their collective butts and do Docker Swarm over again, but this time with k8s-compatible APIs, I'm sure it'd become the most-used "Kubernetes" overnight. Most devs just want a regular Docker host, that they can `kubectl apply -f` against.)
merb 6y ago
> they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed not sure, but with 1$ million on budget, you can run a pretty pretty big gke cluster. btw. we run on e2-standard-2 and pay ~500 per month (would be a little bit more without commited usage). 6 nodes + 2 managed cloud sql databases+several load balancers. really no idea how one can pay a million even in a year. that would not be a small startup... you can probably have like 9x e2-standard-32 (32vcpu+128gb) + postgres ha 16vcpu/256gb mem/500gb ssd for like 10,867.33 per month without traffic (which might be another 5000) so with traffic you would only pay 190407.96 per year?! Edit: even without gke, k3s is way simplier than most "deployment scripts" or whatever. it's always a matter if you want to have ops people or if gke's self managing is enough. Edit2: so many lols at the blog post. I mean who the fuck spends 7000 at a colo for 40.0000 users. my macbook could probably serve that. also "and then script those older Virtual Machines with some mix of Ansible and Terraform (assuming you want to be in the cloud). " is a fucking terrible advice, been there, done that and it soon is a clusterfuck of stuff cobbeled together.
- cwyers 6y ago
  Yeah, from the blog post: > I’ve a good friend at a startup who found a co-lo data center in a small college town in Virginia, where they were able to build out a massive MongoDB database, sharded across 4 massive servers, each server holding 256 gigabytes of RAM, 1 terabyte of RAM in total. They also had a Cassandra database and several servers for their front-end. They were paying $7,000 a month for this. Like, if you're paying AWS to host MongoDB on four EC2 machines with 256gb of RAM each and no autoscaling/load balancing, you can spend a lot of money. But... why would you do that? Porting terrible architectures to cloud VMs with no rearchitecting costs a boatload, but... don't do that?
- lkrubner 6y ago
  "I mean who the fuck spends 7000 at a colo for 40.0000 users." You misread the article. The startup with the $1 million devops ended up with 40,000 users. The startup with the co-lo was a different startup, which in fact had a very large user base.
ncrmro 6y ago
You can actually set up k3os on a single node and get all the benefits eg release a new deployment and have it keep old pods up until new pods are healthy . I have it hosted on my NAS and use ansible to set up deployments and in ci. Tooling has come along way as well. But the learning curve is knarly and I picked up a few tricks from companies I worked at.
ablekh 6y ago
While there is some truth to what you are saying in your linked post, the real truth is more nuanced and, as usual, lies between any extremes. Firstly, when talking about extreme flexibility and experimentation, you're not talking about MVP in my book, but rather about a prototype. You can experiment with a prototype all you want, but no so much with an MVP, which, be definition, is a product, even if a minimal viable one. Secondly, your argument that MVP-stage early startups don't need HA breaks due to the corollary of MVP being a product. If your early-stage startup has, say, 5-10 important enterprise customers (those whose logos you so desperately need for initial sales as well as further marketing) using your MVP as product or even a pilot, guess how many of them will stay with you after a first 5-hour outage of your platform? Thirdly, in order to prevent disastrous exodus situations like mentioned above, an early startup does not need to have a globally scalable geographically distributed HA system (or, as you put it, "multiple servers, multiple load balancers, multiple failover databases, multiple data centers, multiple regions"). It would be quite enough to have a single level of redundancy for each major single point of failure. For a typical cloud architecture, it just means architecting, deploying and maintaining your system for/to/in two availability zones. This IMO represents a pretty good compromise between costs (both monetary and effort-wise) and risks (reputational, current and future sales, etc.). Fourthly, your suggested preference of VMs to containers will most likely lead to a) lower agility [due to spinning a VM being a much slower process versus spinning a container; that would be especially noticeable across development cycles: CI/CD etc.] and b) overspending on infrastructure [due to overhead stemming from the fact that you simply cannot pack VMs as efficient as containers into a given computational asset]. Fifthly, some/many operational aspects that have to be managed - manually or via automation such as Terraform - in non-HA scenarios could be partially or completely eliminated in cloud-based HA scenarios thanks to those aspects being embedded into relevant PaaS products, hence potentially significant cost savings and, equally or more importantly, time savings, thus, enabling a potentially much higher agility, which is so important to early-stage startups.
sieabahlpark 6y ago
I've managed to have really good success on using Kubernetes to actually lower my costs rather than raise them. The issue with Kubernetes is that it's a development mindset. Rather than developing persistence you develop mutations to something persistent.
pyuser583 6y ago
I see Kubernetes as a way to save money on web services. It allows for platform independent deployment. Moving from AWS to Google Cloud to bare metal can save massive amount of money.
pc86 6y ago
Serious question - what's the alternative? We have Docker containers, we want to run in swarm mode, and we want to enable service-to-service communication via gRPC. Kube seems to be the only thing that can support that unless we want to pay Traefik $20k+ which is a hard pass.
raffraffraff 6y ago
I agree that a startup shouldn't (yet) put a lot of time into redundancy if they don't have a product or customers. But a code pipeline that can be used to quickly develop, test and deploy is essential. There is absolutely nothing wrong with using containers. In fact, in 20 years in tech, it's the best packaging solution I've seen. As for Kubernetes, as long as you're not managing the backend, it's fine. I joined a tiny startup less than a year ago, set up EKS with Terraform and created CI automation to build and push containers when tests pass. It's not hard. And it does encourage good practices. If we had to move everything to Azure or GCE, it would be much easier to do so because we haven't painted ourselves into the AWS corner.

barnabask 6y ago

Going by blog posts, 1.17 was released 2019-12-09, 1.18 was released 2020-03-25, 1.19 was released 2020-08-26, and 1.20 was released 2020-12-08 (today). Great cadence. AWS EKS seems to support major versions six or seven months later. Support for 1.17 was announced 2020-07-13, 1.18 was announced 2020-10-13. I'm guessing we can plan for 1.20 support on EKS around June of 2021. No judgement, just an observation.

AYBABTME 6y ago
That cadence is also exhausting to follow, I wish they made an LTS sort of release so people can catch a break.
- znpy 6y ago
  I feel you. Keeping up with kubernetes feels like a ratrace.
- dolni 6y ago
  This is one of Kubernetes' biggest downfalls. On a small team, time is king. You likely don't have a team dedicated to keeping up with Kube releases, troubleshooting issues, etc. When you build something, you need it to work until _you_ decide to futz with it again.
- rafaelturk 6y ago
  This is our pain.. the main reason why we are seeking other alternatives to K8S.
  - toomuchtodo 6y ago
    Hashicorp’s Nomad? Just hit 1.0.
    - AYBABTME 6y ago
      That's something I'd be curious about. How do you feel a later migration from Nomad to k8s would go?

ossusermivami 6y ago

I wonder when kubernetes will become commodity like docker became

stunt 6y ago
"mature"
nojvek 6y ago
It is already a commodity right ? The big cloud providers already provide managed k8s. GKE, EKS, AKS, DKS; Google, Amazon, Azure, Digital Ocean. You can even turn your devbox into k8s via minikube. It’s a great abstraction