21 ms·

Don't use Kubernetes yet

306 points by rckrd 4y ago

Senpitio 4y ago
Use what your ops/DevOps Team know. K8s managed can easily be used by any size startup. I use it in mine because I'm the ops team as well and doing k8s for 4 years.
mkrishnan 4y ago
My advise would be exactly opposite. Dont EVER use serverless container use kubernetes instead if you need scale or not. 1. Use GKS/EKS/LKE/DKS. dont try to setup kubernetes cluster in your server by yourself. 2. It's very simple to setup deployment, database etc. it will take a week of a decent engineer to setup your application in kubernetes cluster - end to end. 3. LKE/DKS is super cheap compared to Heroku. 4. Use github actions (free) and docher hub ($5 month to build your container images) and its very easy, all you need is a weekend. 5. IMPORTANT: It's foolish to architecture your application to fit into serverless container.
- OJFord 4y ago
  Why the paid [docker] hub recommendation if already using GitHub Actions? Its container registry has a free tier too (I assume there's a limit but so is there for Actions).
  - jordemort 4y ago
    Docker Hub is all-you-can-eat with regard to image storage; GitHub's container registry is metered based on storage, and last time I tried it (over a year ago) it was also really hard to clean up old images.
    - xyzzy123 4y ago
      That does seem like one of those "too good to be true" arrangements that can fall victim to pricing updates down the track. That said, not unreasonable to take advantage of it while it lasts.
      - yjftsjthsd-h 4y ago
        Yeah, my immediate reaction was "and that's why Docker Hub will either start charging more or go out of business, both of which will be rather disruptive to your business".
    - OJFord 4y ago
      Does it not have paid tiers too though? Like Actions (which you suggested as free) also increases as you need more minutes. (On the other side, Docker Hub is I think a lot more if not completely free if the underlying repo is public? I know I have a few low use ones and don't pay a penny, anyway.) I suppose my point really is just why suggest paying for image storage intially/before CI, they can both be free.
    - molszanski 4y ago
      We did hit some magical limit on storage and transfer. And we where a paying customer. $250 or something. No warning, no emails, they turned off our our accounts. Hence no deployments. Support was silent. Now we use quay.io and a self hosted harbor docker registry.
- GauntletWizard 4y ago
  I run a quite successful SRE consultancy based on exactly this advice. You don't need all of the features of K8s. You do want enough of them that it's worth going straight to it.
- ramraj07 4y ago
  Can you elaborate on 5? Also why do I need k8s? Beanstalk plus RDS works perfect for my website needs most of the time.
  - koprulusector 4y ago
    Not OP but web apps are probably an exception to the comment. While running a web app is within Kubernetes capabilities, there are better solutions for that specific use case. Obviously, the details will vary somewhat based on the web app. A simple static site with occasional HTTP POST/PUT to server will have somewhat different requirements than a streaming media site (like say, Twitch). In the case of the former, if building in AWS, I’d choose: * Route53 * CloudFront + S3 * API Gateway + Lambda * DynamoDB (or RDS, if necessary) * Terraform + [Terraform Cloud | GitLab | GitHub Actions] * Cognito (if necessary) A website probably wants the CDN distribution, routing, and caching that Kubernetes doesn’t provide. I mean I think there are some Kubernetes @ Edge services, maybe?, but in general, Kubernetes for a website is somewhat overkill/like bringing a bazooka to a knife fight.
    - nlitened 4y ago
      > In the case of the former, if building in AWS, I’d choose: * Route53 * CloudFront + S3 * API Gateway + Lambda * DynamoDB (or RDS, if necessary) * Terraform + [Terraform Cloud | GitLab | GitHub Actions] * Cognito (if necessary) Sounds like an overkill for “a simple static site with occasional HTTP POST/PUT to server”. I’d say, get a VPS for $5/month, run your python server there with local SQLite DB. Maybe put it behind free Cloudflare if you expect HN-frontpage-level traffic.
- quickthrower2 4y ago
  AKS (Azure) is good too. Should take a day to set up the cluster unless it is the first time ever.
charles_f 4y ago
I dont get the shit k8s gets. Especially on managed offerings like Aks or fargate, they're very easy to deploy, having several environments is easy, maintainance is straight forward. As a generic rule I'm against using complex tech where there's no need, but I feel like we're way past that on k8s.
- oppositelock 4y ago
  I've used k8s in startup and now in a large company, so I've seen both worlds. In the cloud, running workloads on k8s is much more expensive than something simpler, like containers on instances in an AutoScaling group on Amazon, for example. You have to pay for the control plane in addition to the worker nodes, and if you only have a few workers, your control plane costs are significant for small infra. It's fairly easy to set up on {A,E,G}KS. This cost, however, buys you flexibility. Say your software needs to run on-prem as well as in the cloud? k8s can be your abstraction, and the additional cost becomes worthwhile. If you use it as a dumb container scheduler, without all this fancy stuff like Persistent Volumes or crazy scheduling constraints, it's not too bad. It does introduce extra complexity around traffic ingress, and is missing some basic functionality, like de-scheduling a Pod from a load balancer, and terminating it once no more connections exist. There are also difficulties around giving people access to the cluster in a safe way (for example, any admin can see the contents of secrets), and kubelets' decision to run without swap also leads to some annoying node failures that would be mere slowdowns if swap was present. Like anything else, k8s is a mixed bag. If it doesn't solve specific problems for you, don't use it, it's pointless.
  - xyzzy_plugh 4y ago
    > is missing some basic functionality, like de-scheduling a Pod from a load balancer, and terminating it once no more connections exist. There are also difficulties around giving people access to the cluster in a safe way (for example, any admin can see the contents of secrets), and kubelets' decision to run without swap also leads to some annoying node failures that would be mere slowdowns if swap was present. As written, none of this is strictly true. I suspect I know why you believe what you wrote is true, but I think you just need to spend some more time understanding the documentation and architecture. As an example, support for swap accounting was added last year, and while swap has not been supported, it's been possible to use swap perhaps forever.
aledalgrande 4y ago
my advice is use k8s if you know it, don't if you don't
- ahmed_ds 4y ago
  How can anyone tell if they know kubernetes? Most people who complain about k8s being complicated, convinced themselves they know it in the first place, and thought it is somehow going to make their life easier down the road. But it is not a plug and play tool as they hoped and unlike most tools where you can learn as you go, with k8s it becomes troubleshooting as you go. It is like when I tried to use git branches on my first git based plaintext writing project. I thought it is nice and all but when you throw merge conflicts and rebase and remote branches within the first 2 weeks, it shifted from doing the actual thing, to just troubleshooting git. With k8s that is exponentially complicated.
  - aledalgrande 4y ago
    I think you answered it yourself. If I find a lot of pain in using it it means that either I just started on it or I made a half assed attempt at learning it. Most people don't go deep. Why people with 10+ years of exp have the same depth as juniors during interviews? It's because of things like this. So my suggestion remains the same. If you reached out and really tried to understand k8s, you'll find it so powerful to work with. Otherwise, please just leave it on the shelf.
  - ramraj07 4y ago
    I doubt it’s the people using k8s wrongly that complain it’s complicated, it’s the people affected by other teams choices and have to suffer unreliable messes as a result. Like me. I can see that it’s too complicated for me, and I want nothing to do with it. The people using it often never acknowledge they’re in way over their heads IME.
- forty 4y ago
  On the other hand, if you _think_ you know kubernertes, don't use it, as it means you just have no idea ;)
sontek 4y ago
> On AWS, that would be Fargate on ECS, or on Google Cloud, Google Cloud Run. > You won't have to manage servers, network overlays, logging, or other necessary middleware. I disagree with this take. EKS is a managed service just like Fargate and you have to learn how to manage both equally (VPCs, CIDR ranges, IAM rules, etc). You might as well start on kubernetes if you are going to switch to it eventually. > I'd suggest that teams adopting Kubernetes (even the managed versions) have an SRE team, or at minimum, a dedicated SRE engineer. I'd love to hear what parts of running EKS require an SRE team and how Fargate/ECS solve that issue and make it self-serviceable.
- nijave 4y ago
  If your developers already have a basic understanding of AWS, the APIs are more similar between ECS and other AWS services. K8s introduces an unrelated API nested inside other Amazon APIs. Imo ECS service setup is a simpler interface/API and the load balancer integration is very good. K8s can go a lot of different ways depending on the type of LB and whether you opt for ingress and whether your ingress is cloud provided or runs in your cluster. Same with logging. ECS you just configure a log group and get persistent logging and basic aggregated searching with Cloudwatch Insights. K8s you get ephemeral, unaggregated logs unless you introduce additional tools. When I worked with ECS, it was originally setup by software engineers and it was scripted using the AWS SDK and worked almost exactly the same as the rest of the stack that used things like S3. Starting with k8s would require learning a new SDK/API versus some new endpoints in the one you're already working with. On the other hand, as we grew, we'd hit weird issues like ECS nodes going dead without the backplane rescheduling containers on working hosts (I think they've improved ECS agent health checks since then, though)
  - sontek 4y ago
    But if we are comparing apples to apples, you would just use AWS provided stuff for EKS right? > K8s can go a lot of different ways depending on the type of LB k8s flexibility shouldn't be counted against it here. If you are considering k8s against fargate, you should only be considering ALB / NLB ingress and not the many more ways you could. Just use what AWS provides and be happy with it :) > ECS you just configure a log group and get persistent logging and basic aggregated searching with Cloudwatch Insights You can log to cloud watch with EKS as well. Fluentd can log to cloudwatch with very little configuration. I agree that if you are already "all AWS" and just want to put one more thing in there, Fargate might match your existing patterns better. But saying "Fargate is easier than managed kubernetes" is very wrong. In general I've seen people have an easier time understanding kubernetes manifests for declaring their services instead of the equivalent terraform to get fargate up and running to do the job.
qbasic_forever 4y ago
If you don't use k8s and just run bespoke containers you still have to figure out how those containers find and talk to each other. Maybe you run some custom DNS setup, maybe you run a purpose built service discovery thing like consul, etc. And you have to figure out how you'll do networking to support public/internet-facing workloads vs. private internal services (and how each can talk to the other). But... if you just use k8s you get things like basic service discovery, networking, ingress, etc. with it and don't have to figure out bespoke solutions (that you'll just chuck anyways once you move to k8s). I do agree though I would be very hesitant to immediately dive in running stateful workloads like databases, etc. on my own k8s cluster. The cloud hosted database services that every provider has is such a significant time and complexity saver, especially if your databases are just getting started and small-ish.
- grogenaut 4y ago
  Discovery can be quite simple, we've been doing it for years. On ECS/AWS we do it with just ALB and the cluster. Turns out DNS works really well, surprising for a fundamental internet technology. Tho if it's broken it's likely DNS that is the culprit. If I need to route to other things I just configure other dns entries with SSM Parameter store... or I just manipulate the local resolvers in the VPC. Very straightforward. But I'm generally running just my software, not other people's, and consuming aws datastores. This is for work.
  - deleted 4y ago
    [deleted]
  - staticassertion 4y ago
    Does k8s not just use dns? That's what Hashicorp Nomad uses - DNS via Consul.
    - qbasic_forever 4y ago
      It does, every pod, service, etc. in the cluster has a fully qualified DNS name resolved by a DNS server in the cluster.
endisneigh 4y ago
My advice would be to try to structure your app to only use the database directly and structure all business logic around asynchronous actions that can be called via server less functions or regular services you’re hosting. IMHO this is the future of most softwares architecture. Set up proper replication for your database and you’re good. Very few companies I’ve seen need more than this in principal. In practice there’s a lot of real time stuff that really isn’t necessary increasing architectural complexity. The amount of service types that inherently need real time processing can probably be counted on a single hand
- humbleMouse 4y ago
- mikhmha 4y ago
  yup this is what we do at my work. There is data and then all business logic / asynchronous processing is done using serverless functions. Ends up being really cost effective and it can scale well. All the data exists (and setup with replication) so you can always migrate ur app to some other platform or architecture if need be. we have CI/CD pipeline that builds and deploys serverless functions using serverless containers. not a serverless enthusiast btw. microservices have their own set of issues but they can be the right fit for certain apps.
- KronisLV 4y ago
  > server less functions This is a nice idea, but many out there will need to work with on-prem infrastructure and self-hosted software. So my question is: why would be the best open FaaS solution that you can setup on your own infrastructure and use without caring too much about the underlying hardware/OS? > Set up proper replication for your database and you’re good. Another thing that I'd consider is how the DB will grow over time. Should you have a single, monolithic DB you'll end up with hundreds if not thousands of tables which will be cumbersome to work with. Personally I think that being able to use foreign keys across your entire domain is like a superpower, but some might advocate for separate DBs for separate domains, once you get that far.
- wruza 4y ago
johnwoods 4y ago
It's funny that so many startups are led to thinking that they "need" K8s
- deleted 4y ago
  [deleted]
- grogenaut 4y ago
  reminds me of when I started at a .com in 99 that failed... they felt they needed oracle and sun and cisco routers... we did everything on linux for dev work but the founder wanted to apparently be able to sell that hardware and impress investors. He got basically nothing back on any of it. They offered me a sun server for like $100 that cost $20k. I took the windows desktop instead.
  - johnwoods 4y ago
    reminds me about the need for AI/ML in random things. maybe it's to sell the "powered by AI/ML" tag. just another hypestorm
- klysm 4y ago
  I think they are led to believe it’s a good bet to make early.
throwaway787544 4y ago
The question of whether to use K8s or not is like wondering what kind of saw you should use to cut wood. There's different saws for different purposes. But even with the right saw, you still have to know how to use it correctly. Better to use a hand saw correctly than a table saw incorrectly. (you can use a table saw incorrectly, but best case the work ends up crap, worst case you lose a finger) After building infrastructure for dozens of teams, I'm quite convinced of the following: - if your people aren't very skilled, they won't build anything well. most software engineers i've seen professionally working in the cloud are handymen trying to build a wood cabinet. - if your people can't build well, it doesn't matter what technology they use. choosing between building a cabinet out of metal or cherry wood doesn't make much difference if they've never built a cabinet before. - if the first two holds: then only use the technology which requires the least skill to use well, and where the amount of maintenance is closest to zero. don't build a wood cabinet from scratch when you can buy flat pack. don't buy flat pack when you can buy an assembled cabinet, get it shipped, and carried into your office. - if using the aforementioned technology requires 'building' or 'assembling', and that is not core to the customer-facing aspect of your product, then you should not be building, you should be buying. if your business doesn't involve assembling flat pack furniture, don't ask your employees to build their own desks and chairs from Home Depot or Ikea parts. buy the premade desk and chairs, use them to make your actual product. - a software engineer knows as much about cloud architecture as a fine woodworker knows about framing. "it's all just wood" until the house takes 10x as long to frame and is 10x as expensive and still doesn't meet code. - people will try to build things they don't fully understand and leave the company before anyone realizes the mess they've made. imagine your retail store is accessible by driving a car over a wooden bridge built by a handyman.
- CoastalCoder 4y ago
  > Better to use a hand saw correctly than a table saw incorrectly. To be fair, better to use a handsaw even incorrectly than a tablesaw incorrectly.
  - Jistern 4y ago
    I concur.
hintymad 4y ago
Curious question: why don’t companies consider an abstraction like EC2? They can run k8s on EC2-like virtualization, right? I had great experience with EC2: the ability to launch thousands of machines reliably with full control to the machines, the ability to manipulate all the metadata and configuration without learning any additional shit like HCL is a huge productivity booster. The simplicity of EC2 seems a great foundation layer for us to build more advanced resource allocation. Case in point, it drives me nuts that one has to spend hours learning how to use a template system in Nomad to pass in the simplest configurations. I can’t fathom why one would be even slightly interested in learning any shit of Nomad just to deploy a god damn docker container. Don’t we have more interesting problems to solve and more general knowledge to master?
- iampims 4y ago
  EC2 instance boots in 2 to 3 minutes, a Docker container starts in seconds. You can specify precisely how much CPU/Memory you need for a container, with a VM, you're stuck with what the cloud provider offers. You still need to get your app artifacts on that VM, how do you plan to do this? EC2 (VM abstraction) is great for some type of software, but don't forget you'll also need a Load Balancer, provisioning TLS certs, security groups, autoscaling groups, VPN, public/private subnets…
  - hintymad 4y ago
    Oh, I love container-based deployment. I was just wondering if it’s worth building abstractions like EC2 that requires very little learning to use, use EC2-like systems to manage machines that host containers
    - coredog64 4y ago
      That’s ECS without the Fargate.
  - nathants 4y ago
    ec2 alpine on t3.small boots in 30 seconds. 45 seconds for spot. default-like vpc with public subnets and internet gateway is fine. zero trust. route53 health checks and returns up to 8 hosts in random order. to the gp’s point, using aws sdk to manage ec2 is shockingly easy. a lambda on a 1 minute timer with an aws sdk runs circles around autoscaling groups et al.
didip 4y ago
The organization must have a need first. And then, after you have a legitimate need, use a hosted Kubernetes solution. Don't roll your own. It's really that simple.
andrewallbright 4y ago
We should choose our technology stack like a hermit crab chooses it's shell. The shell shouldn't be too heavy to move around in but with a little room to expand in. When it's time, we work to find a new shell.
- CharlesW 4y ago
  This is a delightful (and wise, IMO) way to think about right-sizing dependencies.
- papito 4y ago
  I love it.
MarquesMa 4y ago
The problem is there are no sane in-between options. On one end of the spectrum are either neat platforms like Heroku or Vercel or ssh and bear-metal with simple scripts. On the other end of the spectrum, we have Kubernetes. Everything in between: - The learning curve is much steeper than Heroku and Vercel - The skill is not likely to transfer to the next job - The ecosystem is not as complete as Kubernetes Most mid-sized companies went for Kubernetes because the in-betweens are not very optimal and need to take some risks on betting them.
- nathants 4y ago
  i find managing aws from a lambda with an aws sdk to be a good in between. trigger the lambda on events and/or a schedule. lock around dynamodb to ensure a single lambda at a time is mutating aws. example: https://github.com/nathants/libaws/tree/master/examples/complex/s3-ec2 https://github.com/nathants/libaws/tree/master/examples/comp... knowledge and intuition about aws primitives is definitely a transferable skill.
- zmmmmm 4y ago
  I think docker-compose / docker swarm are sane inbetweens. They only do half of what Kubernetes does but it's the half you need at smaller scale.
  - icod 4y ago
    I'd have to disagree. I tried docker compose and swarm. nothing but issues. I went back to old fashioned bash
- KronisLV 4y ago
  > On one end of the spectrum are either neat platforms like Heroku or Vercel or ssh and bear-metal with simple scripts. > On the other end of the spectrum, we have Kubernetes. Someone else mentioned Docker Swarm, but allow me to offer my own thoughts. The simplest option (what you allude to) is probably running containers through the "docker run" command, which can sometimes work for a limited set of circumstances, but doesn't really scale. For single node setups, Docker Compose can also work really nicely, where you have a description of your entire environment in a YAML file and you can "orchestrate" as many containers as you need, as long as you don't need to scale out. The aforementioned Docker Swarm is a simple step up that allows you to use the Compose syntax (which is way easier than what K8s would have you use) but orchestrate containers across multiple nodes and also has networking built in. It's simple to set up (already comes preinstalled with Docker), simple to use and maintain (a CLI like that of kubectl, but smaller), the performance and resource usage is small and the feature set is stable. It's an excellent option, as long as you don't need lots of integrations out there and whatever Docker offers is enough. From there, you might look into the likes of Hashicorp Nomad, though their HCL is a bit more complicated than the Compose format, though still simpler than Kubernetes. The setup is a bit more complicated (if you need Consul and TLS encrypted traffic) but overall it's still just a single binary that can be setup as a client/server depending on your needs. Also, as an added bonus, you can also orchestrate other things than containers, much like how back in the day there was Apache Mesos which supported different types of workloads (e.g. you can launch Java apps or even native processes on nodes that you manage, not just containers). Of course, even when you get into Kubernetes, there are also projects like K3s and k0s, maybe with tools like Portainer or Rancher, which let you manage it in a simpler manner, either through an easy to use UI or with those K8s distributions being slightly cut down, both in the plugins that they come with, as well as their resulting resource usage and data storage solutions (e.g. use SQLite instead of etcd for smaller deployments). In my eyes, betting on OCI is a pretty reasonable option and it allows you to run your containers on whatever you need, depending on what your org would be best suited to. > The skill is not likely to transfer to the next job I'd argue that if you need to read a book to figure out how services are deployed in any given environment, then you probably should have a DevOps/Ops team and not have to worry about it as a dev. Or, if you're a part of said team, you should still mostly just use OCI containers under the hood and the rest should be much like learning a new programming language for the job (e.g. like going from Java to .NET, which are reasonably similar). > The ecosystem is not as complete as Kubernetes Kubernetes largely won the container wars. No other toolchain will ever have as complete of an ecosystem, but at the same time you also dodge the risks of betting on some SaaS solution that will milk your wallet dry with paid tiers and will fold a few years down the line. I'd say that all of the aforementioned technologies support all of the basic concerns (deployments, monitoring, storage, resource limits etc.).
preommr 4y ago
My 2c: Start with basic k8s. Just getting a couple of apps deployed on a cluster with logging, scaling, pipelines takes ~ 2 weeks. Trying to rebuild that basic functionality using ad hoc solutions takes way more time and rapidly becomes more cumbersome as the wheel gets reinvented when k8s as a platform has well documented solutions/tools. Putting it into production leads to a handful of footguns that take a while to sort out related to pull-policies, caching, scaling, security, etc. But it's fairly manageable. And probably easier to premempt than customized solutions since these pitfalls are somewhat well documented. Past a certain point though, especially as the work veers into things like operators, sketchy helm packages, service meshes, k8s falls apart fast if you don't have people on it full time, and it's much better to write some customized code.
- mountainriver 4y ago
  100% I’ve seen this happen in my current organization as they chose ECS, and then reinvented half of kubernetes in house in a fragile way. Basic kubernetes where you are just deploying a web app is incredibly simple. There is no more complexity than any other container deployment tool, but Kube can grow with you like few other tools can.
- icod 4y ago
  > Trying to rebuild that basic functionality using ad hoc solutions takes way more time and rapidly becomes more cumbersome as the wheel gets reinvented when k8s as a platform has well documented solutions/tools. That is just not true. 1 day if you know what you're doing. If you use a well performing language you won't even need scaling for a very long while, unless you expect like "Eurovision Song Contest" type of concurrent users.
mt42or 4y ago
Don't use AWS yet. Cloud setup is so complex that you should avoid using it.
- jalk 4y ago
  Exactly - this discussion sound exactly like the ones we had when AWS started out
  - nathants 4y ago
    aws is fine. just use the good parts, skip the rest.
likortera 4y ago
I've been at several companies. The ones where things were the smoothest were using App Engine, other was using Tsuru, and the other was using heroku. Zero problems. The companies where we suffered the most, had to fight a lot to get anything shipped, and we where expected to understand a custom in house jungle of yaml files and script and half of the features we needed from the platform were half assed were using Kubernetes. That's just my experience.
- mrweasel 4y ago
  Comparing App Engine and Kubernetes makes it feel like we took a step backwards. Rather than just deploying code and have everything managed, now I suddenly have to worry about networking and load balancers again.
etaioinshrdlu 4y ago
I figure that Kubernetes will eventually become very mature and stable, the rate of changes will slow down, and it will become a predictable building block like Linux is now. I'm personally choosing to avoid using it for now.
jschrf 4y ago
I never quite got the K8S "it's too complex" hate, but to be fair I haven't scaled it very high. Sure, it's verbose and there are N levels of abstraction, but it's a declarative API for running foo across multiple environments of bar. I've always wanted this. I like raw, versioned infrastructure config with no extra crap. I have a little K8S.yml snippet I copy+paste+tweak into repos when I want to throw an ad-hoc experiment into a cluster, and then a bigger setup for IRL projects that looks something like this: - k8s - base - api.yml - web.yml - worker.yml - namespace.yml - ingress.yml - overlays - dev ... config to merge ... - staging ... config to merge ... - production ... config to merge ... - shared - ... variable declarations, base config maps, etc ... Everything gets merged into a manifest.yml and version stamped and build-artifacted. Deployment just means applying the config overlays via kustomize based on environment and then pushing out. If things break, I always have an absolute, pull-the-chute, versioned, formal safe point to go back to: kubectl -n production apply -f manifest.version.yml
- deleted 4y ago
  [deleted]
jsdevtom 4y ago
What about if you need rolling deployments?
- nijave 4y ago
  All you need for rolling deployments is >1 instance and load balancing. You can use DNS load balancing which isn't very nuanced, a local load balancer per machine (like nginx with 2 backend on the same machine) or more complex load balancing systems
benreesman 4y ago
Oh man, my org at FB was the test bench for moving to containerization and fancy service discovery and all that (Tupperware, just a different Borg reimplementation). It eventually got pretty good, but it never stopped being mad overkill for single-thousands of boxes. When you’re in the 10s or 100s of thousands in a fleet, or when you’ve got workloads that don’t neatly slot into your SKUs, Kube/Borg/TW/Mesos are The Way. No doubt about it. But it’s always seemed zany to me to stack namespaces/cgroups/etc. on top of Xen or whatever EC2 is using. Yo dawg I heard you like an abstract machine so I put… There are just separate concerns: - reproducibility (shared libraries argggghh) - resource limits to bin-pack SKUs - service discovery - failover - operational affordance And I’ve seen it get so ugly to conflate these very different imperatives. Running 100Ks that need to web serve on demand but web index when idle? Yeah, now you need the whole enchilada. But it’s false and harmful to promote the idea that the minute you need Grafana or DNS or Salt/Ansible/Nix/whatever that you need BORG. There are scenarios where I would enthusiastically break out Kube, but most of the marketing around it falls into my “and you will do nothing, because you can do nothing.” bucket.
- robertlagrant 4y ago
  It's really hard to break through the fog of metaphors here. Are you saying that below 10000 boxes it's worth doing something else to deploy on to them than k8s? What would that be?
jansommer 4y ago
I was in a startup with a small team and we used Kubernetes for servers that required more than 8 GB memory. Cloud Run and App Engine doesn't offer more than that, at least not at the time. The alternative were to deal with virtual machines ourself with Ansible scripts, and I'm not sure how that would auto scale + it would break the existing flow of Docker containers for everything. It took a while to figure out how to put Kubernetes in a closed VPC, but after that it was fairly straight forward.
- davidspiess 4y ago
  As far as i know Cloud Run offers now up to 32 GB memory workloads
yibers 4y ago
The title is actually very confusing and so is the article. After reading it a few times I finally understood that the author is actually pro Kubernetes. He is just saying that if you are in the really early stages of your startup, don't use Kubernetes right away. You will probably want to use it eventually.
nunez 4y ago
i would have agreed with OP three years ago when kubernetes was very niche and setting it up was difficult. today, kubes is very easy to get going with. you can set it up locally with kind or k3s for local dev, and use a cloud vendor's flavor in production. last time i tried to get a local dev env working for lambda, i spent a lot of time hacking on runtime-level stuff. it was not pleasant. additionally, the market of devs and operators fluent in it has grown by a lot. many people are getting their CKx certs, and there are enough companies using it now to create reliable supply. i say this because the lift from "my app is working in docker" to "my app is working in kubernetes" is much smaller than it used to be. given that OP is suggesting container runtimes as the alternative (which can get very expensive; much more so than using kubernetes for everything), i think that if a business is at a point where they are containerizing to accelerate releases, then kubernetes is a natural next step. anything else in between is at best a costly dependency and at worst throwaway.
te_chris 4y ago
GKE autopilot deserves a mention. If you’re going to go k8s, then it’s pretty close to noops
dig1 4y ago
My 2c: you don't need K8s unless you are google-scale. Even if you think you are google-scale, you are not. Maintenance-wise and $$-wise, two VPS boxes with Cloudflare (even with an enterprise account) setup is usually cheaper than an ordinary K8s setup. But again, it all depends on your use case, and people usually overestimate their use cases and the company's growth.
- KronisLV 4y ago
  > Maintenance-wise and $$-wise, two VPS boxes with Cloudflare (even with an enterprise account) setup is usually cheaper than an ordinary K8s setup. How do you feel about the benefits of using containers? And if you do use containers, how do you feel about the benefits of orchestrating them, instead of running something like Docker Compose? E.g. having overlay networks and being able to deploy a new version of your software on multiple nodes simultaneously? I've found that even when you don't want K8s, something like Docker Swarm or Hashicorp Nomad may still have benefits for you, depending on what you're trying to do. Swarm is basically feature-complete, boring and just like adding multi-node capabilities on top of Docker Compose anyways. Nothing to install apart from Docker and nothing to configure, apart from executing a cluster init/join command.
  - dig1 4y ago
    My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I haven't mentioned Swarm intentionally, which IMHO I consider "the next iteration" of your containerized setup: Swarm is a boring, just-works, tool if you need to spread multiple containers, add some networking between them, and have to manage everything by a normal person. And boring is good :)
    - KronisLV 4y ago
      > My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I agree that Ansible is an excellent tool! Though personally, I enjoy establishing a clear boundary between the "infrastructure" and "app" parts of any setup - the former ensuring that OS has whatever runtimes are necessary for any given application to run, user accounts, groups, folders, services etc., whereas the latter is whatever apps are running on the server. So I'd use Ansible for most of the former, setup a container cluster and then use something like https://docs.ansible.com/ansible/latest/collections/community/docker/docker_swarm_module.html#ansible-collections-community-docker-docker-swarm-module https://docs.ansible.com/ansible/latest/collections/communit... to manage actual application deployments. Why? To make the apps more throwaway and limit the fallout in case of bad configurations/deployments, as well as make sharing the base Ansible playbooks for what constitutes a production ready server setup easier amongst projects. Of course, you could as well use Ansible to install something like JDK/Tomcat and set up its configuration which worked well for me in the past, but personally that didn't scale quite as well as running Java/Node/Ruby/Python/PHP in containers and approaching them with Ansible almost like black boxes (e.g. copy a bunch of files, like secrets, into these directories, deploy some arbitrary YAML against the Docker Swarm cluster), which was surprisingly easy to do.
goncaloo 4y ago
Kubernetes are amazing at what they do, but only relevant to ~0.1% of the companies in my opinion. It's way too complex and too much work for the rest of the world, and not worth the time invested. A lot can be accomplished with simple virtual machines and some sort of auto scaling groups (depending on your cloud provider they have different names). Kubernetes are amazing at unifying your workloads on any clouds though. If you care about portability, you should either consider using kubernetes for everything or using a tool that abstracts your configuration in a cloud-agnostic way. Although I'm a bit biased on this one.
- mmcnl 4y ago
  Simple is subjective and dependent on where you're coming from. If you're used to Kubernetes then managing virtual machines seems needlessly time consuming and vice versa.
jbverschoor 4y ago
Well, if you're using it for development, it doesn't make sense.. You're probably doing all the microservices, which should mean that those services are the responsibility of another team. It also means they should have a testing / staging / development version online somewhere. If you're developing MS Paint.. do you really need to have to compile windows and all the dependencies?
- nix23 4y ago
  >You're probably doing all the microservices, which should mean that those services are the responsibility of another team. You can't imagine how many time i had to tell customers what micro-services are. Micro-service-architecture are teams who declare a common interface, and then can change their service without informing other teams and the whole system still works. Declaring a Micro-service-architecture need's a lot of time and highly experienced interface planer's (in a perfect world those should never change).
nickdothutton 4y ago
I’m much more interested in creating the simplified replacement technology for K8.
RiyaadhABR 4y ago
Running Caprover on a VPS has been a very nice alternative to full K8s. It is like your mini Heroku but not too much magic involved. A lightweight wrapper around Docker containers but comes with a nice GUI and handles networking between your applications. the One Click apps are also very useful for quickly spinning up databases and stuff like that.
bradwood 4y ago
My startup just bypassed all this container stuff and went straight for AWS Serverless. If you design it well, it works excellently. If/when we need long-running workloads we'll go to containers but thus far we're just rocking out with lambda, and sns/sqs and Eventbridge.
chrismarlow9 4y ago
Containerization solves a team problem where the system dependencies for an application need to be in control by the team, and allow operations people to specifically focus on the infrastructure supporting the application. From the technical side you can accomplish nearly all of the same goals using machine images and something like packer combined with any config management tool. I guess what I'm saying is you should use containerization when the complexity of your application and complexity of your infrastructure is too high for an operations (DevOps) person to deal with. Or when it changes so frequently it's impossible to keep up with the specific application needs. An example is some poor DevOps engineer who has to maintain terraform scripts for the infrastructure but also needs to know the version of python used in application XXX or the postgres header libs are required for YYY. And a team of 30+ application devs are changing this constantly. It's a burden and a risk to require a DevOps engineer to remember and maintain all of this. So you start looking to docker so this responsibility can be handed off to the team that owns the application. So in short if you're a small startup and have 1 or 2 DevOps guys you'll probably be okay with a very simple system of building machine images. As the complexity grows this handoff of machine requirements can be handed over to the teams by using docker. And if you do this properly by abstraction the system build code away through makefiles or bash scripts the transition from machine images to dockerfiles is pretty straight forward and easy. Possibly as easy as creating a packer file that will build the docker image instead of machine images. Kubernetes is just a tool for the operations people to manage the containers. I guess what I'm saying is if you can't automate properly with basic machine images, you should really tackle that first. And that containers solve a team logistics problem, not a technical one.
BossingAround 4y ago
As a person that is deeply involved in Kubernetes and Istio, I'm starting to get the feeling that in the beginning, it should be totally acceptable to run containers from Docker Swarm. If your main need is "restart container on error" (which seems likely for startups), you probably can't beat the fast and easy deploy time of Swarm. Also, when the time comes to upgrade to Kubernetes, you won't be locked in to some incompatible solution. Of course, if you care about scaling, serverless is probably the way to go.
- mwaitjmp 4y ago
  Is docker swarm still maintained?
  - BossingAround 4y ago
    Ah sorry, I meant Docker Compose, not Docker Swarm.
roxaaaane 4y ago
use K8s if you need it, don't use K8s if you don't, it's that simple ¯\_(ツ)_/¯. It's not rocket science you don't need to read every week's opinion on K8s and you don't need to write one either.
- klysm 4y ago
  I’m not sure the question of “will k8s save me time” is particularly easy to answer.
Too 4y ago
In theory what the article suggest looks sound and easy. I have to strongly disagree though after having the unfortunate experience of hands on trying Azures managed container runtime where everything about it was just plain misery: Getting logs out, updating the running container, connecting to storage, all kinds of esoteric settings hidden under even more strange abstractions, lack of documentation and experience online, strange edge cases everywhere. You still have to understand all the complex cloud stuff like networks, ingress and storage accounts, etc. We changed to managed kubernetes instead and even for a team with no prior experience it was much smoother. Fast declarative deployments, logs, attach, everything is just one instant kubectl command away. Documentation, blogs and resources are in excess so you will always find a way out. There are still some things that are difficult but I attribute most of that to the cloud generally. As someone else said, use it if you already know it and know what you are getting into. If you don’t, be prepared for some initial bumps and surprises. It has its warts but it’s not nearly as bad as some people portray it, those cases are likely more a problem with a micro service architecture gone out of control rather than k8s itself. Sadly there is no middle ground alternative. Closest would be docker-compose, assuming someone runs the VM for you and that still comes with a lot of hassle of deploying files to that VM and you still need to configure all the cloud networking.
- pid-1 4y ago
  AWS ECS isn't much better. It's weird Cloud Providers don't see DevX as a priority, they are at serious risk of becoming commodities due k8s and likes.
- alexalx666 4y ago
  Docker compose on VPS sounds much better than going straight to Azure/AWS
naiv 4y ago
Could someone eli5 what 1e0, 1e1 , 1e2 stand for?
- treis 4y ago
  Maybe I'm salty this morning, but I don't think people should take advice from someone that used more characters to write something in a less clear way.
  - rgaino 4y ago
    I'm with you.
- ocharles 4y ago
  1en = 10 ^ n. So 1e0 = 10 ^ 0 = 1, 1e1 = 10 ^ 1 = 10, 1e2 = 10 ^ 2 = 100, etc.
  - naiv 4y ago
    Thank you!
eof 4y ago
I’d just say “don’t learn k8s yet.“ if you know it, it’s fine, but it’s complicated to get right, so delay if you haven’t made by that journey yet.
darthrupert 4y ago
(2025)
vbezhenar 4y ago
It's funny to read this article and thread while doing exactly what everyone suggests to avoid: building Kubernetes on bare virtual metal for a team with few programmers without any dedicated devops or SRE roles. The reason I'm doing it is because our business owner thinks that we need scalability and high availability. We have law obligations to keep our data inside a country. And we don't have any managed Kubernetes offerings inside our country. The best cloud stuff I've found is hoster with openstack API and that's what I'm building upon. I thought really hard about going with just docker swarm, but it seems that this tech is dying and we should rather invest into learning Kubernetes. Honestly so far I spent few weeks just learning Kubernetes and few days writing terraform+ansible scripts and my kubernetes cluster seems to work good enough. I didn't touch storage part yet, though, just kubeadm-installed kubernetes with openstack load balancer, calico network and nginx ingress. I guess hard part will come with storage stuff. Worst thing is: everyone talks about how hard it is to run Kubernetes on bare metal, yet nobody talks about what exactly issues are and how to avoid them.
- mrweasel 4y ago
  For some projects there's no way around it, you have to build Kubernetes on bare metal/virtual machines. I've faced the same issues, the project called for Kubernetes, we can debate that requirement, but it was specified in the contract. Some project simply requires you to build Kubernetes on-prem, mostly for legal or political reasons. I do question the logic of picking Kubernetes for scalability in those projects though. To make that work, you end up with a lot of excess capacity, unless you can scale down one workload, while scaling up another. E.g. scaling down a website or API at night and use the capacity for batch jobs. Honestly building the cluster isn't my main concern, that pretty easy, I managed to write Ansible code for deploying a cluster in less than a day. My main concern is debugging and maintenance long term. Reading about companies that spin up a new cluster, because it's easier than figuring out while the old one broke, is an indication that Kubernetes might not be completely ready for production use.
  - p_l 4y ago
    Mixed use is, arguably, where k8s shines the most. As in, you have a pool of hardware, and you want to optimize use of its capacity. Mix interactive jobs work batch jobs. Maybe even prod and non-prod. Fit as much as possible on smallest amount of nodes, etc.
AtNightWeCode 4y ago
As a startup it is very important to spend as much time and money as possible on the features not the tech. If the tech is not the feature that is. Most startups should build one single monolith API running on some default offering in the cloud. Serverless services can be used as compliment for some workloads. Building, running, and maintaining containers is unnecessary for this and it does slow down the dev process.
zoomzoom 4y ago
At withcoherence.com, we agree that leaning on managed runtimes for as long as possible makes a ton of sense. I’ve also seen that hiding the complexity of transforming code into deployed containers by fully abstracting away dockerfiles and CI/deploy scripts can lead teams into a tough spot at a bad time to learn what’s really happening. But the appeal of k8s is often the ecosystem of tools that solve real problems these runtimes leave on the table: managing multiple environments, load balancing across services, SSL/TLS, SSH, long-running tasks, managing multiple versions, integrating tests into pipelines. Coherence is working to solve these problems and create a great developer experience without hiding what’s really going on under the hood. (Disclosure, I’m a cofounder)
munchenphile 4y ago
Startups should almost never use k8s. They need to iterate fast and ignore the complexities of infra. k8s is far too complex for most small companies. CapRover Droplet on Digital Ocean + deploy your Rails app with git. Scale your single VPS up as needed. Most don’t need much beyond that for quite a while.
- nouveaux 4y ago
  I have never heard of CapRover and I dont know how to use it. I do know how to use Kubernetes across Digital Ocean, AWS, GCP and Azure. Just had to learn it once. I don't even use it for scaling. Not complex at all. I've used it for four years. Literally never had a problem with it. It just runs.
txtai 4y ago
Interesting article, thank you for the insights.
theptip 4y ago
I agree with the initial advice to use a simple Docker container runner hosted service. I initially ran my Django app as a bare Docker container behind an Nginx reverse proxy, with a “docker pull; docker stop <old>; docker run <new>” script as my deploy job, and that was fine for a year. Took all of an hour to build the plumbing there. A hosted service would have been just as good and probably even less time to wire up. I disagree with the OOM requirements for operating a k8s cluster. I ran our infra on GKE from 3 engineers through 15 as the primary Infra engineer (while also pushing code and being CTO), and it was hours-per-month of labor, not a dedicated SRE. I trained up some of the other engineers and they were able to help with on call after a few hours of training. For a simple app (a few deployments and services) it is really not hard to work with. All that said I agree you don’t _need_ it at 5-person scale. I would not recommend you learn it if you don’t already know how to use it. But if you do already know it, you can get good value from using it much earlier than the article recommends. (For example I found Review Apps to be very useful for fostering collaboration between frontend and backend engineers, and that feature is not too hard to wire up on top of k8s.) If I had to give one-sentence pithy advice I’d probably agree with the OP title.
markstos 4y ago
There was a time when conventional wisdom was that we all needed to be using XML data exchange, all the time. Now the simpler format JSON dominates. I hope Kubernetes ends up being the next XML.
desktopninja 4y ago
Might never need kubernetes: https://stackexchange.com/performance https://stackexchange.com/performance
- p_l 4y ago
  Stack Overflow has been increasingly using Kubernetes for last few years, though.
yetanotherjosh 4y ago
k8s is just one component of the whole architecture, yet people talk about it like it's the singular defining characteristic. It reminds of people talking about building "react apps" when react is one of probably hundreds of essential library dependencies and says nothing about 80% of the tooling. In my rig that I manage in a startup of 2 people, k8s (managed EKS on AWS) is the part that generally just works with very little effort. I provision and deploy to it with terraform and helm. The cluster itself is cattle. I can spin up a second identical cluster with a config file, cut dns over, then spin the old one down. It took around 3 weeks to get everything setup, but k8s was not the hard part, it was making all the OTHER decisions and integrations for things like container building, monitoring/logging/apm, secrets management, setting up a VPC correctly, writing some custom config scripts to generate the right setup for the 2 separate apps we run in both staging and production, etc. The work was undoubtedly far greater outside the k8s domain. In fact when it came to the k8s parts, e.g. defining services, ingress, etc, I was generally relieved and pleased. And now that I've done all this, I feel comfortable repeating it. Things run quite well and I have zero pressure to migrate.
- curious_cat_163 4y ago
  I think the article is congruent with your thinking — k8s shapes those OTHER decisions in critical and may in some cases be the reasons why those decisions exist in the first place. To me, the insight of the article was questioning whether a team of your size should be needing to make those decisions? It may very well be that your case requires k8s. But, I can think of many small shops that would benefit from keeping things simple and paying someone (a cloud provider) in the beginning of their journey and slowly adding complexity as success comes along.
systemBuilder 4y ago
Am I supposed to believe these blanket statements from a person who doesn't realize 1e0 = 1e1 = 1e2 = ... = 1 ?
- klysm 4y ago
  I thought 1e2 = 1 * 10^2

306 points by rckrd 4y ago

Use what your ops/DevOps Team know. K8s managed can easily be used by any size startup. I use it in mine because I'm the ops team as well and doing k8s for 4 years.

mkrishnan 4y ago

My advise would be exactly opposite. Dont EVER use serverless container use kubernetes instead if you need scale or not. 1. Use GKS/EKS/LKE/DKS. dont try to setup kubernetes cluster in your server by yourself. 2. It's very simple to setup deployment, database etc. it will take a week of a decent engineer to setup your application in kubernetes cluster - end to end. 3. LKE/DKS is super cheap compared to Heroku. 4. Use github actions (free) and docher hub ($5 month to build your container images) and its very easy, all you need is a weekend. 5. IMPORTANT: It's foolish to architecture your application to fit into serverless container.

OJFord 4y ago
Why the paid [docker] hub recommendation if already using GitHub Actions? Its container registry has a free tier too (I assume there's a limit but so is there for Actions).
- jordemort 4y ago
  Docker Hub is all-you-can-eat with regard to image storage; GitHub's container registry is metered based on storage, and last time I tried it (over a year ago) it was also really hard to clean up old images.
  - xyzzy123 4y ago
    That does seem like one of those "too good to be true" arrangements that can fall victim to pricing updates down the track. That said, not unreasonable to take advantage of it while it lasts.
    - yjftsjthsd-h 4y ago
      Yeah, my immediate reaction was "and that's why Docker Hub will either start charging more or go out of business, both of which will be rather disruptive to your business".
  - OJFord 4y ago
    Does it not have paid tiers too though? Like Actions (which you suggested as free) also increases as you need more minutes. (On the other side, Docker Hub is I think a lot more if not completely free if the underlying repo is public? I know I have a few low use ones and don't pay a penny, anyway.) I suppose my point really is just why suggest paying for image storage intially/before CI, they can both be free.
  - molszanski 4y ago
    We did hit some magical limit on storage and transfer. And we where a paying customer. $250 or something. No warning, no emails, they turned off our our accounts. Hence no deployments. Support was silent. Now we use quay.io and a self hosted harbor docker registry.
GauntletWizard 4y ago
I run a quite successful SRE consultancy based on exactly this advice. You don't need all of the features of K8s. You do want enough of them that it's worth going straight to it.
ramraj07 4y ago
Can you elaborate on 5? Also why do I need k8s? Beanstalk plus RDS works perfect for my website needs most of the time.
- koprulusector 4y ago
  Not OP but web apps are probably an exception to the comment. While running a web app is within Kubernetes capabilities, there are better solutions for that specific use case. Obviously, the details will vary somewhat based on the web app. A simple static site with occasional HTTP POST/PUT to server will have somewhat different requirements than a streaming media site (like say, Twitch). In the case of the former, if building in AWS, I’d choose: * Route53 * CloudFront + S3 * API Gateway + Lambda * DynamoDB (or RDS, if necessary) * Terraform + [Terraform Cloud | GitLab | GitHub Actions] * Cognito (if necessary) A website probably wants the CDN distribution, routing, and caching that Kubernetes doesn’t provide. I mean I think there are some Kubernetes @ Edge services, maybe?, but in general, Kubernetes for a website is somewhat overkill/like bringing a bazooka to a knife fight.
  - nlitened 4y ago
    > In the case of the former, if building in AWS, I’d choose: * Route53 * CloudFront + S3 * API Gateway + Lambda * DynamoDB (or RDS, if necessary) * Terraform + [Terraform Cloud | GitLab | GitHub Actions] * Cognito (if necessary) Sounds like an overkill for “a simple static site with occasional HTTP POST/PUT to server”. I’d say, get a VPS for $5/month, run your python server there with local SQLite DB. Maybe put it behind free Cloudflare if you expect HN-frontpage-level traffic.
quickthrower2 4y ago
AKS (Azure) is good too. Should take a day to set up the cluster unless it is the first time ever.

charles_f 4y ago

I dont get the shit k8s gets. Especially on managed offerings like Aks or fargate, they're very easy to deploy, having several environments is easy, maintainance is straight forward. As a generic rule I'm against using complex tech where there's no need, but I feel like we're way past that on k8s.

oppositelock 4y ago
I've used k8s in startup and now in a large company, so I've seen both worlds. In the cloud, running workloads on k8s is much more expensive than something simpler, like containers on instances in an AutoScaling group on Amazon, for example. You have to pay for the control plane in addition to the worker nodes, and if you only have a few workers, your control plane costs are significant for small infra. It's fairly easy to set up on {A,E,G}KS. This cost, however, buys you flexibility. Say your software needs to run on-prem as well as in the cloud? k8s can be your abstraction, and the additional cost becomes worthwhile. If you use it as a dumb container scheduler, without all this fancy stuff like Persistent Volumes or crazy scheduling constraints, it's not too bad. It does introduce extra complexity around traffic ingress, and is missing some basic functionality, like de-scheduling a Pod from a load balancer, and terminating it once no more connections exist. There are also difficulties around giving people access to the cluster in a safe way (for example, any admin can see the contents of secrets), and kubelets' decision to run without swap also leads to some annoying node failures that would be mere slowdowns if swap was present. Like anything else, k8s is a mixed bag. If it doesn't solve specific problems for you, don't use it, it's pointless.
- xyzzy_plugh 4y ago
  > is missing some basic functionality, like de-scheduling a Pod from a load balancer, and terminating it once no more connections exist. There are also difficulties around giving people access to the cluster in a safe way (for example, any admin can see the contents of secrets), and kubelets' decision to run without swap also leads to some annoying node failures that would be mere slowdowns if swap was present. As written, none of this is strictly true. I suspect I know why you believe what you wrote is true, but I think you just need to spend some more time understanding the documentation and architecture. As an example, support for swap accounting was added last year, and while swap has not been supported, it's been possible to use swap perhaps forever.

aledalgrande 4y ago

my advice is use k8s if you know it, don't if you don't

ahmed_ds 4y ago
How can anyone tell if they know kubernetes? Most people who complain about k8s being complicated, convinced themselves they know it in the first place, and thought it is somehow going to make their life easier down the road. But it is not a plug and play tool as they hoped and unlike most tools where you can learn as you go, with k8s it becomes troubleshooting as you go. It is like when I tried to use git branches on my first git based plaintext writing project. I thought it is nice and all but when you throw merge conflicts and rebase and remote branches within the first 2 weeks, it shifted from doing the actual thing, to just troubleshooting git. With k8s that is exponentially complicated.
- aledalgrande 4y ago
  I think you answered it yourself. If I find a lot of pain in using it it means that either I just started on it or I made a half assed attempt at learning it. Most people don't go deep. Why people with 10+ years of exp have the same depth as juniors during interviews? It's because of things like this. So my suggestion remains the same. If you reached out and really tried to understand k8s, you'll find it so powerful to work with. Otherwise, please just leave it on the shelf.
- ramraj07 4y ago
  I doubt it’s the people using k8s wrongly that complain it’s complicated, it’s the people affected by other teams choices and have to suffer unreliable messes as a result. Like me. I can see that it’s too complicated for me, and I want nothing to do with it. The people using it often never acknowledge they’re in way over their heads IME.
forty 4y ago
On the other hand, if you _think_ you know kubernertes, don't use it, as it means you just have no idea ;)

sontek 4y ago

> On AWS, that would be Fargate on ECS, or on Google Cloud, Google Cloud Run. > You won't have to manage servers, network overlays, logging, or other necessary middleware. I disagree with this take. EKS is a managed service just like Fargate and you have to learn how to manage both equally (VPCs, CIDR ranges, IAM rules, etc). You might as well start on kubernetes if you are going to switch to it eventually. > I'd suggest that teams adopting Kubernetes (even the managed versions) have an SRE team, or at minimum, a dedicated SRE engineer. I'd love to hear what parts of running EKS require an SRE team and how Fargate/ECS solve that issue and make it self-serviceable.

nijave 4y ago
If your developers already have a basic understanding of AWS, the APIs are more similar between ECS and other AWS services. K8s introduces an unrelated API nested inside other Amazon APIs. Imo ECS service setup is a simpler interface/API and the load balancer integration is very good. K8s can go a lot of different ways depending on the type of LB and whether you opt for ingress and whether your ingress is cloud provided or runs in your cluster. Same with logging. ECS you just configure a log group and get persistent logging and basic aggregated searching with Cloudwatch Insights. K8s you get ephemeral, unaggregated logs unless you introduce additional tools. When I worked with ECS, it was originally setup by software engineers and it was scripted using the AWS SDK and worked almost exactly the same as the rest of the stack that used things like S3. Starting with k8s would require learning a new SDK/API versus some new endpoints in the one you're already working with. On the other hand, as we grew, we'd hit weird issues like ECS nodes going dead without the backplane rescheduling containers on working hosts (I think they've improved ECS agent health checks since then, though)
- sontek 4y ago
  But if we are comparing apples to apples, you would just use AWS provided stuff for EKS right? > K8s can go a lot of different ways depending on the type of LB k8s flexibility shouldn't be counted against it here. If you are considering k8s against fargate, you should only be considering ALB / NLB ingress and not the many more ways you could. Just use what AWS provides and be happy with it :) > ECS you just configure a log group and get persistent logging and basic aggregated searching with Cloudwatch Insights You can log to cloud watch with EKS as well. Fluentd can log to cloudwatch with very little configuration. I agree that if you are already "all AWS" and just want to put one more thing in there, Fargate might match your existing patterns better. But saying "Fargate is easier than managed kubernetes" is very wrong. In general I've seen people have an easier time understanding kubernetes manifests for declaring their services instead of the equivalent terraform to get fargate up and running to do the job.

qbasic_forever 4y ago

If you don't use k8s and just run bespoke containers you still have to figure out how those containers find and talk to each other. Maybe you run some custom DNS setup, maybe you run a purpose built service discovery thing like consul, etc. And you have to figure out how you'll do networking to support public/internet-facing workloads vs. private internal services (and how each can talk to the other). But... if you just use k8s you get things like basic service discovery, networking, ingress, etc. with it and don't have to figure out bespoke solutions (that you'll just chuck anyways once you move to k8s). I do agree though I would be very hesitant to immediately dive in running stateful workloads like databases, etc. on my own k8s cluster. The cloud hosted database services that every provider has is such a significant time and complexity saver, especially if your databases are just getting started and small-ish.

grogenaut 4y ago
Discovery can be quite simple, we've been doing it for years. On ECS/AWS we do it with just ALB and the cluster. Turns out DNS works really well, surprising for a fundamental internet technology. Tho if it's broken it's likely DNS that is the culprit. If I need to route to other things I just configure other dns entries with SSM Parameter store... or I just manipulate the local resolvers in the VPC. Very straightforward. But I'm generally running just my software, not other people's, and consuming aws datastores. This is for work.
- deleted 4y ago
  [deleted]
- staticassertion 4y ago
  Does k8s not just use dns? That's what Hashicorp Nomad uses - DNS via Consul.
  - qbasic_forever 4y ago
    It does, every pod, service, etc. in the cluster has a fully qualified DNS name resolved by a DNS server in the cluster.

endisneigh 4y ago

My advice would be to try to structure your app to only use the database directly and structure all business logic around asynchronous actions that can be called via server less functions or regular services you’re hosting. IMHO this is the future of most softwares architecture. Set up proper replication for your database and you’re good. Very few companies I’ve seen need more than this in principal. In practice there’s a lot of real time stuff that really isn’t necessary increasing architectural complexity. The amount of service types that inherently need real time processing can probably be counted on a single hand

humbleMouse 4y ago
mikhmha 4y ago
yup this is what we do at my work. There is data and then all business logic / asynchronous processing is done using serverless functions. Ends up being really cost effective and it can scale well. All the data exists (and setup with replication) so you can always migrate ur app to some other platform or architecture if need be. we have CI/CD pipeline that builds and deploys serverless functions using serverless containers. not a serverless enthusiast btw. microservices have their own set of issues but they can be the right fit for certain apps.
KronisLV 4y ago
> server less functions This is a nice idea, but many out there will need to work with on-prem infrastructure and self-hosted software. So my question is: why would be the best open FaaS solution that you can setup on your own infrastructure and use without caring too much about the underlying hardware/OS? > Set up proper replication for your database and you’re good. Another thing that I'd consider is how the DB will grow over time. Should you have a single, monolithic DB you'll end up with hundreds if not thousands of tables which will be cumbersome to work with. Personally I think that being able to use foreign keys across your entire domain is like a superpower, but some might advocate for separate DBs for separate domains, once you get that far.
wruza 4y ago

johnwoods 4y ago

It's funny that so many startups are led to thinking that they "need" K8s

deleted 4y ago
[deleted]
grogenaut 4y ago
reminds me of when I started at a .com in 99 that failed... they felt they needed oracle and sun and cisco routers... we did everything on linux for dev work but the founder wanted to apparently be able to sell that hardware and impress investors. He got basically nothing back on any of it. They offered me a sun server for like $100 that cost $20k. I took the windows desktop instead.
- johnwoods 4y ago
  reminds me about the need for AI/ML in random things. maybe it's to sell the "powered by AI/ML" tag. just another hypestorm
klysm 4y ago
I think they are led to believe it’s a good bet to make early.

throwaway787544 4y ago

The question of whether to use K8s or not is like wondering what kind of saw you should use to cut wood. There's different saws for different purposes. But even with the right saw, you still have to know how to use it correctly. Better to use a hand saw correctly than a table saw incorrectly. (you can use a table saw incorrectly, but best case the work ends up crap, worst case you lose a finger) After building infrastructure for dozens of teams, I'm quite convinced of the following: - if your people aren't very skilled, they won't build anything well. most software engineers i've seen professionally working in the cloud are handymen trying to build a wood cabinet. - if your people can't build well, it doesn't matter what technology they use. choosing between building a cabinet out of metal or cherry wood doesn't make much difference if they've never built a cabinet before. - if the first two holds: then only use the technology which requires the least skill to use well, and where the amount of maintenance is closest to zero. don't build a wood cabinet from scratch when you can buy flat pack. don't buy flat pack when you can buy an assembled cabinet, get it shipped, and carried into your office. - if using the aforementioned technology requires 'building' or 'assembling', and that is not core to the customer-facing aspect of your product, then you should not be building, you should be buying. if your business doesn't involve assembling flat pack furniture, don't ask your employees to build their own desks and chairs from Home Depot or Ikea parts. buy the premade desk and chairs, use them to make your actual product. - a software engineer knows as much about cloud architecture as a fine woodworker knows about framing. "it's all just wood" until the house takes 10x as long to frame and is 10x as expensive and still doesn't meet code. - people will try to build things they don't fully understand and leave the company before anyone realizes the mess they've made. imagine your retail store is accessible by driving a car over a wooden bridge built by a handyman.

CoastalCoder 4y ago
> Better to use a hand saw correctly than a table saw incorrectly. To be fair, better to use a handsaw even incorrectly than a tablesaw incorrectly.
- Jistern 4y ago
  I concur.

hintymad 4y ago

Curious question: why don’t companies consider an abstraction like EC2? They can run k8s on EC2-like virtualization, right? I had great experience with EC2: the ability to launch thousands of machines reliably with full control to the machines, the ability to manipulate all the metadata and configuration without learning any additional shit like HCL is a huge productivity booster. The simplicity of EC2 seems a great foundation layer for us to build more advanced resource allocation. Case in point, it drives me nuts that one has to spend hours learning how to use a template system in Nomad to pass in the simplest configurations. I can’t fathom why one would be even slightly interested in learning any shit of Nomad just to deploy a god damn docker container. Don’t we have more interesting problems to solve and more general knowledge to master?

iampims 4y ago
EC2 instance boots in 2 to 3 minutes, a Docker container starts in seconds. You can specify precisely how much CPU/Memory you need for a container, with a VM, you're stuck with what the cloud provider offers. You still need to get your app artifacts on that VM, how do you plan to do this? EC2 (VM abstraction) is great for some type of software, but don't forget you'll also need a Load Balancer, provisioning TLS certs, security groups, autoscaling groups, VPN, public/private subnets…
- hintymad 4y ago
  Oh, I love container-based deployment. I was just wondering if it’s worth building abstractions like EC2 that requires very little learning to use, use EC2-like systems to manage machines that host containers
  - coredog64 4y ago
    That’s ECS without the Fargate.
- nathants 4y ago
  ec2 alpine on t3.small boots in 30 seconds. 45 seconds for spot. default-like vpc with public subnets and internet gateway is fine. zero trust. route53 health checks and returns up to 8 hosts in random order. to the gp’s point, using aws sdk to manage ec2 is shockingly easy. a lambda on a 1 minute timer with an aws sdk runs circles around autoscaling groups et al.

didip 4y ago

The organization must have a need first. And then, after you have a legitimate need, use a hosted Kubernetes solution. Don't roll your own. It's really that simple.

andrewallbright 4y ago

We should choose our technology stack like a hermit crab chooses it's shell. The shell shouldn't be too heavy to move around in but with a little room to expand in. When it's time, we work to find a new shell.

CharlesW 4y ago
This is a delightful (and wise, IMO) way to think about right-sizing dependencies.
papito 4y ago
I love it.

MarquesMa 4y ago

The problem is there are no sane in-between options. On one end of the spectrum are either neat platforms like Heroku or Vercel or ssh and bear-metal with simple scripts. On the other end of the spectrum, we have Kubernetes. Everything in between: - The learning curve is much steeper than Heroku and Vercel - The skill is not likely to transfer to the next job - The ecosystem is not as complete as Kubernetes Most mid-sized companies went for Kubernetes because the in-betweens are not very optimal and need to take some risks on betting them.

nathants 4y ago
i find managing aws from a lambda with an aws sdk to be a good in between. trigger the lambda on events and/or a schedule. lock around dynamodb to ensure a single lambda at a time is mutating aws. example: https://github.com/nathants/libaws/tree/master/examples/complex/s3-ec2 https://github.com/nathants/libaws/tree/master/examples/comp... knowledge and intuition about aws primitives is definitely a transferable skill.
zmmmmm 4y ago
I think docker-compose / docker swarm are sane inbetweens. They only do half of what Kubernetes does but it's the half you need at smaller scale.
- icod 4y ago
  I'd have to disagree. I tried docker compose and swarm. nothing but issues. I went back to old fashioned bash
KronisLV 4y ago
> On one end of the spectrum are either neat platforms like Heroku or Vercel or ssh and bear-metal with simple scripts. > On the other end of the spectrum, we have Kubernetes. Someone else mentioned Docker Swarm, but allow me to offer my own thoughts. The simplest option (what you allude to) is probably running containers through the "docker run" command, which can sometimes work for a limited set of circumstances, but doesn't really scale. For single node setups, Docker Compose can also work really nicely, where you have a description of your entire environment in a YAML file and you can "orchestrate" as many containers as you need, as long as you don't need to scale out. The aforementioned Docker Swarm is a simple step up that allows you to use the Compose syntax (which is way easier than what K8s would have you use) but orchestrate containers across multiple nodes and also has networking built in. It's simple to set up (already comes preinstalled with Docker), simple to use and maintain (a CLI like that of kubectl, but smaller), the performance and resource usage is small and the feature set is stable. It's an excellent option, as long as you don't need lots of integrations out there and whatever Docker offers is enough. From there, you might look into the likes of Hashicorp Nomad, though their HCL is a bit more complicated than the Compose format, though still simpler than Kubernetes. The setup is a bit more complicated (if you need Consul and TLS encrypted traffic) but overall it's still just a single binary that can be setup as a client/server depending on your needs. Also, as an added bonus, you can also orchestrate other things than containers, much like how back in the day there was Apache Mesos which supported different types of workloads (e.g. you can launch Java apps or even native processes on nodes that you manage, not just containers). Of course, even when you get into Kubernetes, there are also projects like K3s and k0s, maybe with tools like Portainer or Rancher, which let you manage it in a simpler manner, either through an easy to use UI or with those K8s distributions being slightly cut down, both in the plugins that they come with, as well as their resulting resource usage and data storage solutions (e.g. use SQLite instead of etcd for smaller deployments). In my eyes, betting on OCI is a pretty reasonable option and it allows you to run your containers on whatever you need, depending on what your org would be best suited to. > The skill is not likely to transfer to the next job I'd argue that if you need to read a book to figure out how services are deployed in any given environment, then you probably should have a DevOps/Ops team and not have to worry about it as a dev. Or, if you're a part of said team, you should still mostly just use OCI containers under the hood and the rest should be much like learning a new programming language for the job (e.g. like going from Java to .NET, which are reasonably similar). > The ecosystem is not as complete as Kubernetes Kubernetes largely won the container wars. No other toolchain will ever have as complete of an ecosystem, but at the same time you also dodge the risks of betting on some SaaS solution that will milk your wallet dry with paid tiers and will fold a few years down the line. I'd say that all of the aforementioned technologies support all of the basic concerns (deployments, monitoring, storage, resource limits etc.).

preommr 4y ago

My 2c: Start with basic k8s. Just getting a couple of apps deployed on a cluster with logging, scaling, pipelines takes ~ 2 weeks. Trying to rebuild that basic functionality using ad hoc solutions takes way more time and rapidly becomes more cumbersome as the wheel gets reinvented when k8s as a platform has well documented solutions/tools. Putting it into production leads to a handful of footguns that take a while to sort out related to pull-policies, caching, scaling, security, etc. But it's fairly manageable. And probably easier to premempt than customized solutions since these pitfalls are somewhat well documented. Past a certain point though, especially as the work veers into things like operators, sketchy helm packages, service meshes, k8s falls apart fast if you don't have people on it full time, and it's much better to write some customized code.

mountainriver 4y ago
100% I’ve seen this happen in my current organization as they chose ECS, and then reinvented half of kubernetes in house in a fragile way. Basic kubernetes where you are just deploying a web app is incredibly simple. There is no more complexity than any other container deployment tool, but Kube can grow with you like few other tools can.
icod 4y ago
> Trying to rebuild that basic functionality using ad hoc solutions takes way more time and rapidly becomes more cumbersome as the wheel gets reinvented when k8s as a platform has well documented solutions/tools. That is just not true. 1 day if you know what you're doing. If you use a well performing language you won't even need scaling for a very long while, unless you expect like "Eurovision Song Contest" type of concurrent users.

mt42or 4y ago

Don't use AWS yet. Cloud setup is so complex that you should avoid using it.

jalk 4y ago
Exactly - this discussion sound exactly like the ones we had when AWS started out
- nathants 4y ago
  aws is fine. just use the good parts, skip the rest.

likortera 4y ago

I've been at several companies. The ones where things were the smoothest were using App Engine, other was using Tsuru, and the other was using heroku. Zero problems. The companies where we suffered the most, had to fight a lot to get anything shipped, and we where expected to understand a custom in house jungle of yaml files and script and half of the features we needed from the platform were half assed were using Kubernetes. That's just my experience.

mrweasel 4y ago
Comparing App Engine and Kubernetes makes it feel like we took a step backwards. Rather than just deploying code and have everything managed, now I suddenly have to worry about networking and load balancers again.

etaioinshrdlu 4y ago

I figure that Kubernetes will eventually become very mature and stable, the rate of changes will slow down, and it will become a predictable building block like Linux is now. I'm personally choosing to avoid using it for now.

jschrf 4y ago

I never quite got the K8S "it's too complex" hate, but to be fair I haven't scaled it very high. Sure, it's verbose and there are N levels of abstraction, but it's a declarative API for running foo across multiple environments of bar. I've always wanted this. I like raw, versioned infrastructure config with no extra crap. I have a little K8S.yml snippet I copy+paste+tweak into repos when I want to throw an ad-hoc experiment into a cluster, and then a bigger setup for IRL projects that looks something like this: - k8s - base - api.yml - web.yml - worker.yml - namespace.yml - ingress.yml - overlays - dev ... config to merge ... - staging ... config to merge ... - production ... config to merge ... - shared - ... variable declarations, base config maps, etc ... Everything gets merged into a manifest.yml and version stamped and build-artifacted. Deployment just means applying the config overlays via kustomize based on environment and then pushing out. If things break, I always have an absolute, pull-the-chute, versioned, formal safe point to go back to: kubectl -n production apply -f manifest.version.yml

deleted 4y ago
[deleted]

jsdevtom 4y ago

What about if you need rolling deployments?

nijave 4y ago
All you need for rolling deployments is >1 instance and load balancing. You can use DNS load balancing which isn't very nuanced, a local load balancer per machine (like nginx with 2 backend on the same machine) or more complex load balancing systems

benreesman 4y ago

Oh man, my org at FB was the test bench for moving to containerization and fancy service discovery and all that (Tupperware, just a different Borg reimplementation). It eventually got pretty good, but it never stopped being mad overkill for single-thousands of boxes. When you’re in the 10s or 100s of thousands in a fleet, or when you’ve got workloads that don’t neatly slot into your SKUs, Kube/Borg/TW/Mesos are The Way. No doubt about it. But it’s always seemed zany to me to stack namespaces/cgroups/etc. on top of Xen or whatever EC2 is using. Yo dawg I heard you like an abstract machine so I put… There are just separate concerns: - reproducibility (shared libraries argggghh) - resource limits to bin-pack SKUs - service discovery - failover - operational affordance And I’ve seen it get so ugly to conflate these very different imperatives. Running 100Ks that need to web serve on demand but web index when idle? Yeah, now you need the whole enchilada. But it’s false and harmful to promote the idea that the minute you need Grafana or DNS or Salt/Ansible/Nix/whatever that you need BORG. There are scenarios where I would enthusiastically break out Kube, but most of the marketing around it falls into my “and you will do nothing, because you can do nothing.” bucket.

robertlagrant 4y ago
It's really hard to break through the fog of metaphors here. Are you saying that below 10000 boxes it's worth doing something else to deploy on to them than k8s? What would that be?

jansommer 4y ago

I was in a startup with a small team and we used Kubernetes for servers that required more than 8 GB memory. Cloud Run and App Engine doesn't offer more than that, at least not at the time. The alternative were to deal with virtual machines ourself with Ansible scripts, and I'm not sure how that would auto scale + it would break the existing flow of Docker containers for everything. It took a while to figure out how to put Kubernetes in a closed VPC, but after that it was fairly straight forward.

davidspiess 4y ago
As far as i know Cloud Run offers now up to 32 GB memory workloads

yibers 4y ago

The title is actually very confusing and so is the article. After reading it a few times I finally understood that the author is actually pro Kubernetes. He is just saying that if you are in the really early stages of your startup, don't use Kubernetes right away. You will probably want to use it eventually.

nunez 4y ago

i would have agreed with OP three years ago when kubernetes was very niche and setting it up was difficult. today, kubes is very easy to get going with. you can set it up locally with kind or k3s for local dev, and use a cloud vendor's flavor in production. last time i tried to get a local dev env working for lambda, i spent a lot of time hacking on runtime-level stuff. it was not pleasant. additionally, the market of devs and operators fluent in it has grown by a lot. many people are getting their CKx certs, and there are enough companies using it now to create reliable supply. i say this because the lift from "my app is working in docker" to "my app is working in kubernetes" is much smaller than it used to be. given that OP is suggesting container runtimes as the alternative (which can get very expensive; much more so than using kubernetes for everything), i think that if a business is at a point where they are containerizing to accelerate releases, then kubernetes is a natural next step. anything else in between is at best a costly dependency and at worst throwaway.

te_chris 4y ago

GKE autopilot deserves a mention. If you’re going to go k8s, then it’s pretty close to noops

dig1 4y ago

My 2c: you don't need K8s unless you are google-scale. Even if you think you are google-scale, you are not. Maintenance-wise and $$-wise, two VPS boxes with Cloudflare (even with an enterprise account) setup is usually cheaper than an ordinary K8s setup. But again, it all depends on your use case, and people usually overestimate their use cases and the company's growth.

KronisLV 4y ago
> Maintenance-wise and $$-wise, two VPS boxes with Cloudflare (even with an enterprise account) setup is usually cheaper than an ordinary K8s setup. How do you feel about the benefits of using containers? And if you do use containers, how do you feel about the benefits of orchestrating them, instead of running something like Docker Compose? E.g. having overlay networks and being able to deploy a new version of your software on multiple nodes simultaneously? I've found that even when you don't want K8s, something like Docker Swarm or Hashicorp Nomad may still have benefits for you, depending on what you're trying to do. Swarm is basically feature-complete, boring and just like adding multi-node capabilities on top of Docker Compose anyways. Nothing to install apart from Docker and nothing to configure, apart from executing a cluster init/join command.
- dig1 4y ago
  My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I haven't mentioned Swarm intentionally, which IMHO I consider "the next iteration" of your containerized setup: Swarm is a boring, just-works, tool if you need to spread multiple containers, add some networking between them, and have to manage everything by a normal person. And boring is good :)
  - KronisLV 4y ago
    > My tool of choice is usually Ansible, so, from the Ansible point of view, deploying to a container, VPS, bare server, or a cluster, looks pretty much the same. I agree that Ansible is an excellent tool! Though personally, I enjoy establishing a clear boundary between the "infrastructure" and "app" parts of any setup - the former ensuring that OS has whatever runtimes are necessary for any given application to run, user accounts, groups, folders, services etc., whereas the latter is whatever apps are running on the server. So I'd use Ansible for most of the former, setup a container cluster and then use something like https://docs.ansible.com/ansible/latest/collections/community/docker/docker_swarm_module.html#ansible-collections-community-docker-docker-swarm-module https://docs.ansible.com/ansible/latest/collections/communit... to manage actual application deployments. Why? To make the apps more throwaway and limit the fallout in case of bad configurations/deployments, as well as make sharing the base Ansible playbooks for what constitutes a production ready server setup easier amongst projects. Of course, you could as well use Ansible to install something like JDK/Tomcat and set up its configuration which worked well for me in the past, but personally that didn't scale quite as well as running Java/Node/Ruby/Python/PHP in containers and approaching them with Ansible almost like black boxes (e.g. copy a bunch of files, like secrets, into these directories, deploy some arbitrary YAML against the Docker Swarm cluster), which was surprisingly easy to do.

goncaloo 4y ago

Kubernetes are amazing at what they do, but only relevant to ~0.1% of the companies in my opinion. It's way too complex and too much work for the rest of the world, and not worth the time invested. A lot can be accomplished with simple virtual machines and some sort of auto scaling groups (depending on your cloud provider they have different names). Kubernetes are amazing at unifying your workloads on any clouds though. If you care about portability, you should either consider using kubernetes for everything or using a tool that abstracts your configuration in a cloud-agnostic way. Although I'm a bit biased on this one.

mmcnl 4y ago
Simple is subjective and dependent on where you're coming from. If you're used to Kubernetes then managing virtual machines seems needlessly time consuming and vice versa.

jbverschoor 4y ago

Well, if you're using it for development, it doesn't make sense.. You're probably doing all the microservices, which should mean that those services are the responsibility of another team. It also means they should have a testing / staging / development version online somewhere. If you're developing MS Paint.. do you really need to have to compile windows and all the dependencies?

nix23 4y ago
>You're probably doing all the microservices, which should mean that those services are the responsibility of another team. You can't imagine how many time i had to tell customers what micro-services are. Micro-service-architecture are teams who declare a common interface, and then can change their service without informing other teams and the whole system still works. Declaring a Micro-service-architecture need's a lot of time and highly experienced interface planer's (in a perfect world those should never change).

nickdothutton 4y ago

I’m much more interested in creating the simplified replacement technology for K8.

RiyaadhABR 4y ago

Running Caprover on a VPS has been a very nice alternative to full K8s. It is like your mini Heroku but not too much magic involved. A lightweight wrapper around Docker containers but comes with a nice GUI and handles networking between your applications. the One Click apps are also very useful for quickly spinning up databases and stuff like that.

bradwood 4y ago

My startup just bypassed all this container stuff and went straight for AWS Serverless. If you design it well, it works excellently. If/when we need long-running workloads we'll go to containers but thus far we're just rocking out with lambda, and sns/sqs and Eventbridge.

chrismarlow9 4y ago

Containerization solves a team problem where the system dependencies for an application need to be in control by the team, and allow operations people to specifically focus on the infrastructure supporting the application. From the technical side you can accomplish nearly all of the same goals using machine images and something like packer combined with any config management tool. I guess what I'm saying is you should use containerization when the complexity of your application and complexity of your infrastructure is too high for an operations (DevOps) person to deal with. Or when it changes so frequently it's impossible to keep up with the specific application needs. An example is some poor DevOps engineer who has to maintain terraform scripts for the infrastructure but also needs to know the version of python used in application XXX or the postgres header libs are required for YYY. And a team of 30+ application devs are changing this constantly. It's a burden and a risk to require a DevOps engineer to remember and maintain all of this. So you start looking to docker so this responsibility can be handed off to the team that owns the application. So in short if you're a small startup and have 1 or 2 DevOps guys you'll probably be okay with a very simple system of building machine images. As the complexity grows this handoff of machine requirements can be handed over to the teams by using docker. And if you do this properly by abstraction the system build code away through makefiles or bash scripts the transition from machine images to dockerfiles is pretty straight forward and easy. Possibly as easy as creating a packer file that will build the docker image instead of machine images. Kubernetes is just a tool for the operations people to manage the containers. I guess what I'm saying is if you can't automate properly with basic machine images, you should really tackle that first. And that containers solve a team logistics problem, not a technical one.

BossingAround 4y ago

As a person that is deeply involved in Kubernetes and Istio, I'm starting to get the feeling that in the beginning, it should be totally acceptable to run containers from Docker Swarm. If your main need is "restart container on error" (which seems likely for startups), you probably can't beat the fast and easy deploy time of Swarm. Also, when the time comes to upgrade to Kubernetes, you won't be locked in to some incompatible solution. Of course, if you care about scaling, serverless is probably the way to go.

mwaitjmp 4y ago
Is docker swarm still maintained?
- BossingAround 4y ago
  Ah sorry, I meant Docker Compose, not Docker Swarm.

roxaaaane 4y ago

use K8s if you need it, don't use K8s if you don't, it's that simple ¯\_(ツ)_/¯. It's not rocket science you don't need to read every week's opinion on K8s and you don't need to write one either.

klysm 4y ago
I’m not sure the question of “will k8s save me time” is particularly easy to answer.

Too 4y ago

In theory what the article suggest looks sound and easy. I have to strongly disagree though after having the unfortunate experience of hands on trying Azures managed container runtime where everything about it was just plain misery: Getting logs out, updating the running container, connecting to storage, all kinds of esoteric settings hidden under even more strange abstractions, lack of documentation and experience online, strange edge cases everywhere. You still have to understand all the complex cloud stuff like networks, ingress and storage accounts, etc. We changed to managed kubernetes instead and even for a team with no prior experience it was much smoother. Fast declarative deployments, logs, attach, everything is just one instant kubectl command away. Documentation, blogs and resources are in excess so you will always find a way out. There are still some things that are difficult but I attribute most of that to the cloud generally. As someone else said, use it if you already know it and know what you are getting into. If you don’t, be prepared for some initial bumps and surprises. It has its warts but it’s not nearly as bad as some people portray it, those cases are likely more a problem with a micro service architecture gone out of control rather than k8s itself. Sadly there is no middle ground alternative. Closest would be docker-compose, assuming someone runs the VM for you and that still comes with a lot of hassle of deploying files to that VM and you still need to configure all the cloud networking.

pid-1 4y ago
AWS ECS isn't much better. It's weird Cloud Providers don't see DevX as a priority, they are at serious risk of becoming commodities due k8s and likes.
alexalx666 4y ago
Docker compose on VPS sounds much better than going straight to Azure/AWS

naiv 4y ago

Could someone eli5 what 1e0, 1e1 , 1e2 stand for?

treis 4y ago
Maybe I'm salty this morning, but I don't think people should take advice from someone that used more characters to write something in a less clear way.
- rgaino 4y ago
  I'm with you.
ocharles 4y ago
1en = 10 ^ n. So 1e0 = 10 ^ 0 = 1, 1e1 = 10 ^ 1 = 10, 1e2 = 10 ^ 2 = 100, etc.
- naiv 4y ago
  Thank you!

eof 4y ago

I’d just say “don’t learn k8s yet.“ if you know it, it’s fine, but it’s complicated to get right, so delay if you haven’t made by that journey yet.

darthrupert 4y ago

(2025)

vbezhenar 4y ago

It's funny to read this article and thread while doing exactly what everyone suggests to avoid: building Kubernetes on bare virtual metal for a team with few programmers without any dedicated devops or SRE roles. The reason I'm doing it is because our business owner thinks that we need scalability and high availability. We have law obligations to keep our data inside a country. And we don't have any managed Kubernetes offerings inside our country. The best cloud stuff I've found is hoster with openstack API and that's what I'm building upon. I thought really hard about going with just docker swarm, but it seems that this tech is dying and we should rather invest into learning Kubernetes. Honestly so far I spent few weeks just learning Kubernetes and few days writing terraform+ansible scripts and my kubernetes cluster seems to work good enough. I didn't touch storage part yet, though, just kubeadm-installed kubernetes with openstack load balancer, calico network and nginx ingress. I guess hard part will come with storage stuff. Worst thing is: everyone talks about how hard it is to run Kubernetes on bare metal, yet nobody talks about what exactly issues are and how to avoid them.

mrweasel 4y ago
For some projects there's no way around it, you have to build Kubernetes on bare metal/virtual machines. I've faced the same issues, the project called for Kubernetes, we can debate that requirement, but it was specified in the contract. Some project simply requires you to build Kubernetes on-prem, mostly for legal or political reasons. I do question the logic of picking Kubernetes for scalability in those projects though. To make that work, you end up with a lot of excess capacity, unless you can scale down one workload, while scaling up another. E.g. scaling down a website or API at night and use the capacity for batch jobs. Honestly building the cluster isn't my main concern, that pretty easy, I managed to write Ansible code for deploying a cluster in less than a day. My main concern is debugging and maintenance long term. Reading about companies that spin up a new cluster, because it's easier than figuring out while the old one broke, is an indication that Kubernetes might not be completely ready for production use.
- p_l 4y ago
  Mixed use is, arguably, where k8s shines the most. As in, you have a pool of hardware, and you want to optimize use of its capacity. Mix interactive jobs work batch jobs. Maybe even prod and non-prod. Fit as much as possible on smallest amount of nodes, etc.

AtNightWeCode 4y ago

As a startup it is very important to spend as much time and money as possible on the features not the tech. If the tech is not the feature that is. Most startups should build one single monolith API running on some default offering in the cloud. Serverless services can be used as compliment for some workloads. Building, running, and maintaining containers is unnecessary for this and it does slow down the dev process.

zoomzoom 4y ago

At withcoherence.com, we agree that leaning on managed runtimes for as long as possible makes a ton of sense. I’ve also seen that hiding the complexity of transforming code into deployed containers by fully abstracting away dockerfiles and CI/deploy scripts can lead teams into a tough spot at a bad time to learn what’s really happening. But the appeal of k8s is often the ecosystem of tools that solve real problems these runtimes leave on the table: managing multiple environments, load balancing across services, SSL/TLS, SSH, long-running tasks, managing multiple versions, integrating tests into pipelines. Coherence is working to solve these problems and create a great developer experience without hiding what’s really going on under the hood. (Disclosure, I’m a cofounder)

munchenphile 4y ago

Startups should almost never use k8s. They need to iterate fast and ignore the complexities of infra. k8s is far too complex for most small companies. CapRover Droplet on Digital Ocean + deploy your Rails app with git. Scale your single VPS up as needed. Most don’t need much beyond that for quite a while.

nouveaux 4y ago
I have never heard of CapRover and I dont know how to use it. I do know how to use Kubernetes across Digital Ocean, AWS, GCP and Azure. Just had to learn it once. I don't even use it for scaling. Not complex at all. I've used it for four years. Literally never had a problem with it. It just runs.

txtai 4y ago

Interesting article, thank you for the insights.

theptip 4y ago

I agree with the initial advice to use a simple Docker container runner hosted service. I initially ran my Django app as a bare Docker container behind an Nginx reverse proxy, with a “docker pull; docker stop <old>; docker run <new>” script as my deploy job, and that was fine for a year. Took all of an hour to build the plumbing there. A hosted service would have been just as good and probably even less time to wire up. I disagree with the OOM requirements for operating a k8s cluster. I ran our infra on GKE from 3 engineers through 15 as the primary Infra engineer (while also pushing code and being CTO), and it was hours-per-month of labor, not a dedicated SRE. I trained up some of the other engineers and they were able to help with on call after a few hours of training. For a simple app (a few deployments and services) it is really not hard to work with. All that said I agree you don’t _need_ it at 5-person scale. I would not recommend you learn it if you don’t already know how to use it. But if you do already know it, you can get good value from using it much earlier than the article recommends. (For example I found Review Apps to be very useful for fostering collaboration between frontend and backend engineers, and that feature is not too hard to wire up on top of k8s.) If I had to give one-sentence pithy advice I’d probably agree with the OP title.

markstos 4y ago

There was a time when conventional wisdom was that we all needed to be using XML data exchange, all the time. Now the simpler format JSON dominates. I hope Kubernetes ends up being the next XML.

desktopninja 4y ago

Might never need kubernetes: https://stackexchange.com/performance https://stackexchange.com/performance

p_l 4y ago
Stack Overflow has been increasingly using Kubernetes for last few years, though.

yetanotherjosh 4y ago

k8s is just one component of the whole architecture, yet people talk about it like it's the singular defining characteristic. It reminds of people talking about building "react apps" when react is one of probably hundreds of essential library dependencies and says nothing about 80% of the tooling. In my rig that I manage in a startup of 2 people, k8s (managed EKS on AWS) is the part that generally just works with very little effort. I provision and deploy to it with terraform and helm. The cluster itself is cattle. I can spin up a second identical cluster with a config file, cut dns over, then spin the old one down. It took around 3 weeks to get everything setup, but k8s was not the hard part, it was making all the OTHER decisions and integrations for things like container building, monitoring/logging/apm, secrets management, setting up a VPC correctly, writing some custom config scripts to generate the right setup for the 2 separate apps we run in both staging and production, etc. The work was undoubtedly far greater outside the k8s domain. In fact when it came to the k8s parts, e.g. defining services, ingress, etc, I was generally relieved and pleased. And now that I've done all this, I feel comfortable repeating it. Things run quite well and I have zero pressure to migrate.

curious_cat_163 4y ago
I think the article is congruent with your thinking — k8s shapes those OTHER decisions in critical and may in some cases be the reasons why those decisions exist in the first place. To me, the insight of the article was questioning whether a team of your size should be needing to make those decisions? It may very well be that your case requires k8s. But, I can think of many small shops that would benefit from keeping things simple and paying someone (a cloud provider) in the beginning of their journey and slowly adding complexity as success comes along.

systemBuilder 4y ago

Am I supposed to believe these blanket statements from a person who doesn't realize 1e0 = 1e1 = 1e2 = ... = 1 ?

klysm 4y ago
I thought 1e2 = 1 * 10^2