29 ms·
“Let’s use Kubernetes.” Now you have eight problems
- tdons 7y agoWe use k8s at $JOB so I decided to look into it on my own time. A wasted weekend trying to install it and a profanity laden #ragequit later I moved all my stuff over to OpenBSD instead. Never been happier. In the case of k8s: ignorance truly is bliss. Keep it simple people.
- FlyingSnake 7y agoI've taken over a project containing 6 DB entities. Instead of building a monolith (or normal REST API), the Architects used 7 µServices based on k8s and NoSQL DB. Now simple development tasks take extra time, and anything that affects multiple µServices needs n times the development efforts. I wish they had started with a simple monolith, and refactored to µServices if needed.
- theK 7y agoYour problem, my friend, is not Kubernetes or anything else technological. It is the people around you that call themselves architects :-)
- FlyingSnake 7y agoTrue that, I must add that k8s added to the delay for features and enhancements. Like most enterprise projects, by the time I got my hands on this project, all the Architecture Astronauts had already moved to their next planet :-)
- michaldudek 7y agoI’m a very happy user of Rancher 1.6 for years. Simple, nice GUI, got everything I need, works fast, can deploy as many apps /services as you wish, no new concepts to learn (if you know Docker that is). Used it in my previous agency to manage clients websites and use it now in my startup to manage multiple envs with few apps (api, front end, workers) and nice and easy deployments via GitLab CI.
- fock 7y agoit seems too me like kubernetes aims to replace the existing service-mesh consisting of de facto microservices (load-balancers, remote-logging, systemd, xinetd, ...) bonded by unixy conventions with.... a monolithic, proprietary system. Proponents are then advocating to build decoupled microservices on top of this. Am I the only one who thinks this is shizophrenic? (on the other hand: companies claiming to make the world a better place by selling ads to the highest bidder are shizophrenic...)
- theK 7y agoI’m positively attuned to the ... de facto Microservices bonded by unixy conventions ... part. And it has worked well in the past. You had much more freedom but everyone needed to pretty much roll their own. Kubernetes is a compromise on that. You don’t need to build you ops framework from scratch now and deploying something has very well defined APIs. OTOH you now are in a world of finding which k8s extensions/plugins/whatever you should use and which are just “fancy”. And answer questions like Is Knative V0.8 good enough for our workloads? Because Ai Defi needs that for some reason... EDIT: Kubernetes is open source though right?
- fock 7y agoyeah it is open source, but half a million loc for a simple admin tool seems like the perfect path to vendor lockin ;)
- ex_amazon_sde 7y agoThis is exactly what lock-in looks like.
- BerislavLopac 7y agoSoftware engineering is the perfect example of the "blind scientists and the elephant" problem. It is a very complex field, with a number of related but distinct disciplines and activities required to make it work; it's impossible to be an expert in everything, so we tend to specialise: we have back-end engineers, front-end engineers, data engineers, SRE experts, devops specialists, database experts, data scientists and so on. Additionally, the software we are building varies wildly in terms of complexity, dependencies, external requirements etc; and finally, the scale of that software and the teams building it can vary from one person to literally thousands. Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped". The reality is that various technologies and patterns -- microservices, monoliths, Kubernetes, Heroku, AWS, whatever -- are tools that enable us to solve certain problems in software development. And different teams have different problems and need different solutions, and each needs to carefully weigh their options and adopt the solutions that work the best for them. Yes, choosing the wrong solutions can be expensive and might take a long time to fix, but that can happen to everyone and actually shows how important it is to understand what is actually needed. And it's completely pointless to berate someone for their choices unless you have a very detailed insight into their particular needs.
- alexandercrohde 7y ago> Articles like this one, and even more comments on HN and similar sites, generally suffer from a perspective bias, with people overestimating the frequency of their own particular circumstances and declaring something outside of their needs as "niche" and generally misguided and "overhyped". It's my experience the opposite is true. The blindness is people overestimating their needs (or resume-padding) and using specialized, overcomplicated tools meant for traffic in the billions (e.g. cassandra, kafka, mapreduce) for 20-person startups that haven't hit rapid growth (most of which never do).
- BerislavLopac 7y agoI'm afraid you might be falling into the exact trap I have described. Realistically, how many such cases have you seen? And of those you have, how many did actually implement such a complex solution and ran it for a long time without either closing down or transforming to something more suited to their needs?
- newcrobuzon 7y agoIf your solution comprises only of one or few systems, and primary reason for which you are considering k8s is just to tackle clustering/scalability/ service discovery then you can always just try to start by building simple clustering into your system. Here is how I built it into mine: https://www.titanoboa.io/cluster.html https://www.titanoboa.io/cluster.html Obviously this will not always be the right solution, but in some cases might be better fit than k8s...
- k__ 7y agoI'm coming from the serverless side of things where people always say that only a few companies on earth even need K8s, like cloud providers. How do you rationalize using it in your company? For example, a learning platform that allows to integrate code for frontend and backend into your lessons uses Docker containers for their product. They can't offer runtimes for every programming language running on the frontend and backend, so they allow the teachers to upload their own container. I'd say they are a good example of a company that could need k8s.
- atonse 7y agoThe odd thing about having 20 years of experience (while simultaneously being wide-eyed about new tech), is that I now have enough confidence to read interesting posts (like any post on k8s) and not think "I HAVE to be doing this" – and rather think "good to know when I do need it." Even for the highest scale app I've worked on (which was something like 20 requests per second, not silicon valley insane but more than average), we got by perfectly fine with 3 web servers behind a load balancer, hooked up to a hot-failover RDS instance. And we had 100% uptime in 3 years. I feel things like Packer (allowing for deterministic construction of your server) and Terraform are a lot more necessary at any scale for generally good hygiene and disaster recovery.
- wpietri 7y agoSame. I like trying out new things, so I have a feel for what they're good for. I tried setting up Kubernetes for my home services and pretty quickly got to "nope!" As the article says, it surely makes sense at Google's scale. But it has a large cognitive and operational burden. Too large, I'd say, for most one-team projects.
- hinkley 7y agoI have, at various times in my career, tried to convince others that there is an awful, awful lot of stuff you can get done with a few copies of nginx. The first “service mesh” I ever did was just nginx as a forward proxy on dev boxes, so we could reroute a few endpoints to new code for debugging purposes. And the first time I ever heard of Consul was in the context of automatically updating nginx upstreams for servers coming and going. There is someone at work trying to finish up a large raft of work, and if I hadn’t had my wires crossed about a certain feature set being in nginx versus nginx Plus, I probably would have stopped the whole thing and suggested we just use nginx for it. I think I have said this at work a few times but might have here as well: if nginx or haproxy could natively talk to Consul for upstream data, I’m not sure how much of this other stuff would have ever been necessary. And I kind of feel like Hashicorp missed a big opportunity there. Their DNS solution, while interesting, doesn’t compose well with other things, like putting a cache between your web server and the services. I think we tried to use that DNS solution a while back and found that the DNS lookups were adding a few milliseconds to each call. Which might not sound like much except we have some endpoints that average 10ms. And with fanout, those milliseconds start to pile up.
- shoulderfake 7y agoThen dont use it. Like wtf.
- daitangio 7y agoK8s is complex. For this reason Cloud providers sell it as a service. K8s and microservices are a trend topic, so it is true you must think with your own head before creating microservices at will. But I think the article is a bit too negative. For instance, in my humble experience the application server is always a bottleneck before the database (i.e. if the database is Oracle or PostgreSQL). Microservices move a bit of complexity on the client side, ask for smarter clients and offer a lot more resilience and fault tolerance. The article focus only on scaling and forget the "Single Point of Failure" problem.
- cmhnn 7y agoIt's sold as a service because it drives compute. Compute is one of the top money makers. The more people who buy into things like k8 that are essentially VM deployment launchers the better. The providers aren't emotional about this like 90% of the posters here. If a new flavor of the week gets people to launch VMs next year then that will be sold.
- snicker7 7y agoI feel that the pro-monolith / anti-microservice attitude has become something of a cargo cult (at least here on HN).
- hedora 7y agoCargo cults were island nations that saw cargo planes land. They didn’t understand why the planes landed, but they did want to trade with them. They built makeshift runways and put out boxes of cargo in the hope of attracting planes. If anything, the 5 person startup with 20 microservices “because Google/Netflix” and “we want to scale” is the cargo cult in this debate. Not saying one side’s wrong or right, just that cargo cult doesn’t seem to being used correctly here. (I admit I’m skeptical of microservices, since they add so much complexity, and even Netflix suffers partial service outages on a regular basis...)
- Hippocrates 7y agoThere’s a lot of configuration to understand with k8s and even GKE. Badly configured probes, resource budgets, pod disruption budgets, node affinities etc. can have disastrous effects. I’m pushing my teams more towards serverless since it takes out nearly all ops/scaling/rollout complexities. Right now we’re seeing our serverless apps on GCF, GAE and cloud run outperform our GKE apps easily in scaling, reliability, and simplicity (configuration and time spent getting it deployed in a satisfactory manner)
- spyspy 7y agoThis is the lesson my last company learned hard. For anything serving less than tens of thousands of requests a second, you just can't really beat GAE in terms of simplicity and cost.
- samblr 7y agoIm planning to use GAE in a production environment. Can you share the specifics on how GAE managed to scale please ?
- awinter-py 7y agothey say kube is greek for 'pilot' but it also means 'dice' are you feeling lucky punk?
- earthboundkid 7y agoBecause Greek y and u are the same, "cyber" is from the same word as "kube". "Cybernetics" was supposed to be the art of piloting, but it was very loosely defined and the term was deliberately overhyped.
- jeffadotio 7y agoKubernetes is supposed to make distributed computing more reliable. Of course a non-distributed system will not be magically made simpler by using an inappropriate tool for the job. The concept of a single filesystem as the source of truth for not just a program but also the systems that run the program will be the lasting legacy of Kubernetes.
- SirensOfTitan 7y agoI disagree with the HN consensus here: I think managed kubernetes is really useful for startups and small teams. I also commonly hear folks recommending that I use docker-compose or nomad or something: I don't want to manage a cluster, I want my cloud to do that. We run a fairly simple monolith-y app inside kubernetes: no databases, no cache, no state: 2 deployments (db-based async jobs and webserver), an ingress (nginx), a load balancer, and several cron jobs. Every line of infrastructure is checked into our repo and code reviewed. With k8s we get a lot for free: 0 downtime deployments, easy real time logging, easy integration with active-directory for RBAC, easy rollbacks.
- flowerlad 7y agoI am a solo developer (full stack, but primarily frontend), and Kubernetes has been a game changer for me. I could never run a scalable service on the cloud without Kubernetes. The alternative to Kubernetes is learning proprietary technologies like "Elastic Beanstalk" and "Azure App Service" and so on. No thank you. Kubernetes is very well designed, a pleasure to learn and a breeze to use. This article seems to be about setting up your own Kubernetes cluster. That may be hard; I don't know; I use Google Kubernetes Engine. For others considering Kubernetes: go for it. Sometimes you learn a technology because your job requires it, sometimes you learn a technology because it is so well designed and awesome. Kubernetes was the latter for me, although it may also be the former for many people. The first step is to learn Docker. Docker is useful in and of itself, whether you use Kubernetes or not. Once you learn Docker you can take advantage of things like deploying an app as a Docker image to Azure, on-demand Azure Container Instances and so on. Once you know Docker you will realize that all other ways of deploying applications are outmoded. Once you know Docker it is but a small step to learn Kubernetes. If you have microservices then you need a way for services to discover each other. Kubernetes lets you use DNS to find other services. Learn about Kubernetes' Pods (one or more Containers that must reside on the same machine to work), ReplicaSets (run multiple copies of a Pod), Services (exposes a microservice internally using DNS), Deployments (lets you reliably roll out new software versions without downtime, and restarts pods if they die) and Ingress (HTTP load balancing). You may also need to learn PersistentVolumes and StatefulSets. The awesome parts of Kubernetes include the kubectl exec command which lets you log into any container without almost any setup or password, kubectl logs to view stdout from your process, kubectl cp to copy files in and out, kubectl port-forward to make remote services appear to be running on your dev box, and so on.
- shusson 7y ago> I could never run a scalable service on the cloud without Kubernetes Can you give us an indication of the scale of your app? e.g rpm.
- flowerlad 7y agoIt is still in development, so no rpm at the moment. That’s another thing: some people think Kubernetes is something you use if you need high scalability. I disagree. Kubernetes should be the default if your app consists of more than 1 service. If you don’t have high scalability requirements you can rent a single-node GKE “cluster” for about $60 per month. If you have just 1 service then a single Docker container is all you need, so Kubernetes not needed.
- humbleMouse 7y agoOnce you configure the kubernetes network layer with whatever hosting platform you’re using, it’s really not difficult to administrate. It’s funny to me how much kubernetes hate there is on hn.
- turtlebits 7y agoUntil need to debug an issue and you're way over your head due to all the moving pieces.
- humbleMouse 7y agoThen assign someone to debug it who understands that kubernetes is a wrapper of common linux functionality. It’s not that hard to debug issues in kubernetes. Check status of pods, memory levels, storage mounts, network configurations, and the stacktrace you’re debugging. Not that difficult. I’m not saying there aren’t edge cases, but if you set up your system with centralized logging (filebeat) and have some way to scrape metrics (jmx, built in tooling) you’ll be fine.
- thiago_fm 7y agoIt's not that hard to use Kubernetes and it makes the developer's life easy. It's very easy to deploy helm charts and even though that there are many gotchas and complex things, if you want to deploy something simple, it is easy and completely doable to do it even solo. (rant) After over 10 years in development I've done and used literally all the things people complain here a lot about: Virtual machines, Single page apps, docker, microservices, FP and the list goes on. Even though I've struggled, I feel very lucky to be able to try all those things and it's a joy to use and I've shipped shitloads of great code that is making a lot of money to a lot of people and improving businesses in general. I don't mean you need to use K8S or even like it, but there is definitely developers which know their shit very well, and can also make great Single page apps using more than 3 different JS framework, also write good backend code and so on. And also enjoy all of this and make companies definitely successful. It sickens me a bit how so much posts of this kind get a lot of attention and could be replaced by "yes, software, like everything in life, is complex!!!11". I think the article itself is completely shallow to actually touch the difficulties there is with using kubernetes and is mostly useless information. There are at least 10 posts with a better and more structure criticism, but it's just because it's cool to complain about new things, it gets automatically traction in HN(which used to be a place where people like new things...). So... yes, you shouldn't use K8S everywhere(also applies to everything...), but it is the new thing(well, not really new...). Should we just talk about Apache mod_php? It's natural that people want to try new stuff and actually enjoy working with software. Not everybody sees everything as problems. "Now you have eight problems, hehehehehe!!11". Am I the only one that found this post completely useless and at some degree, toxic? (/rant)
- alexandercrohde 7y agoWell good for you that your SPAs work every time, and you never broke production with kubernetes. Here's a cookie. Now for the rest of us who work with engineers across all skill ranges and experience levels, we actually do need to care about such factors. The question is -- you hire a guy off craigslist to run your site, and every minute of downtime costs $1,000. Are you going to want him to use Kubernetes or a braindead simple hosted solution?
- aganame 7y ago
- throwawaymanbot 7y agoA solution in search of a problem is how I class Kubernetes.
- partiallypro 7y agoProbably unpopular, but I am generally opposed to using Docker/Kubernetes for ~75%+ of projects. I've been in arguments over this, but containers being unmaintained and the complexity of Kubernetes can cause major issues. It's over engineering for smaller projects. That's just my opinion. I think a flat VM is more appropriate most of the time. But there is no denying the advantages of Docker when it's done right and used right. A developer told me just a few weeks ago that you should "always" use Docker, which I just found to be so ridiculous.
- rantwasp 7y agothat’s not unpopular at all. everyone that has had to run k8s and keep it uptodate or deal with unmaintained docker containers understands this 100%
- kube-system 7y agoPeople who have issues with unmaintained docker containers are doing it wrong. You still need to assess the quality of your dependencies for container images just like you should be doing for any other dependency. The issue is that docker lets some developers get in over their head very easily. Many orgs have system admins to install and configure server operating systems, but docker shifts some of those responsibilities back on to the developer.
- rantwasp 7y agoyes, but no. usually I would agree with you, but in today's world where we curl install stuff from the internet you'll always have someone pull a container to 'just get it to work'. Once the prototype works it's production. People that have the discipline to actually research the quality of deps or... god forbid... actually build the containers that rely on from scratch will not get into this kind of issue, but again: kids these days...
- Glyptodon 7y agoSo I wasn't really on the Docker train for a good while, but using containers takes so many sysadmin and environment issues out of play that I've started to change my mind. The position I find myself in is actually that for development environments Docker Compose and similar tools are wildly convenient, but that there's often not a really obvious path from that to production at small scale low cost.
- halbritt 7y ago> The more you buy in to Kubernetes, the harder it is to do normal development This demonstrates the bias and perspective of the author. The best way I can describe it is code-centric rather than system centric. If that's "normal" then the article makes some very valid points. For example, I've seen quite a few folks make the attempt to scale out badly when they could've scaled up rather easily. Very many "bigdata" problems can be handled on a single machine with a terabyte of memory. If one shares that code-centric perspective, then yeah, k8s probably isn't for you. The real benefit in overcoming the very validly criticized complexity of k8s is the number of things that happen without intervention. From a systems-level perspective, all these things are crucial. Services are abstracted with endpoints by default. Liveness and readiness are built in. Self-healing is built in. A consistent model by which apps are deployed is built in. Logging, metrics, and SLA monitoring while not built in can all be added and employed without intervention. Ideally, these things abstract the infrastructure sufficiently well that it allows developers to focus on development, rather than ancillary tasks like deployment, monitoring, resilience, etc.
- peterwwillis 7y agoA lot of the problem with using Kubernetes is it appears to be the only option for running microservices in a cloud environment. People choose it because they think there's no other option (and they're somewhat right). But there's Nomad, DC/OS, Docker Swarm (for a little bit anyway), ECS, GKE, etc. That's still not a ton of options, but there are options. That's just microservice orchestration. That's a small part of the totality of things needed to implement a full-out SDLC. You can't just build Kubernetes and think you're done; your code will need to integrate into a lot more stuff, and you'll end up writing 10 layers of glue because that's just how many use cases you have to support. And it's weird that all that glue doesn't use standards. I mean, we have TCP/IP & RPC & REST, we have pipes & filehandles, we have the OCI specs. That gets us to a point where (at most) half of the stack of an architecture is portable and interoperable with any system following those standards. But then there's every other component of the architecture that connects all the pieces together, meaning you're writing glue that will only work for one implementation. Change your implementation, and you have to change your glue, and probably more stuff. I think a lot of that non-reusable glue could be erased if it all followed standards, such that the configuration and operation of each part followed a standard interface, set of data types, etc. Tools and libraries could just "talk container orchestrator" or "talk load balancer" or "talk object storage" or "talk secrets management", and virtually any component could be integrated into any other, by virtue of either a system-wide or application-specific configuration. You could argue we have something like that now with a "kubectl file" or similar, but that's not only still platform-specific, but other tools don't speak it, so K8s has to do everything, because it's the only thing that speaks its language (config file/backend data store/IAM/secrets/roles/etc). Rather than resign ourselves to those limitations, we could bundle everything in an implementation-agnostic standard way with standard interfaces. The exact same configuration (as code) could be used to run the same complete architecture on a dozen different platforms, because every component would speak the same language and handle all the other components in the same ways. The backend services could all translate the standard based on how they were configured, such that generic instructions are then translated into implementation-specific actions. You could really write your architecture once and run it anywhere, without the caveat of "anywhere on this platform only". I feel like we're not talking about doing that because we keep getting caught up in "Fuck, Kubernetes is pretty hard" conversations. Yeah, it's hard; building and operating an 18-wheeler is hard. But what about the roads? What about the gas stations? What about the containers we put on the trucks? All that stuff is standard, and so we don't have to worry about what implementation of gas station or road we use. I feel like we still don't have those things in the cloud, and it's just weird.
- johnmarcus 7y agoLol. I've been using k8 for 5+ years and am currently interviewing for more 'traditional' sys admin roles. The number of traditional problems I don't have by running even a small k8 cluster has become even more obvious as I go through the process. For every alternative you may suggest, I can point out dozens of problems you can run into that are solved with k8s. Just like any other operations tool, yes, it takes time and effort to learn...that's why sys admins exist. I always find these posts by non-sys admins humourous. Devs often have so little respect for the knowledge base required to maintain a stable, secure, scalable, and flexible system. It ain't nothing, no matter the tool you use.
- acd 7y agoI think the network part of Kubernetes is hard todo right and it’s very complex. Further more I ponder on the performance if you compare a local monolith service on metal with full local cpu cache vs a distributed network requests Microservices. Disclosure I run Kubernetes in Production.
- p_l 7y agoIt's IMO much easier if you don't run complex CNIs, but unfortunately not everyone can afford that :(
- i_dursun 7y agoFlash News! Software engineering is hard.
- hypewatch 7y agoIt’s interesting that this critique of kubernetes is on a blog called “python speed” because my most recent project with kubernetes was deploying a large dask cluster. For this use case k8s was really valuable. It made the devops part so much easier than it otherwise would have been, so we could put most of our time into application logic. In other words, when we wanted to achieve substantial “python speed” kubernetes was very helpful. For data engineering projects, even with a small number of data engineers, it can be a big productivity booster. Personally, I like kubernetes and find it easier to use than other devops tool sets, so it’s become my go-to tool. Probably wouldn’t recommend it to someone who doesn’t know it and has a simple app architecture.
- papito 7y agoI find the assumption of many companies that they NEED stuff like K8S because they are going to be so "webscale", frankly, pretty arrogant. Most systems out there can run on a 2010 laptop, if done right.
- Saaster 7y agoIn my previous company we had 20 million daily active users and we ran that on 4x M4.large EC2 instances. 4 instances not because it had any significant load (probably ~10-15% sustained), but purely because of high-availability and the ability to do a rolling release update.
- dan_quixote 7y ago> "You can get cloud VMs with up to 416 vCPUs and 8TiB RAM, a scale I can only truly express with profanity. It’ll be expensive, yes, but it will also be simple." It's simple until you need to update. Good luck meeting any SLAs with your fleet of singletons.
- nova22033 7y agoMost of the "You don't need kubernetes" posts should be "you don't need docker".
- pnathan 7y agoKubernetes solves very real problems in a way that handles a full suite of them. This is very complex because the problem set is complex. If you're running a substantially smaller system, k8s makes less sense. That said, if you're familiar with running and monitoring k8s, a gke deploy will solve a lot of the pain a traditional LB + EC2 ASG will incur out of the gate. Let me explain: Notionally, we need 4 basic services operationally for a single typical service deployment. 1 of FooService, 1 load balancer, 1 database, 1 monitoring/logging system. All of these should tolerate node death; this means roughly 3 pieces of hardware for this notional system. This is complexity that k8s covers, at a high cost of knowledge. If you're bought into AWS, the Beanstalk system will do this decently well, last I checked. I think there is room for a k8s-like tool that is good for teams with < 10 services, and less than 10 engineers. Even k3s (https://rancher.com/docs/k3s/latest/en/ https://rancher.com/docs/k3s/latest/en/) has substantial complexity at the networking layer that, I think, can be stripped for the "Small Team". So I agree with the author in theory that k8s is overkill. But also other infra types can start getting difficult to deal with in time, and "just deploy onto a single big box" doesn't cover the operational needs.
- theseadroid 7y agoWould AWS Elastic Beanstalk fit that <10 services profile?
- pnathan 7y agoyes, it would. Costs start really getting heavy with EB at a certain point, since you're spinning up 1+ ASG & LB per service (a tier is an ASG and a LB, possibly a DB). I wouldn't build a microservice architecture against EB, at _all_. I'd say EB probably is cost effective up to, IDK, maybe 3 services with 3 nodes per ASG. Then you're breaking even or worse with k8s ops cost, and now you're looking at "how much time (= money) is it to manage k8s with KOPS" vs "how much are we spending on EB". KOPS is a very low-effort solution once you get it rolling.
- pm90 7y agoHeh, it’s quite amusing to see the posts here arguing that “you can do the same thing with multi az deployments on aws with VMs, packer and ebs. Kubernetes needs you to learn so much shit” ... do you even read what you write? Kubernetes is not gospel. It’s an opinionated, incomplete framework for orchestrating container workloads. There are other ways to do the same thing which are fine too. It works well for the most part but has disgusting failure scenarios. So do other techs. People who use and like kubernetes are comfortable with its trade offs and portability. You may not be. It’s fine. Shitting on kubernetes just because you’re comfortable with another technology just because you can: that’s not fine.
- zachguo 7y agoFor our team with many containers scattered around ElasticBeanstalk and ECS, K8S makes everything much cleaner.
- AcerbicZero 7y agoThere is something amusingly ironic about a python blog complaining k8's being overly popular relative to how difficult it is to actually use. K8's is extremely complex sometimes, but at least it maintains some semblance of semantic logical consistency, unlike certain other tools.
- AcerbicZero 7y agoThere is something amusingly ironic about a python blog complaining k8's being overly popular relative to how difficult it is to actually use. To be fair, I've had nothing but mediocre experiences with Python, so I'm a bit jaded. On the actual content of the article....well it gets worse somehow. There are good arguments against using K8's (or any tool, really) but I don't think any of them made it into this article. "Why scale with microservices when you can just get a single massive VM" was probably my fav.
- martythemaniak 7y agoHow about this rule of thumb when it comes to the question of "Should you Kube" How many developers in your organization? 0-10: No 10-100: Maybe 100-1000: Yes 1000+: Definitely
- rammy1234 7y agoKubernetes local environment setup Kubernetes debugging Too many configurations to worry about Ever evolving features No good practices All these apply for small teams who wanted to get the products out to the market
- Havoc 7y agoYeah always seemed like madness to me. Docker compose seems to be the sweet spot. Still sorta infrastructure as code via yaml without the fleet/swarm logic overkill
- rossdavidh 7y agoHaving been at a company that was starting to move things to Kubernetes, when it had absolutely no reason to, I can say that it was being done because: 1) the developers wanted to be able to say they knew how to use Kubernetes, when they applied for their next job (perhaps at a company big enough to need it) 2) the managers didn't really understand much about what it was, to evaluate if it was necessary, but 3) some of the managers wanted to say they had managed teams that used Kubernetes, for the same reason as the developers Which is not to say that it should never be used. But we have a recurring pattern of really, really large companies (like FAANG) developing technologies that make sense for them, and then it gets used at lots of other companies that will never, ever be big enough to have it pay off. On the other hand, they now need 2-3x the developers they used to, because they have too many things going on, mostly related to solving scale problems they'll never have. Don't use a semi-tractor trailer to get your groceries. Admit it when you're not a shipping company. For most of us, the compact car is a better idea.
- deepsun 7y agoWell, for engineers sometimes it's just interesting to learn new tech, you know. And managers are ok with keeping the team motivated, especially if they can deliver new features while playing with the new tech. UPDATE: I mean, it's not about resume in most cases I've seen. Usually, people who don't love programming do not care about what tech to use, so resume-oriented people are usually not very vocal about using new cool and shiny tech. It's nerds who are, that keeps them motivated.
- BostonEnginerd 7y agoPlaying with new tech is fun. Using a nuclear weapon when a hammer would do is bad engineering.
- zaat 7y agoWe are talking about an orchestrator here, right? it's a piece of software that is safe for use in front of kids, last time I checked. Linux is preety complex system too, you know, with millions lines of code and gazillion moving parts. It's probably safe to say that no one alive, including Linus himself, ever read all the codebase required for running a base linux system serving Apache with a static site. And yet we do. Anyone who know how to code well can learn how to pack containers and run them within a kubernetes cluster in a way that will work reliably, even if those containers do nothing except serving pure html text files over port 80, and in fact the whole cluster is waste of money and it is not required and not helping a thing. It will serve the content just fine. No nukes will be hurt during the operation, don't worry.
- battery_cowboy 7y agoFor my personal projects, k8s is so useful that I wouldn't ever build a server by hand again. I can spin up my blog or whatever easily on one cluster, and if it becomes too expensive I can just move elsewhere, or if I want to reduce my costs, I could just run a single-node-cluster (I don't need HA) on a DO droplet or something and still get the ease of being able to destroy and rebuild my apps anytime I want to. It might be "overkill", but so are most of the tools I use each day. Of course, I never create my own clusters, but it isn't that hard to follow a tutorial if I had to.
- sethammons 7y agoWe used to manually ssh to deploy our dozens of nodes, just a handful on developers. git pull, restart service. Then we got to hundreds of nodes. Chef, chef, and more chef. Deploys were typically run with a chef-client run via chef ssh (well, a wrapper around that for retries). With dozens of services and many dozens of engineers, this worked well enough. Then we got to thousands of nodes. And hundreds of developers working on a multitude of services. We've adopted k8s. It has been a lot of work, but the deploy story is wonderful. We make a PR and between BuildKite and ArgoCD we can manage canary nodes, full roll outs, roll backs, etc. We can make config changes or code changes easily, monitor the roll out easily, and revert anytime. I still don't _like_ k8s mind you - I don't think programming with templates and yaml is a good thing. But I've come to terms with that being the best we will have for now.
- rbanffy 7y ago> Then we got to hundreds of nodes. Until you get there, Kubernetes is most likely overkill.
- NickKampe 7y agoWe deploy small clusters everywhere in the same pattern, I love argocd. This article fails to understand the use case for kubernetes, and arguably doesn't fully understand the cloud. Kubernetes is revolutionary, to think it's not is foolish.
- devy 7y agoIt would be unfair to simply mention about a technology's "problems" without mentioning even a single point of "features". IMO, Itamar Turner-Trauring has misrepresented Kubernetes. The sole reason of Kubernetes being popular and uprising is because there are genuine pros and features that make many people's lives easier instead of more miserable. So let me point out the pros here: [1]: https://www.infoworld.com/article/3173266/4-reasons-you-should-use-kubernetes.html https://www.infoworld.com/article/3173266/4-reasons-you-shou... [2]: https://hackernoon.com/why-and-when-you-should-use-kubernetes-8b50915d97d8 https://hackernoon.com/why-and-when-you-should-use-kubernete... [3]: https://opensource.com/article/19/6/reasons-kubernetes https://opensource.com/article/19/6/reasons-kubernetes [4]: https://www.weave.works/technologies/the-journey-to-kubernetes/ https://www.weave.works/technologies/the-journey-to-kubernet...
- zaro 7y agoI'll take kubernetes over the miriad of AWS services any day. Kubernetes is de facto the cloud standard. If I have to know stuff about the cloud, I would like my knowledge to be transferable to the other cloud also. I understand that these clouds have to compete and so on, but why do I have to pay the price to learn their particular ways of naming things and private apis, and what not. So of course devs like kubernetes.
- kirbypineapple 7y agoWhat service would the HN folks recommend for someone who needs to run a few dozen different docker services that require persistent storage? It would be nice to just have a pool of compute resources tied to persistent storage and be able to spin up docker instances at will. K8's has been suggested as the correct solution but it sounds like a lot of overhead for services that require no scaling at all.
- sascha_sl 7y agoEndpointSlice is a really bad cherry pick. I have written K8S controllers for 2+ years and only recently learned about it when I had to write an ingress controller. Funnily enough I also had to learn about externalTrafficPolicy because it turns out if you have a lot of pods and use an AWS ELB the traffic distribution can be terrible, so a daemonset with local-only routing to them and then round-robining to pods works wonders. You need to know none of this if you're not even close to the level of scale we are.
- sub7 7y agoYou are the man. Preach on. I wonder how many teams with active users in the 1000s have fully dockerized/Kubernetes/microservice type shit designed for 10000x load that they will likely never get to because they didn't spend their time iterating on product.
- TheKarateKid 7y agoEvery time I feel secretly embarrassed for running my small projects on a simple cloud VPS VM, an article like this comes along and restores faith in my decision to not over-engineer things. This has come up on HN before, and it's a great read - "You are not Google": https://news.ycombinator.com/item?id=19576092 https://news.ycombinator.com/item?id=19576092
- privateSFacct 7y agoYou can get pretty far with docker and things like ECS / Fargate etc too.
- alexellisuk 7y agoWe originally built OpenFaaS for Swarm, then moved to Kubernetes and support both now. The complexity of K8s is harrowing, but by and large works well, if you can keep up with the pace of change. Try running a controller you last modified 12 months ago on Kubernetes 1.17. Now we've spent time looking into containerd and trying to provide microservices/faas on top of that instead - without the clustering (https://github.com/openfaas/faasd https://github.com/openfaas/faasd) Something I do like about K8s is the ecosystem - in 5 minutes I can automate TLS with LetsEncrypt on a managed cluster.
- yayajacky 7y agoGreat tools make hard problems approachable. They also reduce impossible problems to hard ones. But, it takes experience (having solved things with easy and hard ways) to prefer easy problems and easy solutions. It's a good time to be a consultant where you get to solve the same problems using different approaches.
- mancerayder 7y agoWe're being pushed to move to Kubernetes/EKS (or, ECS as a plan B) away from Elastic Beanstalk. Elastic Beanstalk does the job but it's slow to deploy things, it's inflexible, and the Amazon AMIs that underly them are riddled with ancient packages that infuriate security scanning security people. Are we making a mistake moving to Kubernetes instead of re-designing an infra-as-code using docker+terraform+ansible+a CI pipeline? We run a client-facing app but it doesn't have super low latency requirements, although it does need to be able to scale up to run jobs. Something else. I have to be honest, the complaints here about "people are doing it for their CV," while true, need to be understood in the context of, it's extremely hard to move jobs without significant kubernetes and Docker experience.
- ryanthedev 7y agoKube isn't for small businesses. It's for enterprises. Most developers aren't running the clusters, they are 99% of the time managed by the cloud provider. Why would I ever order VMs and manually run a kube cluster. But as an operator needing to manage clusters and multiple teams, I spend my time coding automation. I'm not saying kube is not simple, but it's a lot easier then managing application runtime on VMs. Before kube I was managing 260+ VMs (hyperv was internal managed DCs central/east) for my products in-house app. I had to essentially build my own poor man's orchestration platform to manage applications and deploying.
- adgasf 7y agoSo if Kubernetes makes management simpler and more robust for teams of 500+, but is overly complex for teams of 5, what solution would people recommend for teams of 5?
- imtringued 7y agoDocker compose and a bunch of shell scripts?
- marcodave 7y agoSo, Kubernetes is the 2020 version of the enterprise application server of the '00s ?
- znpy 7y agothe proper way to use kubernetes is: - if your org is big enough: hire/train your devops engineers to manage the kubernetes cluster(s) - if your org is small or waaay too big: use some form of managed kubernetes cluster (aws, gke, do-k8s etc) - don't
- bdcravens 7y ago> there are wide variety of tools that will do just as well: from Docker Compose on a single machine, to Heroku and similar systems, to something like Snakemake for computational pipelines. Even though it has the disadvantages of being vendor-dependent and not open source, I've found ECS to be a very nice solution. Conceptually, it's very similar to Kubernetes, with much of the plumbing that makes Kubernetes so complex baked into the AWS platform (much of which you're already using if you're on AWS)
- akvadrako 7y agoI wouldn’t recommend ECS to anyone. It’s technically worse than k8s in every way and tied to a vendor. For me the biggest issue is speed of deployments. In practice it’s hard to get deploys on ECS under 1 minute. With k8s, 5 seconds is easy.
- layoric 7y ago+1 for ECS as well, I think it’s a good middle ground and much easier to use. Is getting better as well with spot instance termination detection in the ALB and more seem to be coming soon. If you are on AWS is worth considering rather than EKS or role your own K8s cluster.
- sho 7y agoThis, and the other articles like it, should be required reading on any "how to startup" list. I personally know startups for whom I believe drinking the k8s/golang/microservices kool-aid has cost them 6-12 months of launch delay and hundreds of thousands of dollars in wasted engineering/devops time. For request loads one hundredth of what I was handling effortlessly with a monolithic Rails server in 2013. It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business. k8s on day one at a startup is like a mom and pop grocery store buying SAP. It wouldn't be acceptable in any other industry, and can be a death sentence.
- collyw 7y agoSome seniors get exited by that crap as well. Tech lead in my team is full of common sense when it comes to not getting exited by JavaScripts framework of the season. But he loves to add new tools to the back end. He is choosing the "best tool" for the job, but we have a small team and already have way more things than we can comfortably manage. Good enough tools that are already in our system would be more pragmatic.
- valkon 7y agoI think it's because you don't see rapid changes on the back-end. We still consider Java 8 new.
- zentiggr 7y agoHopefully that's not a case of "must be seen to be doing something" syndrome. If you can use pre-existing tools and they serve the purpose, how easy or difficult is it to resist the "shift to this" pressure?
- marcinzm 7y ago>It is the job of the CTO to steer excitable juniors away from the new hotness, and what might look best on their resumes, towards what is tried, true, and ultimately best for the business. Then they might simply join another startup or a big tech company as competition for good engineers is fierce. Startups also famously underpay versus larger companies so you need to entice engineers with something.
- WnZ39p0Dgydaz1 7y agoI'm getting tired of these "you don't need k8 posts". Sure, if you have a simple web application with a REST API, don't use k8, unless it's for learning purposes. But nobody does that anyway. If you have something more complex with many moving parts that are separate services, k8 is a great option. I've been using it in production for close to 2 years now - not a single service downtime, great fault-tolerance, and absolutely zero management effort. Deploying complex applications, databases, and monitoring systems is easier than ever before. I don't think using k8 is overly complex. Yes, you need to invest some time to learn it, but that's the case for every new technology.
- FlyingSnake 7y ago> But nobody does that anyway. Please speak for yourself. In my experience, most of the software that's written using k8s/µServices could've been written using a simple Flask+Nginx+monitoring on a DO Droplet. K8s and the extra overhead leads to extra efforts in developement and testing, and many software teams are perennially behind schedule on delivering features. There are ways to achieve fault tolerence without tying in to the complex ecosystem of K8s.
- vincnetas 7y ago"But nobody does that anyway." You'd be surprised, but they do. EDIT. I work at company with < 5 devs and we are in progress of moving our services to k8s. Will se how it goes.
- onion2k 7y agoPractically every SaaS business could be built as a single application or a set of services, especially at the beginning when all the traffic could be handled on one server. So whether or not k8s is appropriate isn't driven by your business's complexity but rather by your architecture choices. Choosing to build microservices is what adds complexity and makes tools like k8s necessary, not the business. That complexity can be useful. It can mean you don't need to build a simple version and a more scalable version later. It means you can replace parts as the business grows rather than the entire thing. You can outsource or buy some services. These are benefits, but they are also choices, and all of them increase the time it takes to get to market. In a very well funded startup that has years of runway sometimes choosing microservices makes sense. In most small startups that are just proving their model they'd be much better off hacking something together that they'll throw away later in order to validate the business idea. Finding out that your business doesn't work as a business is far more useful than building something that can scale just in case it does. The real difficulty in that is accepting that the idea you have might not actually work.
- igammarays 7y agoOverengineering is a real problem out there. I’ve seen k8s deployed for internal back office apps that have literally 5 users - a raspberry pi could’ve hosted it. Keeping things simple and reliable is often a harder skill to learn than $BIGCO_TECH, and often confounded by political incentives.
- Glyptodon 7y agoSo if you do that another internal enforcement group will come and be like "policy is that everything is cloud now, what's your migration plan?" regardless of anything else, and the only answer is to have a plan that keeps costs similar to the Raspberry Pi.
- hedora 7y agoSo, request budget to move the app to a cloud VM / container, and work out the security issues, etc. The latter will be expensive, but the RPi deferred the cost. Bean counters like deferred costs. Then, ask finance to figure out the billing. It costs about $5 / month to rent a raspberry pi equivalent, but multitenancy might reduce that.
- adieu 7y agok8s is raw technology like linux kernel. You shouldn't use it directly which will be hard to maintain. There are bunch of packaged solutions around k8s like Google GKE or AWS EKS. By leverage them, you'll be working on a higher level of abstraction and bring productivity back.
- polskibus 7y agoWhat's the best package for running in-house?
- pan69 7y agoNot sure about your use case but you could have a look at Minikube: https://kubernetes.io/docs/setup/learning-environment/minikube/ https://kubernetes.io/docs/setup/learning-environment/miniku...
- OJFord 7y agoI assume GP means Rancher (not a recommendation for nor against) or similar.
- theK 7y agoRancher labs have a wide offering that has worked well for me. I’m currently running a couple K3S clusters on an array of Rock64 sbcs and have used the „real“ rancher distro for a while too with no problems. But I’m hardly an expert regarding the k8s on prem market so there might be even better ones!
- the_angry_angel 7y agoI’m only on the start of my k8s journey but I’m finding typhoon a nice option for on prem currently.
- yrro 7y agoOpenShift is pretty neat!
- supermatt 7y agoI see the author is a proponent of docker-compose, which I use myself for small projects. I have a docker-compose configuration in all my repos, and a `docker-compose up` brings the app up on my laptop. I could use minikube in almost exactly the same way. i.e. there is effectively no difference from a development perspective. If you are managing kubernetes yourself, on your own hardware, the moving parts can indeed be a burden for a small team - but all of these pain points go away with a managed kubernetes, as offered by most IaaS providers. i.e. if you are using an IaaS provider, there is (usually) no difference from a production perspective. There are less moving parts in docker compose, and its easier to run on a single VM - but it doesnt offer any of the dynamic features of kubernetes that you would want at scale. The same containers can run on both. If you need to dynamically scale your application, or grow beyond a single machine (I disagree with the vertical scaling proposed by the author - thats for a very specific use-case IMHO), then docker-compose is simply no good. Then you need to use docker-swarm. At this point, you either need to manage a docker-swarm cluster or a kubernetes one. Kubernetes is the obvious choice here. Fortunately, there is a trivial migration path from docker-compose to kubernetes.
- Legogris 7y agoOr Nomad or Mesos!
- supermatt 7y agoIll check these out. Thanks!
- emptysea 7y agoAlso docker-compose doesn't have zero downtime deploys which might be a deal breaker depending on your application.
- supermatt 7y agoTo be fair, its quite simple to do this. As part of your config, you would specify a reverse proxy service, and a couple of app aliases. Then you would bring up the new deployment under one alias, and shut down the other. But yeah, its nice to have that kind of stuff (and much more) built in. docker-swarm also offers that out of the box, but my experience leans more towards k8s here.
- jblake 7y agoI run a Saas business solo, for eight years now, netting six figures, and I've been on Heroku the entire time for just under $1,000 a month. Monolithic rails app on a single database, 300 tables. Sometimes I feel teased by 'moving to EC2' or another hot topic to save a few bucks, but the reality is I've spent at most 2 hours a month doing `heroku pg:upgrade` for maintenance once a year, and `git push production master` for deploys and I'd like to keep it that way. I just hope Heroku doesn't get complacent as they are showing signs of aging. They need a dyno refresh, http/2, and wildcard SSL out of the box. I honestly have no idea what the equivalent EC2/RDS costs are and I'm not sure I want to know.
- infamia 7y agoYou should look into render.com which provides a service similar to Heroku. I haven't used them myself and have no connection with them. Their name does pop up a fair bit though.
- winrid 7y agoCongrats on the business taking off! What is it?
- eitland 7y agoBased on jblake's profile it seems to be guestmanager.com, but I might be wrong. (Also, for everyone who doesn't know: - clicking on a username takes you to that users profile - clicking on the n minutes/hours/days ago take you to a permalink directly to that comment )
- gdm85 7y ago> you can use docker-autoheal or something similar to automatically restart those processes I consider it sloppy to accept that a process will crash and become unresponsive as a normal fact of life, and that subsequently they have to be automatically restarted. A process should stay put at what it was designed to do. Reasons for the crash/unresponsiveness should be investigated (memory leaks, race conditions etc) and not swept under a carpet with an automatic restart.
- hinkley 7y agoRestarts should go into your reporting too. Erlang seems to take the opposite approach. Processes are cheap, when one wears out you dispose of it instead of trying to fix it.
- deleted 7y ago[deleted]
- tyingq 7y agoIts also more common now to add on 3rd party functionality like a service mesh, various serverless implementations, secrets management, logging frameworks, etc. Making it even more complex. Not disputing some of these add value, but the number of moving parts is high.
- skrebbel 7y agoI'm curious what HN would recommend as alternatives, especially for small/early teams that are outgrowing single-machine setups. It seems to me that there's something of a gap between "for single machine setups" (eg docker-compose) and "for 500-engineer teams" (eg kubernetes).
- eitland 7y agoOften you don't need docker-compose either as long as you aren't deploying things that have weird dependencies. Single monolithic application? Can run on a VM. Multiple smaller applications written in Java, C# or Go? Can probably run side by side on a VM.
- skrebbel 7y agoFwiw docker compose is great for running stuff side by side on a VM :-)
- michaldudek 7y agoI’ve been using Rancher for years. Version 2 is all about K8s, but version 1 is super simple to set up and manage with a nice and clean GUI and pretty much no new concepts. Unfortunately version 1 is abandoned now (I think) but it’s stable and I haven’t had any problems with it. Give it a go.
- eysi 7y agoFull-disclosure: I co-founded a company that's building a developer tool for K8s (and distributed systems in general). But we've had a lot of success with running on GKE. The tool that we're making takes away the complexity of building, testing and deploying the stack, and GKE takes care of running it. In fact we use GKE for both our development and staging/prod environments. There are a lot of great tools out there that vastly improve the K8s developer experience[1] and the cloud providers take away of the pain of operating it. And with tools like Terraform and Pulumi you can codify the whole setup. Here's an example of how you can quite easily get started on GKE: https://medium.com/garden-io/gke-and-cloud-sql-a-complete-workflow-with-garden-and-terraform-1cad091e2be2 https://medium.com/garden-io/gke-and-cloud-sql-a-complete-wo... Here's a video of the same workflow: https://www.youtube.com/watch?v=iHyeD97GrE4 https://www.youtube.com/watch?v=iHyeD97GrE4. [1] https://github.com/GoogleContainerTools/skaffold https://github.com/GoogleContainerTools/skaffold https://github.com/garden-io/garden https://github.com/garden-io/garden https://github.com/windmilleng/tilt https://github.com/windmilleng/tilt
- jorams 7y agoThese kinds of posts always focus on the complexity of running k8s, the large amount of concepts it has, the lack of a need to scale, and that there is a "wide variety of tools" that can replace it, but the advice never seems to become more concrete. We are running a relatively small system on k8s. The cluster contains just a few nodes, a couple of which are serving web traffic and a variable number of others that are running background workers. The number of background workers is scaled up based on the amount of work to be done, then scaled down once no longer necessary. Some cronjobs trigger every once in a while. It runs on GKE. All of this could run on anything that runs containers, and the scaling could probably be replaced by a single beefy server. In fact, we can run all of this on a single developer machine if there is no load. The following k8s concepts are currently visible to us developers: Pod, Deployment, Job, CronJob, Service, Ingress, ConfigMap, Secret. The hardest one to understand is Ingress, because it is mapped to a GCE load balancer. All the rest is predictable and easy to grasp. I know k8s is a monster to run, but none of us have to deal with that part at all. Running on GKE gives us the following things, in addition to just running it all, without any effort on our part: centralized logging, centralized monitoring with alerts, rolling deployments with easy rollbacks, automatic VM scaling, automatic VM upgrades. How would we replace GKE in this equation? what would we have to give up? What new tools and concepts would we need to learn? How much of those would be vendor-specific? If anyone has a solution that is actually simpler and just as easy to set up, I'm very much interested.
- envoked 7y agoI'm in the same camp. I think a lot of these anti-k8s articles are written by software developers who haven't really been exposed to the world of SRE and mostly think in terms of web servers. A few years ago I joined a startup where everything (including the db) was running on one, not-backed-up, non-reproducible, VM. In the process of "productionizing" I ran into a lot of open questions: How do we handle deploys with potentially updated system dependencies? Where should we store secrets (not the repo)? How do we manage/deploy cronjobs? How do internal services communicate? All things a dedicated SRE team managed in my previous role. GKE offered a solution to each of those problems while allowing me to still focus on application development. There's definitely been some growing pains (prematurely trying to run our infra on ephemeral nodes) but for the most part, it's provided a solid foundation without much effort.
- mosselman 7y agoI use docker swarm (mode? I am not sure) in production and it works great.
- whatsmyusername 7y agoI really like docker swarm for the on-prem stuff we have.
- rawoke083600 7y agoI will in the future on my CV also have a "Will Not Work With Skillz-Matrix" Kubernetes goes on first... Unless I'm applying to Google or something of similar size.