11 ms·
Dear friend, you have built a Kubernetes
- coldtea 2y ago>I am afraid to inform you that you have built a Kubernetes. I know you wanted to "choose boring tech" to just run some containers. You said that "Kubernetes is overkill" and "it's just way too complex for a simple task" and yet, six months later, you have pile of shell scripts that do not work—breaking every time there's a slight shift in the winds of production. Or you know, I have a set of scripts that work all the time, with 1/1000 the feature-creep and complexity of Kubernetes, tailored precisely to my needs!
- fargle 2y agoasking for a friend: can i combine systemd and selinux with k8s?
- probablybetter 2y agoAuthor presumes we need Docker in the first paragraph. I suppose this is why they think we need Kubernetes. I propose using "the operating system" as the basic-unit here. It already runs on shared hardware thanks to a hypervisor. Operating systems know how to network. The entire desire for Docker in a production app comes down to willful ignorance of how software one depends upon is configured.
- levifig 2y agoEveryone talking about Kubernetes as if it was merely a "hyperscaler" and the biggest benefit of Kubernetes over a bunch of custom scripts is consistency and the ability to have everyone work on an industry standard, which makes it easier to onboard new hires, to write scripts and documentation against, etc… Repeat with me: K8s is but an API.
- zug_zug 2y agoFor what it's worth, I've worked at multiple places that ran shell scripts just fine for their deploys. - One had only 2 services [php] and ran over 1 billion requests a day. Deploy was trivial, ssh some new files to the server and run a migration, 0 downtime. - One was in an industry that didn't need "Webscale" (retirement accounts). Prod deploys were just docker commands run by jenkins. We ran two servers per service from the day I joined the day I left 4 years later (3x growth), and ultimately removed one service and one database during all that growth. Another outstanding thing about both of these places was that we had all the testing environments you need, on-demand, in minutes. The place I'm at now is trying to do kubernetes and is failing miserably (ongoing nightmare 4 months in and probably at least 8 to go, when it was allegedly supposed to only take 3 total). It has one shared test environment that it takes 3-hours to see your changes in. I don't fault kubernetes directly, I fault the overall complexity. But at the end of the day kubernetes feels like complexity trying to abstract over complexity, and often I find that's less successful that removing complexity in the first place.
- leetrout 2y agoYea but that doesn't sound shiny on your resume.
- nine_k 2y agoDepends on what kind of company you want to join. Some value simplicity and efficiency more.
- txutxu 2y agoI never did choose any single thing in my job, just because of how it could look in my resume. After +20 years of Linux sysadmin/devops, and because a spinal disc herniation last year, now I'm looking for a job. 99% of job offers, will ask for EKS/Kubernetes now. It's like the VMware of the years 200[1-9], or like the "Cloud" of the years 201[1-9]. I've always specialized in physical datacenters and servers, being it on-premises, colocation, embedded, etc... so I'm out of the market now, at least in Spain (which always goes like 8 years behind the market). You can try to avoid it, and it's nice when you save thousands of operational/performance/security/etc issues and dollars to your company across the years, and you look like a guru that goes ahead of industry issue to your boss eyes, but, it will make finding a job... 99% harder. It doesn't matter if you demonstrate the highest level on Linux, scripting, ansible, networking, security, hardware, performance tuning, high availability, all kind of balancers, switching, routing, firewalls, encryption, backups, monitoring, log management, compliance, architecture, isolation, budget management, team management, provider/customer management, debugging, automation, programming full stack, and a long etc. If you say "I never worked with Kubernetes, but I learn fast", with your best sincerity at the interview, then you're automatically out of the process. No matter if you're talking with human resources, a helper of the CTO, or the CTO. You're out.
- aguaviva 2y agoIf you say "I never worked with X, but I learn fast", with your best sincerity at the interview, then you're automatically out of the process. Where X can be not just k8s but any other bullet point on the job req. It's interesting that the very things that people used to say to get the job 20 years ago -- and not as a plattitude (it's a perfectly reasonable and intelligent thing to say, and in a rational world, exactly what one would hope to hear from a candidate) -- are now considered as red flags that immediately disqualify one for the job. Very sorry to hear about your current situation - best of luck.
- leetrout 2y ago> Spawning containers, of course, requires you to mount the Docker socket in your web app, which is wildly insecure Dear friend, you are not a systems programmer
- pzmarzly 2y agoTo expand on this, the author is describing the so-called "Docker-out-of-Docker (DooD) pattern", i.e. exposing Docker's Unix socket into the container. Since Docker was designed to work remotely (CLI on another machine than DOCKER_HOST), this works fine, but essentially negates all isolation. For many years now, all major container runtimes support nesting. Some make it easy (podman and runc just work), some hard (systemd-nspawn requires setting many flags to work nested). This is called "Docker-in-a-Docker (DinD)".
- waynesonfire 2y agoFreeBSD has supported nesting of jails natively since version 8.0, which dates back to 2009. I prefer FreeBSD to K8s.
- rthnbgrredf 2y agoI think we need to distinguish between two cases: For a hobby project, using Docker Compose or Podman combined with systemd and some shell scripts is perfectly fine. You’re the only one responsible, and you have the freedom to choose whatever works best for you. However, in a company setting, things are quite different. Your boss may assign you new tasks that could require writing a lot of custom scripts. This can become a problem for other team members and contractors, as such scripts are often undocumented and don’t follow industry standards. In this case, I would recommend using Kubernetes (k8s), but only if the company has a dedicated Kubernetes team with an established on-call rotation. Alternatively, I suggest leveraging a managed cloud service like ECS Fargate to handle container orchestration. There’s also strong competition in the "Container as a Service" (CaaS) space, with smaller and more cost-effective options available if you prefer to avoid the major cloud providers. Overall, these CaaS solutions require far less maintenance compared to managing your own cluster.
- chamomeal 2y agoHow would you feel if bash scripts were replaced with Ansible playbooks? At a previous job at a teeny startup, each instance of the environment is a docker-compose instance on a VPS. It works great, but they’re starting to get a bunch of new clients, and some of them need fully independent instances of the app. Deployment gets harder with every instance because it’s just a pile of bash scripts on each server. My old coworkers have to run a build for each instance for every deploy. None of us had used ansible, which seems like it could be a solution. It would be a new headache to learn, but it seems like less of a headache than kubernetes!
- klooney 2y agoAnsible ultimately runs scripts, in parallel, in a defined order across machines. It can help a lot, but it's subject to a lot of the same state bitrot issues as a pole of shell scripts.
- rthnbgrredf 2y agoAnsible is better than Bash if your goals include: * Automating repetitive tasks across many servers. * Ensuring idempotent configurations (e.g., setting up web servers, installing packages consistently). * Managing infrastructure as code for better version control and collaboration. * Orchestrating complex workflows that involve multiple steps or dependencies. However, Ansible is not a container orchestrator. Kubernetes (K8s) provides capabilities that Ansible or Docker-Compose cannot match. While Docker-Compose only supports a basic subset, Kubernetes offers: * Advanced orchestration features, such as rolling updates, health checks, scaling, and self-healing. * Automatic maintenance of the desired state for running workloads. * Restarting failed containers, rescheduling pods, and replacing unhealthy nodes. * Horizontal pod auto-scaling based on metrics (e.g., CPU, memory, or custom metrics). * Continuous monitoring and reconciliation of the actual state with the desired state. * Immediate application of changes to bring resources to the desired configuration. * Service discovery via DNS and automatic load balancing across pods. * Native support for Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) for storage management. * Abstraction of storage providers, supporting local, cloud, and network storage. If you need these features but are concerned about the complexity of Kubernetes, consider using a managed Kubernetes service like GKE or EKS to simplify deployment and management. Alternatively, and this is my prefered option, combining Terraform with a Container-as-a-Service (CaaS) platform allows the provider to handle most of the operational complexity for you.
- majkinetor 2y agoHighly amateurish take if you call shell spaghetti a Kubernates, especially if we compare complexity of both... You know what would be even more bad? Introducing kubernates for your non-Google/Netflix/WhateverPlanetaryScale App instead of just writing few scripts...
- hyperadvanced 2y agoHell, I’m a fan of k8s even for sub-planetary scale (assuming that scale is ultimately a goal of your business, it’s nice to build for success). But I agree that saying “well, it’s either k8s or you will build k8s yourself” is just ignorant. There are a lot of options between the two poles that can be both cheap and easy and offload the ugly bits of server management for the right price and complexity that your business needs.
- elktown 2y agoThis is so unnuanced that it reads like rationalization to me. People seem to get stuck on mantras that simple things are inherently fragile which isn't really true, or at least not particularly more fragile than navigating a jungle of yaml files and k8s cottage industry products that link together in arcane ways and tend to be very hard to debug, or just to understand all the moving parts involved in the flow of a request and thus what can go wrong. I get the feeling that they mostly just don't like that it doesn't have professional aesthetics.
- nbk_2000 2y agoThis reminds me of the famous Taco Bell Programming post [1]. Simple can surprisingly often be good enough. [1] http://widgetsandshit.com/teddziuba/2010/10/taco-bell-programming.html http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
- floydnoel 2y agowoah, that's a fantastic blog post! """After all, functionality is an asset, but code is a liability.""" I say this all the time! But i've not heard others saying this also. Great to see some like minded developers!!
- TacticalCoder 2y ago> People seem to get stuck on mantras that simple things are inherently fragile which isn't really true... Ofc it isn't true. Kubernetes was designed at Google at a time when Google was already a behemoth. 99.99% of all startups and SMEs out there shall never ever have the same scaling issues and automation needs that Google has. Now that said... When you begin running VMs and containers, even only a very few of them, you immediately run into issues and then you begin to think: "Kubernetes is the solution". And it is. But it is also, in many cases, a solution to a problem you created. Still... the justification for creating that problem, if you're not Google scale, are highly disputable. And, deep down, there's another very fundamental issue IMO: many of those "let's have only one process in one container" solutions actually mean "we're totally unable to write portable software working on several configs, so let's start with a machine with zero libs and dependencies and install exactly the minimum deps needed to make our ultra-fragile piece of shit of a software kinda work. And because it's still going to be a brittle piece of shit, let's make sure we use heartbeats and try to shut it down and back up again once it'll invariably have memory leaked and/or whatnots". Then you also gained the right to be sloppy in the software you write: not respecting it. Treating it as cattle to be slaughtered, so it can be shitty. But you've now added an insane layer of complexity. How do you like your uninitialized var when a container launchs but then silently doesn't work as expected? How do you like them logs in that case? Someone here as described the lack of instant failure on any uninitialized var as the "billion dollar mistake of the devops world". Meanwhile look at some proper software like, say, the Linux kernel or a distro like Debian. Or compile Emacs or a browser from source and marvel at what's happening. Sure, there may be hickups but it works. On many configs. On many different hardware. On many different architectures. These are robust software that don't need to be "pid 1 on a pristine filesystem" to work properly. In a way this whole "let's have all our software each as pid 1 each on a pristine OS and filesystem" is an admission of a very deep and profound failure of our entire field. I don't think it's something to be celebrated. And don't get me started on security: you know have ultra complicated LANs and VLANs, with a near impossible to monitor traffic, with shitloads of ports open everywhere, the most gigantic attack surface of them all and heartbeats and whatsnots constantly polluting the network, where nobody doesn't even know anymore what's going on. Where the only actual security seems to rely on the firewall being up and correctly configured, which is incredibly complicated to do seen the insane network complexity you added to your stack. "Oh wait, I have an idea, let's make configuring the firewall a service!" (and make sure to not forget to initialize one of the countless var or it'll all silently break and just be not be configuring firewalling for anything). Now though love is true love: even at home I'm running an hypervisor with VMs and OCI containers ; )
- do_not_redeem 2y ago> The inscrutable iptables rules? You mean the list of calls right there in the shell script? > Who will know about those undocumented sysctl edits you made on the VM? You mean those calls to `sysctl` conveniently right there in the shell script? > your app needs to programmatically spawn other containers Or you could run a job queue and push tasks to it (gaining all the usual benefits of observability, concurrency limits, etc), instead of spawning ad-hoc containers and hoping for the best.
- jrs235 2y ago"We don't know how to learn/read code we are unfamiliar with... Nor do we know how to grok and learn things quickly. Heck, we don't know what grok means "
- ewuhic 2y agoWho do you quote?
- ZeroSolstice 2y agoThis quote mostly applies to people who don't want to spend the time learning existing tooling, making improvements and instead create a slightly different wheel but with different problems. It also applies to people trying to apply "google" solutions to a non-google company. Kubernetes and all tooling in the cloud native computing foundation(CNCF) were created to have people adopt the cloud and build communities that then created jobs roles that facilitated hiring people to maintain cloud presences that then fund cloud providers. This is the same playbook that Microsoft did at Universities. They would give the entire suite of tools in the MSDN library away then then in roughly (4) years collect when another seat needs to be purchased for a new hire that has only used Microsoft tools for the last (4) years.
- ChoHag 2y agoThose who do not understand Unix are condemned to reinvent it, poorly.
- 2y ago
- diminish 2y agoOne can build a better container orchestration than kubernetes; things don't need to be that complex.
- greenie_beans 2y agoi'm at this crossroads right now. somebody talk me out of deploying a dagster etl on azure kubernetes service rather than deploying all of the pieces onto azure container apps with my own bespoke scripts / config
- greenie_beans 2y agowriting this out helped me re-validate what i need to do
- avandekleut 2y agowhat did you decide to do?
- greenie_beans 2y agokubernetes. it's well documented and better designed than whatever i could put together instead.
- deleted 2y ago[deleted]
- kasey_junk 2y agoBoth this piece and the piece it’s imitating seem to have 2 central implicit axioms that in my opinion don’t hold. The first, that the constraints of the home grown systems are all cost and the second that the flexibility of the general purpose solution is all benefit. You generally speaking do not want a code generation or service orchestration system that will support the entire universe of choices. You want your programs and idioms to follow similar patterns across your codebase and you want your services architected and deployed the same way. You want to know when outliers get introduced and similarly you want to make it costly enough to require introspection on if the value of the benefit out ways the cost of oddity.
- jesseendahl 2y agooutweighs* Only offering the correction because I was confused at what you meant by “out ways” until I figured it out.
- jerf 2y agoI like to say, you can make anything look good by considering only the benefits and anything look bad by considering only the costs. It's a fun philosophy for online debates, but an expensive one to use in real engineering.
- relaxing 2y ago> You generally speaking do not want a code generation or service orchestration system that will support the entire universe of choices. This. I will gladly give up the universe of choices for a one size fits most solution that just works. I will bend my use cases to fit the mold if it means not having to write k8s configuration in a twisty maze of managed services.
- dogleash 2y agoThe compiler one read to me like a reminder to not ignore the lessons of compiler design. The premise being that even though you have small scope project compared to a "real" compiler, you will evolve towards analogues of those design ideas. The databases and k8s pieces are more like don't even try a small scope project because you'll want the same features eventually.
- incrudible 2y agoDear friend, you have made a slippery slope argument.
- nine_k 2y agoYes, because the whole situation is a slippery slope (ony upwards). In the initial state, k8s is obviously overkill; in the end state, k8s is obviously adequate. The problem is choosing the point of transition, and allocating resources for said transition. Sometimes it's easier to allocate a small chunk to update your bespoke script right now instead of sinking more to a proper migration. It's a typical dilemma of taking debt vs paying upfront. (BTW the same dilemma exists with running in the cloud vs running on bare metal; the only time when a migration from the cloud is easy is the beginning, when it does not make financial sense.)
- incrudible 2y agoOdds are you have 100 DAUs and your "end state" is an "our incredible journey" blog post. I understand that people want to pad their resume with buzzwords on the way, but I don't accept making a virtue out of it.
- nine_k 2y agoExactly. Don't start with k8s unless you're already comfortable troubleshooting it at 3am half asleep. Start with one of the things you're comfortable with. Among these things, apply YAGNI liberally, only making certain that you're not going to paint yourself into a corner. Then, if and when you've become so large that the previous thing has become painful and k8s started looking like a really right tool for the job, allocate time and resources, plan a transition, implement it smoothly. If you have grown to such a size, you must have had a few such transitions in your architecture and infrastructure already, and learned to handle them.
- hamilyon2 2y agoFor the uninitiated: how does k8s handle OS upgrades? If development moves to next version of Debian, because it should eventually, are upgrades, for example, 2x harder vs docker-compose? 2x easier? About the same? Is it even right question to ask?
- JanMa 2y agoIt doesn't. The usual approach is to create new nodes with the updated OS, migrate all workloads over and then throw away the old ones
- Thiez 2y agoYour cluster consists of multiple machines ('nodes'). Upgrading is as simple as adding a new, upgraded node, then evicting everything from one of the existing nodes, then take it down. Repeat until every node is replaced. Downtime is the same as with a deploment, so if you run at least 2 copies of everything there should be no downtime. As for updating the images of your containers, you build them again with the newer base image, then deploy.
- ianburrell 2y agoAre you talking about upgrades of the host OS or the base of the image? I think you are talking about the latter. Others covered updating the host. Upgrades of the Docker image are done by pushing a new image, and updating the Deployment to use the new image, and applying it. Kubernetes will start new containers for the new image, and when they are running, kill off the old containers. There should be no interruption. It isn't any different than normal deploy.
- JanMa 2y agoDear friend, you should first look into using Nomad or Kamal deploy instead of K8S
- mdaniel 2y agoYou mean the rugpull-stack? "Pray we do not alter the deal further when the investors really grumble" https://github.com/hashicorp/nomad/blob/v1.9.3/LICENSE https://github.com/hashicorp/nomad/blob/v1.9.3/LICENSE As for Kamal, I shudder to think of the hubris required to say "pfft, haproxy is for lamez, how hard can it be to make my own lb?!" https://github.com/basecamp/kamal-proxy https://github.com/basecamp/kamal-proxy
- signal11 2y agoDear Friend, This fascination with this new garbage-collected language from a Santa Clara vendor is perplexing. You’ve built yourself a COBOL system by another name. /s I love the “untested” criticism in a lot of these use-k8s screeds, and also the suggestion that they’re hanging together because of one guy. The implicit criticism is that doing your own engineering is bad, really, you should follow the crowd. Here’s a counterpoint. Sometimes just writing YAML is enough. Sometimes it’s not. Eg there are times when managed k8s is just not on the table, eg because of compliance or business issues. Then you’ve to think about self-managed k8s. That’s rather hard to do well. And often, you don’t need all of that complexity. Yet — sometimes availability and accountability reasons mean that you need to have a really deep understanding of your stack. And in those cases, having the engineering capability to orchestrate isolated workloads, move them around, resize them, monitor them, etc is imperative — and engineering capability means understanding the code, fixing bugs, improving the system. Not just writing YAML. It’s shockingly inexpensive to get this started with a two-pizza team that understands Linux well. You do need a couple really good, experienced engineers to start this off though. Onboarding newcomers is relatively easy — there’s plenty of mid-career candidates and you’ll find talent at many LUGs. But yes, a lot of orgs won’t want to commit to this because they don’t want that engineering capability. But a few do - and having that capability really pays off in the ownership the team can take for the platform. For the orgs that do invest in the engineering capability, the benefit isn’t just a well-running platform, it’s having access to a team of engineers who feel they can deal with anything the business throws at them. And really, creating that high-performing trusted team is the end-goal, it really pays off for all sorts of things. Especially when you start cross-pollinating your other teams. This is definitely not for everyone though!
- Spivak 2y agoInfra person here, this is such the wrong take. > Do I really need a separate solution for deployment, rolling updates, rollbacks, and scaling. Yes it's called an ASG. > Inevitably, you find a reason to expand to a second server. ALB, target group, ASG, done. > Who will know about those undocumented sysctl edits you made on the VM You put all your modifications and CIS benchmark tweaks in a repo and build a new AMI off it every night. Patching is switching the AMI and triggering a rolling update. > The inscrutable iptables rules These are security groups, lord have mercy on anyone who thinks k8s network policy is simple. > One of your team members suggests connecting the servers with Tailscale: an overlay network with service discovery Nobody does this, you're in AWS. If you use separate VPCs you can peer them but generally it's just editing some security groups and target groups. k8s is forced into needing to overlay on an already virtual network because they need to address pods rather than VMs, when VMs are your unit you're just doing basic networking. You reach for k8s when you need control loops beyond what ASGs can provide. The magic of k8s is "continuous terraform," you will know when you need it and you likely never will. If your infra moves from one static config to another static config on deploy (by far the usual case) then no k8s is fine.
- SahAssar 2y agoI'm sure the American Sewing Guild is fantastic, but how do they help here?
- atsaloli 2y agoASG = Auto-Scaling Group https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-groups.html https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-s...
- sundbry 2y agoYou don't need to use an overlay network. Calico works just fine without an overlay.
- p_l 2y agoAnother reason to use k8s is the original: When you deploy on physical hardware, not VMs, or have to otherwise optimize maximum utilization out of gear you have. Especially since sometimes Cloud just means hemorrhaging money in comparison to something else, especially with ASGs
- jpgvm 2y agok8s is the API. Forget the implementation, it's really not that important. Folks that get tied up in the "complexity" argument are forever missing the point.
- mbrumlow 2y agoThe thing that the k8s api does is force you to do good practices, that is it.
- lttlrck 2y agoI thought k8s might be a solution so I decided to learn through doing. It quickly became obvious that we didn't need 90% of its capabilities but more important it'd put undue load/training on the rest of the team. It would be a lot more sensible to write custom orchestration using the docker API - that was straightforward. Experimenting with k8s was very much worthwhile. It's an amazing thing and was in many ways inspirational. But using it would have been swimming against the tide so to speak. So sure I built a mini-k8s-lite, it's better for us, it fits better than wrapping docker compose. My only doubt is whether I should have used podman instead but at the time podman seemed to be in an odd place (3-4 years ago now). Though it'd be quite easy to switch now it hardly seems worthwhile.
- lousken 2y agowhy adding complexity when many services don't even need horizontal scaling, servers are powerful enough that if you're not stupid to write horrible code, it's fine for millions of requests a day without much of work
- stickfigure 2y agoDear Amazon Elastic Beanstalk, Google App Engine, Heroku, Digital Ocean App Platform, and friends, Thank you for building "a kubernetes" for me so I don't have to muck with that nonsense, or have to hire people that do. I don't know what that other guy is talking about.
- blumomo 2y agoBig fan of Digital Ocean App Platform here. Deploying, running and scaling containers is so easy and their very good abstractions and tooling are worth the money.
- deleted 2y ago[deleted]
- cedws 2y agoNow compare cloud bills.
- nanomcubed 2y agoLike, okay, if that's how you see it, but what's with the tone and content? The tone's vapidity is only comparable to the content's. This reads like mocking the target audience rather than showing them how you can help. A write up that took said "pile of shell scripts that do not work" and showed how to "make it work" with your technology of choice would have been more interesting than whatever this is.
- hamdouni 2y agoI was using some ansible playbook scripts to deploy to production some web app. One day the scripts stopped working because of a boring error about python version mismatch. I rewrite all the deployment scripts with bash (took less than a hour) and never had a problem since. Morality: it's hard to find the right tool for the job
- mbrumlow 2y agoMost of the complaints in this fun post are just bad practice, and really nothing to do with “making a Kubernetes”. Sans bad engineering practices, if you built a system that did the same things as kubernetes I would have no problem with it. In reality I don’t want everybody to use k8s. I want people finding different solutions to solve similar problems. Homogenized ecosystems create walls they block progress. One is the big things that is overlooked when people move to k8s, and why things get better when moving to k8s, is that k8s made a set of rules that forced service owners to fix all of their bad practices. Most deployment systems would work fine if the same work to remove bad practices from their stack occurred. K8s is the hot thing today, but mark my words, it will be replaced with something far more simple and much nicer to integrate with. And this will come from some engineer “creating a kubernetes” Don’t even get me started on how crappy the culture of “you are doing something hard that I think is already a solved problem” is. This goes for compilers and databases too. None is these are hard, and neither is k8s, and all good engineers tasked with making one, be able to do so.
- Kinrany 2y agoSo you're saying companies should move to k8s and then immediately move to bash scripts
- mbrumlow 2y agoNo. I am saying that companies should have their engineers understand why k8s works and make those reasons an engineering practice. As it is today the pattern is spend a ton of money moving to k8s (mostly costly managed solutions) in the process fix all the bad engineering patterns, forced by k8s. To then have an engineer save the company money by moving back to a more home grown solution, a solution that fits the companies needs and saves money, something that would only be possible once the engineering practices were fixed.
- mac-chaffee 2y agoI welcome a k8s replacement! Just how there are better compilers and better databases than we had 10-20 years ago, we need better deployment methods. I just believe those better methods came from really understanding the compilers and databases that came before, rather than dismissing them out of hand.
- danjl 2y agoI love that the only alternative is a "pile of shell scripts". Nobody has posted a legitimate alternative to the complexity of K8S or the simplicity of doctor compose. Certainly feels like there's a gap in the market for an opinionated deployment solution that works locally and on the cloud, with less functionality than K8S and a bit more complexity than docker compose.
- Melatonic 2y agoI havent used Nomad but isnt that exactly what everyone always recommends?
- belthesar 2y agoI've personally been investing heavily in [Incus](https://linuxcontainers.org/incus/ https://linuxcontainers.org/incus/), which is the Linux Containers project fork and continuation of LXD post Canonical takeover of the LXD codebase. The mainline branch has been seeing some rapid growth, with the ability to deploy OCI Application Containers in addition to the System containers (think Xen paravirtualized systems if you know about those) and VMs, complete with clustering and SDN. There's work by others in the community to create [incus-compose](https://github.com/bketelsen/incus-compose https://github.com/bketelsen/incus-compose), a way to use Compose spec manifests to define application stacks. I'm personally working on middleware to expose instance options under the user keyspace to a Redis API compliant KV store for use with Traefik as an ingress controller. Too much to go into with what Incus does to tell you everything in a comment, but for me, Incus really feels like the right level of "old school" infrastructure platform tooling with "new school" cloud tech to deploy and manage application stacks, the odd Windows VM that accounting/HR/whoever needs to do that thing that can't be done anywhere else, and a great deal more.
- mdaniel 2y agoFor others interested in such things, colima also supports it: https://github.com/abiosoft/colima/tree/v0.8.0#incus https://github.com/abiosoft/colima/tree/v0.8.0#incus
- lithos 2y ago
- highspeedbus 2y ago>Tired, you parameterize your deploy script and configure firewall rules, distracted from the crucial features you should be working on and shipping. Where's your Sysop?
- marcusestes 2y agoYou did a no-SQL, you did a serverless, you did a micro-services. This makes it abundantly clear you do not understand the nature of your architectural patterns and the multiplicity of your offenses.
- physicsguy 2y agoKubernetes biggest competitor isn’t a pile of bash scripts and docker running on a server, it’s something like ECS which comes with a lot of the benefits but a hell of a lot less complexity
- FridgeSeal 2y agoFWIW I’ve been using ECS at my current work (previously K8s) and to me it feels just flat worse: - only some of the features - none of the community - all of the complexity but none of the upsides. It was genuinely a bit shocking that it was considered a serious product seeing as how chaotic it was.
- avandekleut 2y agoCan you elaborate on some of the issues you faced? I was considering deploying to ECS fargate as we are all-in on AWS.
- FridgeSeal 2y agoAny kind of git-ops style deployment was out. ECS merges “AWS config” and “app/deployment config together” so it was difficult to separate “what should go in TF, and what is a runtime app configuration. In comparison this is basically trivial ootb with K8s. I personally found a lot of the moving parts and names needlessly confusing. Tasks e.g. were not your equivalent to “Deployment”. Want to just deploy something like Prometheus Agent? Well, too bad, the networking doesn’t work the same, so here’s some overly complicated guide where you have to deploy some extra stuff which will no doubt not work right the first dozen times you try. Admittedly, Prom can be a right pain to manage, but the fact that ECS makes you do _extra_ work on top of an already fiddly piece of software left a bad taste in my mouth. I think ECS get a lot of airtime because of Fargate, but you can use Fargate on K8s these days, or, if you can afford the small increase in initial setup complexity, you can just have Fargates less-expensive, less-restrictive, better sibling: Karpenter on Spot instances.
- 2y ago
- alganet 2y agoWhy do I feel this is not so simple as the compiler scenario? I've seen a lot of "piles of YAML", even contributed to some. There were some good projects that didn't end up in disaster, but to me the same could be said for the shell.
- czhu12 2y agoI think one thing that is under appreciated with kubernetes is how massive the package library is. It becomes trivial to stand up basically every open source project with a single command via helm. It gets a lot of hate but for medium sized deployments, it’s fantastic. Before helm, just trying to run third party containers on bare metal resulted in constant downtime when the process would just hang for no reason, and and engineer would have to SSH and manually restart the instance. We used this as a previous start up to host metabase, sentry and airbyte seamlessly, on our own cluster. Which let us break out of the constant price increases we faced for hosted versions of these products. Shameless plug: I’ve been building https://github.com/czhu12/canine https://github.com/czhu12/canine to try to make Kubernetes easier to use for solo developers. Would love any feedback from anyone looking to deploy something new to K8s!
- tptacek 2y agoRight, but this isn't a post about why K8s is good, it's a post about why K8s is effectively mandatory, and it isn't, which is why the post rankles some people.
- vanschelven 2y agoExactly, an alternative reading here is "metabase, sentry and airbyte are so complicated to self-host you'll need Kubernetes for it".
- czhu12 2y agoYeah I mostly agree. I'd even add that even K8 YAML's are not trivial to maintain, especially if you need to have them be produced by a templating engine.
- p_l 2y agoThey become trivial once you stop templating them with text templating engine. They are serialized json objects, the YAML is there just because raw JSON is not user friendly when you need something done quick and dirty or include comments. Proper templating should never use text templating on manifests.
- mildred593 2y agoStarted with a large shell script, the next itération was written in go and less specific. I still think for some things, k8s is just too much https://github.com/mildred/conductor.go/ https://github.com/mildred/conductor.go/
- knowsuchagency 2y agoFor my own websites, I host everything on a a single $20/month hetzner instance using https://dokploy.com/ https://dokploy.com/ and I'm never going back.
- hcrean 2y agoThe elephant in the room: People who have gotten over the K8s learning curve almost all tell you it isn't actually that bad. Most people who have not attempted the learning curve, or have just dipped their toe in, will tell you they are scared of the complexity. An anecdotal datapoint: My standard lecture teaching developers how to interact with K8s takes almost precisely 30 minutes to have them writing Helm charts for themselves. I have given it a whole bunch of times and it seems to do the job.
- nikisweeting 2y agoI personally know many k8s experts that vehemently recommend against using it unless you have no other option.
- thunky 2y ago> My standard lecture teaching developers how to interact with K8s takes almost precisely 30 minutes to have them writing Helm charts for themselves And I can teach someone to write "hello world" in 10 languages in 30 minutes, but that doesn't mean they're qualified to develop or fix production software.
- hnthrowaway0328 2y agoOne has to start from somewhere I guess. I doubt anyone would learn K8s thoroughly before getting any such job. Tried once and the whole thing bored me out in the fourth video.
- eadmund 2y ago> People who have gotten over the K8s learning curve almost all tell you it isn't actually that bad. I have been using K8s for nearly a decade. I use it both professionally and personally. I chose to use it personally. I appreciate why it exists, and I appreciate what it does. I believe that I have gotten over the learning curve. And I will tell you: it really is that bad. I mean, it’s not worse than childhood cancer. But it is a terrible, resource-heavy, misbegotten system. Its data structures are diseased. Its architecture is baroque. It is a disaster. But it’s also useful, and there is currently no real alternative. There really should be, though. I strongly believe that there can be, and I hope that there will be. The first time someone started writing templated YAML should have been the moment of clarity.
- skeptrune 2y agoI was very scared of K8s for a long time then we started using it and it's actually great. Much less complex than its reputation suggests.
- umvi 2y agoI had the exact opposite experience. I had a cloud run app in gcp and experimented with moving it to k8s and I was astonished with the amount of new complexity I had to manage
- reissbaker 2y agoEven without needing to spawn additional Docker containers, I think people are more afraid of Kubernetes than is warranted. If you use a managed K8s service like Azure, AWS, GCP, and tons of others provide, it's... Pretty simple and pretty bulletproof, assuming you're doing simple stuff with it (i.e. running a standard web app). The docs for K8s are incredibly bad for solo devs or small teams, and introduce you to a lot of unnecessary complexity upfront that you just don't need: the docs seem to be written with megacorps in mind who have teams managing large infrastructure migrations with existing, complex needs. To get started on a new project with K8s, you just need a pretty simple set of YAML files: 1. An "ingress" YAML file that defines the ports you listen to for the outside world (typically port 80), and how you listen to them. Using Helm, the K8s package manager, you can install a simple default Nginx-based ingress with minimal config. You probably were going to put Nginx/Caddy/etc in front of your app anyway, so why not do it this way? 2. A "service" YAML file that allocates some internal port mapping used for your web application (i.e. what port do you listen on within the cluster's network, and what port should that map to for the container). 3. A "deployment" YAML file that sets up some number of containers inside your service. And that's it. As necessary you can start opting into more features; for example, you can add health checks to your deployment file, so that K8s auto-restarts your containers when they die, and you can add deployment strategies there as well, such as rolling deployments and limits on how many new containers can be started before old ones are killed during the deploy, etc. You can add resource requests and limits e.g. make sure my app has at least 500MB RAM, and kill+restart it if it cross 1GB. But it's actually really simple to get started! I think it compares pretty well even to the modern Heroku-replacements like Fly.io... It's just that the docs are bad and the reputation is that it's complicated — and a large part of that reputation is from existing teams who try to do a large migration, and who have very complex needs that have evolved over time. K8s generally is flexible enough to support even those complex needs, but... It's gonna be complex if you have them. For new projects, it really isn't. Part of the reason other platforms are viewed as simpler IMO is just that they lack so many features that teams with complex needs don't bother trying to migrate (and thus never complain about how complicated it is to do complicated things with it). You can have Claude or ChatGPT walk you through a lot of this stuff though, and thereby get an easier introduction than having to pore through the pretty corporate official docs. And since K8s supports both YAML and JSON, in my opinion it's worth just generating JSON using whatever programming language you already use for your app; it'll help reduce some of the verbosity of YAML.
- jbverschoor 2y agoFriend, you built a J2EE server
- jeroenhd 2y agoKubernetes is so easy that you only need two or three dedicated full-time employees to keep the mountains of YAML from collapsing in on themselves before cutting costs and outsourcing your cluster management to someone else. Sure, it can be easy, just pick one of the many cloud providers that fix all the complicated parts for you. Though, when you do that, expect to pay extra for the privilege, and maybe take a look at the much easier proprietary alternatives. In theory the entire thing is portable enough that you can just switch hosting providers, in practice you're never going to be able to do that without seriously rewriting part of your stack anyway. The worst part is that the mountains of YAML were never supposed to be written by humans anyway, they're readable configuration your tooling is supposed to generate for you. You still need your bash scripts and your complicated deployment strategies, but rather than using them directly you're supposed to compile them into YAML first. Kubernetes is nice and all but it's not worth the effort for the vast majority of websites and services. WordPress works just fine without automatic replication and end-to-end microservice TLS encryption.
- maxdo 2y agoI ran small bootstrapped startup , I used GKE. Everything was templated. each app has it's own template e.g. nodejs-worker, and you don't change the template unless you really needed. i spent ~2% of my manger+eng leader+hiring manger+ god knows what else people do at startup on managing 100+ microservices because they were templates.
- chrisandchris 2y agoWe may have a different understanding of "small" if you say 100+ services. Did each employee have 2 to 3 services to maintain? If so, that sounds like an architectural mistake to me.
- lmm 2y agoThat works great until you want to change something low-level and have to apply it to all those templates.
- 2y ago
- stevefan1999 2y agoYes, but the people just cannot comprehend the complexity of it. Even my academic professor for my FYP back when was an undergrad, now he reverted back to Docker Compose, citing the integration is so convoluted that developing for it is very difficult. That's why I'm aiming to cut down the complexity of Kubernetes with a low-friction, turnkey solution, but I guess the angel investors in Hong Kong aren't buying into it yet. I'm still aiming to try again after 2 years when I can at least get an MVP that is complete though (I don't like to present imperfect stuff, either you just have the idea or you give me the full product and not half baked shit)
- deleted 2y ago[deleted]
- jb_gericke 2y agoI don’t think scale is the only consideration for using Kubernetes. The ops overhead in managing traditional infrastructure, especially if you’re a large enterprise, drops massively if you really buy into cloud native. Kubernetes converges application orchestration, job scheduling, scaling, monitoring/observability, networking, load balancing, certificate management, storage management, compute provisioning - and more. In a typical enterprise, doing all this requires multiple teams. Changes are request driven and take forever. Operating systems need to be patched. This all happens after hours and costs time and money. When properly implemented and backed by the right level of stakeholder, I’ve seen orgs move to business day maintenance, while gaining the confidence to release during peak times. It’s not just about scale, it’s about converging traditional infra practices into a single, declarative and eventually consistent platform that handles it all for you.
- fizx 2y agoMuch like Javascript, the problem isn't Kubernetes, its the zillions of half-tested open-source libraries that promise to make things easier but actually completely obfuscate what the system is doing while injecting fantastic amounts of bugs.
- donatj 2y agoI'm giggling at the idea you'd need Kubernetes for a mere two servers. We don't run any application with less than two instances for redundancy. We've just never seen the need for Kubernetes. We're not against it as much as the need to replace our working setup just never arrived. We run EC2 instances with a setup shell script under 50loc. We autoscale up to 40-50 web servers at peak load of a little over 100k concurrent users. Different strokes for different folks but moreso if it ain't broke, don't fix it
- walterburns 2y agoI wish the world hadn't consolidated around Kubernetes. Rancher was fantastic. Did what 95% of us need, and dead simple to add and manage services.
- globular-toast 2y agoDid you find Rancher v2 (which uses Kubernetes instead of their own Cattle system) is worse?
- 015a 2y agoI am 100% sure that the author of this post has never "built a kubernetes", holds at least one kubernetes cert, and maybe even works for a company that sells kubernetes products and services. Never been more certain of anything in my life. You could go point by point but its just so tiring arguing with these people. Like, the whole "who will maintain these scripts when you go on vacation" my brother in christ have you seen the kubernetes setups some of these people invent? They are not easier to be read into, this much is absolute. At least a shell script has a chance of encoding all of its behavior in the one file, versus putting a third of its behavior in helm variables, a third in poorly-named and documented YAML keys, and a third in some "manifest orchestrator reconciler service deployment system" that's six major versions behind an open source project that no one knows who maintains anymore because their critical developer was a Belarusian 10x'er who got mad about a code of conduct that asked him to stop mispronouning contributors.
- selcuka 2y ago> Inevitably, you find a reason to expand to a second server. The author has some good points, but not every project needs multiple servers for the same reasons as a typical Kubernetes setup. In many scenarios those servers are dedicated to separate tasks. For example, you can have a separate server for a redundant copy of your application layer, one server for load balancing and caching, one or more servers for the database, another for backups, and none of these servers requires anything more than separate Docker Compose configs for each server. I'm not saying that Kubernetes is a bad idea, even for the hypothetical setup above, but you don't necessarily need advanced service discovery tools for every workload.
- seanhunter 2y agoIt sometimes blows my mind how reductionist and simplistic a world-view it's possible to have and yet still attain some degree of success. Shovels and mechanical excavators both exist and have a place on a building site. If you talk to a workman he may well tell you he has regular hammer with him at all times but will use a sledgehammer and even rent a pile driver on occasion if the task demands it. And yet somehow we as software engineers are supposed to restrict ourselves to The One True Tool[tm] (which varies based on time and fashion) and use it for everything. It's such an obviously dumb approach that even people who do basic manual labour realise its shortcomings. Sometimes they will use a forklift truck to move things, sometimes an HGV, sometimes they will put things in a wheelbarrow and sometimes they will carry them by hand. But us? No. Sophisticated engineers as we are there is One Way and it doesn't matter if you're a 3 person startup or you're Google, if you deploy once per year to a single big server or multiple times per day to a farm of thousands of hosts you're supposed to do it that one way no matter what. The real rule is this: Use your judgement. You're supposed to be smart. You're supposed to be good. Be good. Figure out what's actually going on and how best to solve the problems in your situation. Don't rely on everyone else to tell you what to do or blindly apply "best practises" invented by someone who doesn't know a thing about what you're trying to do. Yes consider the experiences of others and learn from their mistakes where possible, but use your own goddamn brain and skill. That's why they pay you the big bucks.
- d0mine 2y ago“Everything is a tradeoff” is the popular (if not the most popular) maxim in distributed computing.
- pkstn 2y agoSwarm Mode ftw!
- KronisLV 2y agoWhat I really like are solutions that are simple to both install and maintain. For example, look at Docker Swarm: https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/ https://docs.docker.com/engine/swarm/swarm-tutorial/create-s... or Hashicorp Nomad: https://developer.hashicorp.com/nomad/tutorials/get-started/gs-start-a-cluster https://developer.hashicorp.com/nomad/tutorials/get-started/... or Kubernetes distros like K3s: https://docs.k3s.io/quick-start https://docs.k3s.io/quick-start Either of those would work better than a bespoke solution a lot of the time, for anyone who is not the original person writing that hodgepodge of shell scripts, because of how much information and documentation exists out there about those solutions. Either of those would also work better than a very complex Kubernetes cluster that was built with HA in mind and tries to be web scale for anyone who isn't a DevOps engineer that's ready to spend a bunch of time maintaining it, given how complex things can get. Curiously, a previous article by the author calls out Docker Compose as not being good enough https://www.macchaffee.com/blog/2024/docker-compose/ https://www.macchaffee.com/blog/2024/docker-compose/ and also critiques Docker Swarm without much elaboration (because that's not the main focus): > Single-node only. Many apps never reach a point where they need more than one node, but having to either rip out your entire existing deployment method or invest in Swarm are not good options. I've seen both Docker Compose and Docker Swarm be used in production with good results - in problem spaces where minutes or hours of downtime (e.g. non-global audience, evenings) was acceptable and having zero downtime deployments wasn't required, though usually there were no big outages anyways. Admittedly, that's a pretty good place to be in, even if not that much interesting happens there. Just pick whatever fits your requirements, there are also very sane Kubernetes distros out there, even for something like test environments or running locally.
- glonq 2y agoWe chose Docker Swarm for a new project that we started two years ago, mostly because our developers liked its learning curve more than k8s's. We are happily running on Docker Swarm in production now, and don't see that changing in the foreseeable future.
- tonymet 2y agoDir friend, you just built a giant YAML-based RPC system to manage your symbol table when you should have just used a linker.
- kittikitti 2y agoIt was horrible advice from Big Tech professionals when I asked whether I should use docker compose or Kubernetes for a project. The thing is, they were so condescending and belittling that I wasn't using Kubernetes. After switching over to Kubernetes, I realized that it was a huge mistake as I had another layer of complexity, installs, and domain specific languages that infected every part of the code base.
- znpy 2y ago> In the last leg of your journey to avoid building a Kubernetes, your manager tells you that your app needs to programmatically spawn other containers. Spawning containers, of course, requires you to mount the Docker socket in your web app, which is wildly insecure. This was true ten years ago, it's not been true for at least 2-3 years. You can run rootless podman in kubernetes (I did) and you can launch pods from there. Securely.