11 ms·

I’ve come to the conclusion that I hate “cloud shit”, and a small part of me is convinced that literally no one actually likes it, and everyone is playing a jok

I’ve come to the conclusion that I hate “cloud shit”, and a small part of me is convinced that literally no one actually likes it, and everyone is playing a joke on me. I have set up about a dozen rack mount servers in my life, installing basically every flavor of Unix and Linux and message busses under the sun in the process, but I still get confused by all the Kubectl commands and GCP integration with it. I might just be stupid, but it feels like all I ever do with Kubernetes is update and break YAML files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange. I cannot imagine how anyone goes to work and actually enjoys working in Kubernetes, though I guess someone must in terms of “law of large numbers”. If I ever start a company, I am going to work my damndest to avoid “cloud integration crap” as possible. Just have a VM or a physical server and let me install everything myself. If I get to tens of millions of users, maybe I’ll worry about it then.

tryauuum 2y ago
I have same thoughts. the only form of kubernetes I would be willing to try is the one with kata-containers for having all the security of virual machines.
nyclounge 2y ago
Or if you got a static ip and fast up speed, then just port forward 80, 443 and start hosting your self. Even an old Intel Mac Book pro from 2000's with 4 GB of RAM may not be that hot running MacOS, but install Debian with no X. It is running smooth as a whistle, while running several conduit matrix, haraka, zone-mta, ice cast, nginx with no issues. WebRTC/TUN/STUN becomes an issue with the nginx config. May consider looking at pingora. The whole rust -> binary + toml file is super nice to run from system admin perspective.
- d3Xt3r 2y ago
  > It is running smooth as a whistle ... until you get hit by a DDoS attack. Not much you can do about it unless your ISP offers protection, or you end up going for Cloudflare or the like instead of exposing your IP and ports.
  - PittleyDunkin 2y ago
    Folding the second you get dos'd is aa feature, not a liability. Staying up through those is a great way to lose a ton of money basically instantly.
  - nyclounge 2y ago
    Or hetzner has also DDoS protect for all their customers. Seems like a lot less rip off and scam than AWS and Cloudflare. Set your TLS a low number, and you can swap whenever you feel like it.
tapoxi 2y ago
This read as "old man yells at cloud" to me. I've managed a few thousand VMs in the past, and I'm extremely grateful for it. An image is built in CI, service declares what it needs, the scheduler just handles shit. I'm paged significantly less and things are predictable and consistent unlike the world of VMs where even your best attempt at configuration management would result in drift, because the CM system is only enforcing a subset of everything that could go wrong. But yes, Kubernetes is configured in YAML, and YAML kind of sucks, but you rarely do that. The thing that changes is your code, and once you've got the boilerplate down CI does the rest.
- catdog 2y ago
  YAML is fine, esp. compared to the huge collection of often 10x worse config formats you have to deal with in the VM world.
- cess11 2y ago
  I'd prefer Ansible if I was running VM:s. Did that at a gig, controlled a vCenter cluster and hundreds of machines in it, much nicer experience than Kubernetes style ops. Easier to do ad hoc troubleshooting and logging for one.
  - vrighter 2y ago
    until, as happened to us, you're in the middle of an upgrade cycle, with a mix of red-hat 6 and red-hat 8 servers, and ansible decide to require support for the latest available version of python on red-hat 8, which isn't available on red-hat 6, so we have no way of using ansible to manage both sets of servers. The python ecosystem is a cancer.
    - cess11 2y ago
      Sure, and I'm also not a fan of RedHat. We ran Ubuntu and Debian on that gig, the few Python issues we ran into we could fix with some package pinnings and stuff like that.
    - mkesper 2y ago
voidfunc 2y ago
I'm always kind of blown away by experiences like this. Admittedly, I've been using Kubernetes since the early days and I manage an Infra team that operates a couple thousand self-managed Kubernetes clusters so... expert blindness at work. Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy. Maybe it's because I adopted early and have grown with the technology it all just makes sense? It's not that complicated if you limit yourself to the core stuff. Maybe I need to write a book like "Kubernetes for Greybeards" or something like that. What does fucking kill me in the Kubernetes ecosystem is the amount of add-on crap that is pitched as "necessary". Sidecars... so many sidecars. Please stop. There's way too much vendor garbage surrounding the ecosystem and dev's rarely stop to think about whether they should deploy something when it's easy as dropping in some YAML and letting the cluster magically run it.
- theptrk 2y ago
  I would pay for the outline of this book.
- t-writescode 2y ago
  > Admittedly, I've been using Kubernetes since the early days and I manage an Infra team I think this is where the big difference is. If you're leading a team and introduced all good practices from the start, then the k8s and Terraform or whatever config files can never get so very complicated that a Gordian knot isn't created. Perhaps k8s is nice and easy to use - many of the commands certainly are, in my experience. Developers have, over years and decades, learned how to navigate code and hop from definition to definition, climbing the tree and learning the language they're operating in, and most of the languages follow similar-enough patterns that they can crawl around. Configuring a k8s cluster has absolutely none of that knowledge built up; and, reading something that has rough practices is not a good place to learn what it should look like.
  - paddy_m 2y ago
    Thank you. I can always xargs grep for a function name, at worst. Dir() in python at a debugger for other things. With YAML, kubernetes and other devops hotness, I frequently can’t even find the relevant scripts/YAML that are executed nor their codebases. This also happens with configuration based packaging setups. Python hatch in particular, but sometimes node/webpack/rollup/vite.
cheriot 2y ago
> I might just be stupid, but it feels like all I ever do with ____ is update and break ____ files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange. I cannot imagine how anyone goes to work and actually enjoys working in ____, though I guess someone must in terms of “law of large numbers”. I'd make a similar statement about the sys admin stuff you already know well. Give me yaml and a rest api any day. I see where you and the article are coming from, though. The article reasonably points out that k8s is heavy for simpler workloads.
stiray 2y ago
I agree, but what pisses me the most is that today higher level abstractions (like cloud, spring boost,...) are hiding lower level functionality so well, that you are literally forced to use obnoxious amounts of time to study documentation (if you are in luck and it is well written), while everything is decorated with new naming of known concepts that was invented by people who didn't know that the concept already exists and has a name or some marketing guy figured out it will sell better with more "cool" name. Like Shakespeare work would be clumsy and half translated to french advertising jargoon and you are forced to read it and make it work on a stage.
- datadeft 2y ago
  The real benefits of LLMs is to give me a good summary of these documents. Still, using these abstractions is probably not worth it.
  - PittleyDunkin 2y ago
    Well, a summary. It's impossible to evaluate the quality of the summary without actually reading the document you're trying to avoid reading.
  - stiray 2y ago
    LLMs are a symptom, not a solution. Just another over-hyped bullshit (those downvoting will agree in few years or never as they are the ones riding the hype) that its only concern is to boost company stocks. Google is the proof, their search started to suck immediately when they added AI into concept. It promotes bullshit, while it doesn't hit the really relevant results, even if you specify them into the details. If AI would have any real value, it would never be given out for free and this is the case: they have a solution and they are searching for a problem it solves.
devjab 2y ago
I don’t mind the cloud, but even in enterprise organisations I fail to see the value of a lot of the more complex tools. I’ve anlways worked with Azure because Denmark is basically Microsoft territory in a lot of non-tech organisations because of the synergy between pricing and IT operations staff. I’ve done bicep, terraform and both Kubernetes and the managed (I forgot what azure conteiner apps running on top of what is basically Kubernetes is called). When I can get away with it I always use the Azure CLI through bash scripts in a pipeline however and build directly into Azure App services for contained which is just so much less complicated than what you probably call “cloud shit”. The cool part about the Azure CLI and their app services is that it hasn’t really changed in the past 3 years, and they are almost one size fit any organisation. So all anyone needs to update in the YAML scripts are the variables. By contrast working with Bicep/Terraform, Jenkins and whatever else people use has been absolutely horrible, sometimes requiring full time staff just to keep it updated. I suppose it may be better now that azure co-pilot can probably auto-generate what you need. A complete waste of resources in my opinion. It used to be more expensive, but with the last price hike of 600% on azure container apps it’s usually cheaper. It’s also way more cost efficient in terms of maintaining since it’ll just work after the initial setup pipeline has run. This is the only way I have found that is easier than what it was when organisations ran their own servers. Whether it was in the basement or at some local hardware house (not exactly sure what you call the places where you rent server rack space). Well places like Digital Ocean are even easier but they aren’t used by enterprise. I’m fairly certain I’ve ever worked with an organisation that needed anything more than that since basically nothing in Denmark scales beyond what can run on a couple of servers behind a load balancer. One of the few exceptions is the tax system which sees almost 0 usage except for the couple of weeks where the entire adult population logs in in at the same time. When DevOps teams push back, I tend to remind them that StackOverflow ran on a couple of IIS servers for a while and that they don’t have even 10% of the users. Eventually the business case for Azure will push people back to renting hardware space or jumping to Hetzner and similar. But that’s a different story.
- DanielHB 2y ago
  Terraform has the same problem as Kubernetes sidecars with terraform providers trying to do magic for you. If you stick to the cloud platform provider it is actually much nicer than using the CLI. Although my experience is with AWS, I find the terraform AWS provider docs better documentation than the official AWS docs for different options. If they don't answer any question I have right away they at least point me where to look for answers in the mess that is AWS docs.
ozim 2y ago
I am running VPSes at our small startup-ish company on IaaS cloud. Every time we get a new guy I have to explain that we are already „in cloud” there is no need to „move to cloud”.
- rcleveng 2y ago
  Do they mean PAAS vs IAAS when they say "move to cloud"?
  - ozim 2y ago
    Mostly business guys don’t know the difference but we are running on local cloud provider and they think of it is not on Azure or AWS it is not in cloud - they understand that we run stuff on servers but they also don’t understand VPS is IaaS. Developers want to use PaaS and also AWS or Azure so they can put it on their resume for the future.
    - bluehatbrit 2y ago
      > Developers want to use PaaS and also AWS or Azure so they can put it on their resume for the future. I think this is a little disingenuous. Developers want to use them because they already know them. The services composing them are also often well documented by the provider. I say all of that as someone trying to move a company away from aws, and over to our own hardware.
      - PittleyDunkin 2y ago
        I want to use them because I'm not paying for it and managing your own hardware is a pain in the ass I'd rather avoid when I could just code instead. Which is not to say that using your own hardware isn't smart, but it is definitely miserable.
        regularfry 2y ago
        In my experience you swap managing your own hardware for managing intensely obtuse service configurations, pretty much 1:1. That might be preferable but I see a lot of folks approach it like the tradeoff doesn't exist.
misswaterfairy 2y ago
I hate "cloud shit" as well, though specifically that there's a vendor-specific 'app', or terminology, or both, for everything that we've had standard terms for, for decades. I just want a damn network, a couple of virtual machines, and a database. Why does each <cloud provider> have to create different fancy wrappers over everything, that not even their own sales consultants, and even engineers, understand?(1) What I do like about Docker and Kubernetes is that shifting from one cloud provider to another, or even back to on-premises (I'm waiting for the day our organisation says "<cloud-provider> is too damn expensive; those damn management consultants lied to us!!!!") is a lot easier than re-building <cloud provider>'s bespoke shit in <another cloud provider>'s bespoke shit, or back on-premises with real tech (the right option in my opinion for anyone less than a (truly) global presence). I do like the feel of, and being able to touch bare metal, though the 180-proof-ether-based container stuff is nice for quick, flexible, and (sometimes) dirty. Especially for experimenting when the Directors for War and Finance (the same person) say "We don't have the budget!! We're not buying another server/more RAM/another disk/newer CPUs! Get fucked!". The other thing about Docker specifically I like is I can 'shop' around for boilerplate templates that I can then customise without having to screw around manually building/installing them from scratch. And if I fuck it up, I just delete the container and spin up another one from the image. (1) The answer is 'vendor lock-in', kids. (I apologise, I've had a looooooong day today.......)
- yungporko 2y ago
  agreed, cloud shit can fuck off. they market themselves as the solution to problems you will never have so that you build your shit on their land and they can charge you rent, and most people eat it up and think they're doing them a favour.
fragmede 2y ago
I'll let you in on the joke. The joke is the demand for 100% availability and instant gratification. we're making services where anything less than 4 nines, which is 5 minutes month, is deemed unacceptable. three nines is 10 minutes a week. two nines is 15 minutes a day. there are some things that are important enough that you can't take a coffee break and wait for, but Kubernetes lets you push four nines of availability, no problem. Kubernetes is solving for that level of availability, but my own body doesn't have anything near that level of availability. demanding that from everything and everyone else is what pushes for Kubernetes level of complexity.
- kitd 2y ago
  Are you the only one waiting on your app if it goes down?
- EVa5I7bHFq9mnYK 2y ago
  4 nines is a lie. That's not what I experience as a user. I experience 403, 404, 405, 500, or other forms of down, such as difficult captchas, 2fa not working, geolocation blockages etc several times a day.
- spacebanana7 2y ago
  Most outages these days are also caused by "devops errors" that're much more likely to happen when working with complex infrastructure stacks. So there's real value in keeping things simple.
- karmarepellent 2y ago
  Its a matter of evaluating what kind of infrastructure your application needs to run on. There are certainly mission critical systems where even a sliver of downtime causes real damage, like lost revenue. If you come to the conclusion that this application and everything it involves better run on k8s for availability reasons, you should probably focus on that and code your application in a k8s-friendly manner. But there are tons of applications that run on over-engineered cloud environments that may or may not involve k8s and probably cost more to operate than they must. I use some tools every day where a daily 15 min downtime would not affect my or my work in the slightest. I am not saying this would be desirable per se. Its just that a lot of people (myself included) are happy to spend an hour of their work day talking to colleagues and drinking coffee, but a 15 min downtime of some tool is seen as an absolute catastrophe.
cess11 2y ago
Once it gets hard to run the needed services for all the applications in the organisation/team on your development machine it'll start to look more attractive to turn to Docker/Podman and the like, and that's when automatic tests and deploys built on disgusting YAML starts to make more sense. I've been at a place where the two main applications were tightly coupled with several support systems and they all were dependent on one or more of Oracle DB, Postgres, Redis, JBoss, Tibco EMS and quite a bit more. Good luck using your development device to boot and run the test suite without containers. Before that team started putting stuff in containers they used the CI/CD environment to run the full set of tests, so they needed to do a PR, get it accepted, maybe wait for someone else's test run to finish, then watch it run, and if something blew, go back to commit, push to PR, &c. all over again. Quite the nuisance. A full test suite had a run time of about an hour too. When I left we'd pushed it to forty minutes on our dev machines. They didn't use raw Kubernetes though, they had RedHat buy-in and used OpenShift, which is a bit more sane. But it's still a YAML nightmare that cuts you with a thousand inscrutable error messages.
teekert 2y ago
Sorry to have to tell you this, but you’re old. Your neural plasticity has gone down and you feel like you have seen it al before. As a result you cling to the old and never feel like you grasp the new. The only reasonable thing to is to acknowledge and accept this and try not let it get in your way. Our generation has seen many things before, but at the same time the world has completely changed and it’s led to the people growing up in it to be different. You and me didn’t fully grasp CPUs anymore. Some people today don’t grasp all the details of the abstractions below K8s anymore and use it when perhaps something simpler (in architecture , not necessarily in use!) could do it better. And yet, they build wonderous things. Without editing php.ini and messing up 2 services to get one working. Do I think K8s is the end all? Certainly not, I agree it’s sometimes overkill. But I bet you’ll like it’s follow-up tech even less. It is the way of things.
- tedk-42 2y ago
  > Is K8s the end all? Certainly not, I agree it’s sometimes overkill. But I bet you’ll like it’s follow-up tech even less. It is the way of things. I agree with your analysis. People wanna talk up about how good the old days were plugging cables into racks but it's really laborious and can take days to debug that a faulty network switch is the cause of these weird packet drop issues seen sporadically on hot days. Same as people saying 'oh yeah calculators are too complicated, pen and paper is what kids should be learning'. It's the tide of change
  - tombert 2y ago
    I reject that comparison, I'm not really resistant to change, I'm resistant the awful bureaucratic crap that k8s and its ilk force you to use. It's not fun, as far as I can tell no one actually understands any of it (young or old), they just copy and past large blocks of YAML from blogs and documentation and then cross their fingers. I'm not saying that plugging in cables and hoping power supplies don't die is "better" in any kind of objective sense, or even subjective sense really, I'm just saying that I hate this cloud cult that has decided that the only way to do anything is to add layers of annoying bureaucratic shit.
- creesch 2y ago
bob1029 2y ago
Cloud is a broad spectrum of tools. I think the Goldilocks zone is something right in between bare-ass VMs and FaaS. If I can restart the machine and it magically resolves the underlying hardware fault (presumably by migrating me to a new host), then I am in a happy place. Most of the other problems can be dealt with using modern tooling. I lean aggressively on things like SQLite and self-contained deployments in .NET to reduce complexity at hosting time. When you can deploy your entire solution with a single zip file you would be crazy to go for something like K8s. One other cloud thing that is useful is S3-style services. Clever use of these APIs can provide incredible horizontal scalability for a single instance solution. SQLite on NVMe is very fast if you are offloading your blobs to another provider.
grishka 2y ago
> If I get to tens of millions of users, maybe I’ll worry about it then. Nope, then you'll set up sharded databases and a bunch of application servers behind a load balancer.
raffraffraff 2y ago
I'm sure you can get some of the handy extras that come with a typical kubernetes deployment without the kubernetes, but overall I'll take kubernetes + cloud. Once you've got the hang of it, it's ok. I have a terraform project that deploys clusters with external-dns, external-secrets, cert-manager, metrics, monitoring stack, scalers and FluxCD. From there, pretty much everything else is done via FluxCD (workloads, dashboards, alerts). And while I detest writing helm charts (and sometimes using them, as they can get "stuck" in several ways) they do allow you to wrap up a lot of the kubernetes components into a single package that accepts more-or-less standardized yaml for stuff like resource limits, annotations (eg for granting and AWS role to a service) etc. And FluxCD .postBuild is extremely handy for defining environment vars to apply to more generic template yaml, so we avoid a sprawl. So much so that I am the one-man-band (Sys|Dev|Sec)Ops for our small company, and that doesn't give me panic attacks. The cloud integration part can be hairy but I have terraform patterns that, once worked out, are cookie cutter. With cloud kubernetes, I can imagine starting from scratch, taking a wrong turn and ending up in hell. But I'm exchanging one problem set for another. Having spent years managing fleets of physical and virtual servers, I'm happier and more productive now. I never need to worry about building systems or automation for doing OS build / patching, config management, application packaging and deployment, secrets management, service discovery, external DNS, load balancering, TSL certs etc. Because while those are just "words" now, back then each one was a huge project involving multiple people fighting over "CentOS Vs Ubuntu", "Puppet Vs Ansible", "RPMs Vs docker containers", "Patching Vs swapping AMIs". If you're using Consul and Vault, good luck - you have to integrate all of that into whatever mess you've built, and you'll likely have to write puppet code and scripts to hook it all up together. I lost a chunk of my life writing 'dockerctl' and a bunch of puppet code that deployed it so it could manage docker containers as systemd services. Then building a vault integration for that. It worked great across multiple data centers, but took considerable effort. And in the end it's a unique snowflake used by exactly one company, hardly documented and likely full of undiscovered bugs and race conditions even after all the hard work. The time it took to onboard new engineers was considerable and it took time away from an existing engineer. And we still had certificates expire in production.
datadeft 2y ago
One thing that might help you in this madness is: https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes Type safe, fat finger safe representation of your YAMLs is grossly underrated.
friendzis 2y ago
In my limited experience k8s is treated as some magic instead of what it is: resource virtualization platform. Sort one (or two) step above hardware virtualization platforms (e.g. esxi). The fact that vmware can migrate a running vm to a different hardware node is surely powerful feature. Do you want to pay for that with complexity? If you are one infra team serving on-prem deployments with consistent loads you provision things when new projects are started and things break. However, if infra team serves internal product teams it is nice to give certain guarantees to them and limit blast radius how they can affect other teams. This is where kubernetes sit. It's a deployment platform, where the deployable is container image instead of VM image. Slice off a namespace to an internal team and have their deployment blast radius contained within their namespace. Do you need such flexibility? I'm pretty sure that roughly 99% of all teams do not. A static set of VMs provisioned with some container orchestration system is more than enough for them. Loads and deployments are too static for it to matter. >>> But it allows seamless upgrades! Dude or dudette, if you can't properly drain your nodes and migrate sessions now, kubernetes will not save you.
torginus 2y ago
Kubernetes starts sucking at first sight and is kind of the sign of the ailment of modern times. Let me try to explain: First, you encounter the biggest impedance mismatch between cloud and on prem: Kubernetes works with pods, while AWS works with instances as the unit of useful work, so they must map to each other, right? Wrong, first each instance needs to run a Kubernetes node, which duplicates the management infrastructure hosted by AWS, and reduces the support for granularity, like if I need 4 cores for my workload, I start an instance with 4 cores, right?. Not so with k8s, you have to start up a big node, then schedule pods there. Yay, extra complexity and overhead! And it's like when you need 3 carrots for the meal your cooking, but the store only sells it in packs of 8, you have to pay for the ones you don't need, and then figure out how to utilize the waste. I'm not even going to talk about on-prem kubernetes, as I've never seen anyone IRL use it. Another thing is that the need for kubernetes is manifestation of crappy modern dev culture. If you wrote your server in node, Python and Ruby, you're running a single threaded app in the era of even laptops having 32 core CPUs. And individual instances are slow, so you're even more dependent on scaling. So, to make use of the extra CPU power, you're forced to turn to Docker/k8s and scale your infra that way, whereas if you went with something like Go, or god forbid, something as deeply uncool as ASP.NET, you could just put your 32 core server, and you get fast single threaded perf, and perfect and simple multi-threaded utilization out of the box without any of the headaches. Also I've found stuff like rolling updates to be a gimmick. Also a huge disclaimer, I don't think k8s is a fundamentally sucky or stupid thing, it's just I've never seen it used as a beneficial architectural pattern in practice.
- erinaceousjones 2y ago
  Being nitpicky about Python specifically, Python not necessarily single-threaded; gunicorn gets you a multiprocess HTTP/WSGI server if you configure it to. asyncio and gevent have made it easier to do things actually-concurrently. Judicious use of generator functions lets you stream results back instead of blocking I/O for big chunks. And we have threads. Yeah, the Global Interpreter Lock is still hanging around, it's not the fastest language, but there are ways to produce a Docker image of an API written in Python which can handle thousands of concurrent HTTP requests and actively use all available CPU cores to do intensive computation.
jmb99 2y ago
I have pretty much the exact same opinion, right down to “cloud shit” (I have even used that exact term at work to multiple levels of management, specifically “I’m not working on cloud shit, you know what I like working on and I’ll do pretty much anything else, but no cloud shit”). I have worked on too many projects where all you need is a 4-8 vCPU VM running nginx and some external database, but for some reasons there’s like 30 containers and 45 different repos and 10k lines of terraform, and the fastest requests take 750ms which should be nginx->SQL->return in <10ms. “But it’s autoscaling and load balanced!” That’s great, we have max 10k customers online at once and the most complex requests are “set this field to true” and “get me an indexed list.” This could be hosted on a raspberry pi with better performance. But for some reason this is what people want to do. They would rather spend hours debugging kubernetes, terraform, and docker, and spending 5 digits on cloud every month, to serve what could literally be proxied authenticated DB lookups. We have “hack days” a few times a year, and I’m genuinely debating rewriting the entire “cloud” portion of our current product in gunicorn or something, host it on a $50/month vps, point it at a mirror of our prod db, and see how many orders of magnitude of performance I can knock off in a day. I’ve only managed to convert one “cloud person” to my viewpoint but it was quite entertaining. I was demoing a side project[0] that involved pulling data from ~6 different sources (none hosted by me), concatenating them, deduping them, doing some math, looking up in a different source an image (unique to each displayed item), and then displaying the list of final items with images in a list or a grid. ~5k items. Load time on my fibre connection was 200-250ms, sorting/filtering was <100ms. As I was demoing this, a few people asked about the architecture, and one didn’t believe that it was a 750 line python file (using multiprocessing, admittedly) hosted on an 8 core VPS until I literally showed up. He didn’t believe it was possible to have this kind of performance in a “legacy monolithic” (his words) application. I think it’s so heavily ingrained in most cloud/web developers that this is the only option that they will not even entertain the thought that it can be done another way. [0] This particular project failed for other reasons, and is no longer live.
- jiggawatts 2y ago
  Speaking of Kubernetes performance: I had a need for fast scale-out for a bulk testing exercise. The gist of it was that I had to run Selenium tests with six different browsers against something like 13,000 sites in a hurry for a state government. I tried Kubernetes, because there's a distributed Selenium runner for it that can spin up different browsers in individual pods, even running Windows and Linux at the same time! Very cool. Except... Performance was woeful. It took forever to spin up the pods, but even once things had warmed up everything just ran in slow motion. Data was slow to collect (single-digit kilobits!), and I even saw a few timeout failures within the cluster. I gave up and simply provisioned a 120 vCPU / 600 GB memory cloud server with spot pricing for $2/hour and ran everything locally with scripts. I ended up scanning a decent chunk of my country's internet in 15 minutes. I was genuinely worried that I'd get put on some sort of "list" for "attacking" government sites. I even randomized the read order to avoid hammering any one site too hard. Kubernetes sounds "fun to tinker with", but it's actually a productivity vampire that sucks up engineer time. It's the Factorio of cloud hosting.
mrweasel 2y ago
I wouldn't throw "cloud shit" in the same bucket as Kubernetes. Professional cloud services are mostly great, but super expensive. Kubernetes is interesting, because it basically takes everything you know and sort of pushes it down the stack (or up, depending on your viewpoint). To some extend I get the impression that the idea was: Wouldn't it be great if we took all the infrastructure stuff, and just merged the whole thing into one tool, which you can configure using a completely unsuitable markup language. The answer is that "No, that would in fact not be great". For me the issue is that Kubernetes is overused. You can not really argue that it's not useful or has its place, but that place is much much small than the Internet wants us to believe. It's one of the few services where I feel like "Call us" would be an appropriate sales method. The article is correct, you probably don't need Kubernetes. It's a amazing piece of technology, but it's not for the majority of us. It is and should be viewed as a niche product.
- nixpulvis 2y ago
  What do you think its niche is?
  - mrweasel 2y ago
    Very large "single product" services, the focus being on "very large". It would also be relevant in the cases where your product is made up of a large number of micro-services, though if you have more than 50 services to support your main product I'd be interested in knowing why you have that many services. There might be completely sane reasons, but it is a special case. Mostly a question of scale to me, I'd guess that the majority (80-90%) of people running Kubernetes don't have large enough scale that it makes sense to take on the extra complexity. Most Kubernetes installations I've seen runs on VMs, three for control plane and 2 - 5 for worker node and I don't think the extra layer is a good trade off for a "better" deployment tool. If you do use Kubernetes as a deployment tool, then I can certainly understand that. It is a reasonably well-known, and somewhat standardised interface and there's not a lot of good alternatives for VMs and bare metal. Personally I'd just much rather see better deployment tools being developed, rather than just picking Kubernetes because Helm charts are a thing. You'd also need to have a rather dynamic workload, in the sense that some of your services is need a lot of capacity at one point in time, while other need the capacity at another time. If you have constant load, then why? It's like Oracle's Exadata servers, it's a product that has its place, but the list of potential buyers isn't that long.
kortilla 2y ago
All of these anecdotes seem to come from people who don’t bother to try to learn kubernetes. > YAML files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange. This is terrible behavior. Its not any different from yanking out pam modules because you’re getting SSH auth failures caused by a bad permission on an SSH key. > If I get to tens of millions of users, maybe I’ll worry about it then. K8s isn’t there for 10s of millions of users. It’s there so you’re not dependent on some bespoke VM state. It also allows you to do code review on infra changes like port numbers being exposed, etc. Separately, your VM likely isn’t coming from any standard build pipeline so now a vulnerability patch is a login to the machine and an update, which hopefully leaves it in the same state as VMs created new… Oh, and assuming you don’t want to take downtime on every update, you’ll want a few replicas and load balancing across them (or active/passive HA at a minimum). Good luck representing that as reviewable code as well if you are running VMs. The people that don’t understand the value prop of infra as code orchestration systems like k8s tend to work in environments where “maintenance downtime” is acceptable and there are only one or two people that actually adjust the configurations.
- secondcoming 2y ago
  Just because you're using VMs doesn't mean you're now dealing with state. It's 100% possible to have stateless VMs running in an auto-scaling instance group (in GCP speak, I forget what AWS calls them)
  - everfrustrated 2y ago
    In the beginning AWS didn't even support state on their VMs! All VMs were ephemeral with no state persistence when terminated. They later introduced EBS to allow for the more classic enterprise IT use cases.
  - kortilla 2y ago
    Once you have the tools to manage all of that, you effectively have kubernetes. Container vs VM is largely irrelevant to what the op is complaining about when it comes to k8s. People that don’t like k8s tend to be fine with docker. It’s usually that they don’t like declarative state or thinking in selectors and other abstractions.
PittleyDunkin 2y ago
Isn't using VMs or servers just cloud shit in a different form? How does that fix anything? Orchestrating software (or hardware) is a pain in the ass whether it's in your closet or someone else's.
andyjohnson0 2y ago
Kubernetes was developed by Google, and it seems to me it's a classic example of a technology that can work well in a resource-rich (people, money) environment. But in an averge business it becomes a time/money sink quite quickly. Way too many businesses cargo-cult themselves into thinking that they need FAANG-class infra, even though they haven't got the same scale or the same level of resourcing. Devs and ops people love it because they get to cosplay and also get the right words on their CV. If you're not Google-scale then, as you say, a few VMs or physical boxes are all you need for most systems. But its not sexy, so the business needs people who don't mind that.
- xorcist 2y ago
  K8s is absolutely not Google-scale. Not even close. The scheduler is not built for it.
Woeps 2y ago
I don't mind K8s, and it can be usefull if you just stay to the core essentials. But I do agree that there seems to be the sentiment:"Everybody has to like it!" And I mostly think that this is because our collective bias for everything Big Tech does is good. While often it just depends. Just because Google does X or Y doesn't mean it will work for everybody else.
wordofx 2y ago
I’m confused. Do you hate cloud or the container shit? Cloud is awesome. Containers is dumb as hell for 99% of the applications hacker news kids work on.
- tombert 2y ago
  I don't know what "cloud" means. I don't really like Kubernetes, and I don't really like the AWS and GCP systems for doing everything. The fact that WGU offers a bachelors degree in Cloud Computing with focuses on AWS or Azure should be telling. [1] Containers don't bother me that much on their own. I just feel like with k8s and its ilk I end up spending so much time futzing with weird YAML and trying to remember differences between stateful sets and services and fighting with stuff because I missed the thing that mounts a disk so my app will break in three hours. [1] https://www.wgu.edu/online-it-degrees/cloud-computing-bachelors-program.html https://www.wgu.edu/online-it-degrees/cloud-computing-bachel...
chromanoid 2y ago
IMO the actual experience you want is something as simple as you can do with those old-school simple PHP-based "serverless" web hosters. You just upload your stuff and maybe configure a database URL and that's it. Regarding AWS, function.zip Lambda + DynamoDB + S3 + SQS is basically this at "enterprise-grade". Now you have the in-between left, where you want enterprise-grade (availability, scaling and fault tolerance) but with a catch (like lower costs, more control, data sovereignty or things that are not as serverless as you want, e.g. search engine etc.). In these cases you will run into many trade-off decisions, that may lead you to K8s, cloud specific stacks and also simple VM ClickOps / ShellOps setups. As long as you can still be on the pet instead of cattle range of problems, K8s is not what you want. But as soon as you want cattle, reproducible throw-away online environments etc. "just have a VM or a physical server" will become a "build your own private cloud solution" that may or may not become a bad private cloud / K8s.
antihero 2y ago
I think a good compromise is building your applications in a containerised manner so that you can simply run them with docker-compose, but then if it turns out you need some heavy scale, it's merely a case of switching the underlying infrastructure. That said, my experience has been fairly different. Running microk8s on a VPS with some containers and an ingress just seems like a nice way to spin them up as pods and manage them. It really doesn't feel that complicated. Once you integrate with cloud providers and you get more complex systems, sure, it gets more complex. I much prefer the container paradigm for separating parts of apps out and managing dependencies and reproducibility. Having a bunch of raw servers with all the bullshit and drift and mutability that comes along with it feels far more like a headache to me than the abstraction of k8s or docker. If you aren't deploying stuff with GitOps/IaC in 2024 I fear for anyone that has to manage or maintain it. 100% reproducibility and declarative infrastructure are wonderful things I could never come back from.
- pdimitar 2y ago
  I love the idea of declarative infrastructure but I'm never considering a bunch of YAML files as a good practice. Stuff like k9s / microk8s / k3s are clutches and workarounds and I hope we all see it. If they figure to use an actual programming language or just start using much smaller amount of files than they currently do then I'd be the first to learn k8s. Before that, nope. I love the idea but the implementation makes me want to slit my wrists.
gigatexal 2y ago
The yaml and the k8s model can be grokked it just takes a willingness to. That being said there are abstractions terraform, pulumi, others. But my go to is always that most companies will never get to the point where k8s is required — most companies never get to that scale. A well maintained docker compose setup gets you a long way.
Deutschlandds 2y ago
I find it always interesting and weird to read so dimetrical points to kubernetes. 1. Cloud for me is a lot better than what we had before: Before i had to create a ticket for our internal it department, have huge cross-charges (like 500$ for a server, instead of 50), had to wait for a few weeks and than get lectured that installing basic default tools on that suse based server would take a week and add additional cross-charges onto it. Their reasoning? Oh we do backup and upgrades... With cloud, i click a server in a Minute for less money, i upgrade it myself and have snapshots as a basic backup workflow which actually is reliable and works. Then k8s came along and let me be clear: my k8s setup is big, so its definitly worth it but tx to my experience with a bigger k8s setup, my very small ones are also working very very well. I get, out of the box, HA, network policies, snapshotting through my selected storage setup, Infrastructure as code etc. Instead of having shitty shell scripts, ansible setup and co, i only write a little bit of yaml, check it into my git system and roll it out with ansible. Absolut no brainer. And the auto healing solved real issues: Out of memroy? just restart the pod. Out of disk? Just recreate it. Logging and metrics just works out of the box thanks to the prometheus based monitoring stack. Starting with one server, yeah why not but you know you are not doing it right if you are the only one who can set it up and recover it. But if you don't have anyone with expertise, i would also not just start with k8s. If my startup is a pure application + db thingy, i would go with any app platform out there. But we have real servers because we do stuff with data and need sometimes performance, performance is expensive if you run it in the cloud.
- gizzlon 2y ago
  > shitty shell scripts, ansible setup and co, i only write a little bit of yaml Why are the shell scripts shity but the yaml not? When I look at those yaml files I always throw up just a little :P Also, have you tried Cloud Run?
  - Deutschlandds 2y ago
    Yaml is declarative, you tell k8s what you want and how it looks. For shell scripts, try proper error handling. You start doing some catch hooks, you have issues cehcking error codes of different tools, debugging is hard too. In one infra project we swtiched from shell scripts to golang just to have a lot more control/stability of our scripts.
Havoc 2y ago
“let me install everything myself” doesn’t generalise well and gets messy even if you IaC it. There is a reason k8s gang keeps going on about “cattle not pets”. The starting assumptions and goals are fundamentally different vs “give me a physical server” Both have their place I think so not really one is right other is wrong
- immibis 2y ago
  And people tend to vastly underestimate the power of a single server. These days you can get a terabyte of RAM, 96 cores and dual 10G Ethernet for a low-to-mid 4-digit price (used). Do you need the cloud? Some do, but often your highest conceivable scale fits on one or two servers. Stack Exchange famously ran this way for a long time (until recently when they've been bought by large investors and they're going full cloud and AIshit).
  - Havoc 2y ago
    Scaling wasn’t really what I was getting at there per se. Said generalise but that’s admittedly quite fuzzy. Meant it more in the abstraction layer and standardisation sense. Installing stuff straight on server is very messy especially if it’s lots of different providers with their own dependencies. So you need to do some form of containers or VMs to isolate them. At which point you need some sort of tooling around that. And deal with failures etc. Before you know it you’ve reinvented k8s except with less standardization and more duck tape. So it think there is a strong case for a k8s cluster but being mindful to keep it as simple as possible within that paradigm. Ie k8s but just the basic building blocks
- tombert 2y ago
  Everyone says this, but do we actually have data to back that up? I feel like I spend so much time working around CloudSQL for postgres support in GCP at work, to a point where I'm not actually sure I'm saving a ton of time over running and managing it myself. That's probably not true, I'm sure there are edge cases that I'm not accounting for, but I'm a little tired of everyone acting like AWS and GCP and Azure are the "set it and forget it" thing that they advertise.
deleted 2y ago
[deleted]
zelphirkalt 2y ago
I am sure the consultants and others peddling Kubernetes like it, since they are getting the paycheck for bringing it on, or cementing their position, as they are the ones with the expert knowledge of the system then, if anyone is at all, after something has been migrated to Kubernetes.
bananapub 2y ago
> I have set up about a dozen rack mount servers in my life, yes, k8s and co are silly for trivially tiny systems like this.
randomtoast 2y ago
Kubernetes adds an extra layer of abstraction on top of existing ones. When the stack of abstractions grows too large, it can start to feel sluggish and unwieldy. What if we had something like Kubernetes but at the hardware level? Imagine a single Linux installation running across multiple servers, where resources are seamlessly pooled and managed. In htop, you could visualize all CPU cores, with each core labeled by its corresponding node. Now, consider starting a container with Podman: the container would execute on the CPU cores of one node. If you start another container, it could run on the cores of a different node. This approach would essentially transform Linux into an operating system capable of spanning a distributed cluster of nodes. To achieve this, the operating system wouldn’t need to be entirely reinvented—it could simply be Linux, enhanced with the necessary kernel modifications to enable such distributed functionality. This could provide the simplicity and efficiency of a unified OS while leveraging the power of a distributed system. Or maybe it's a pipe dream.
CursedSilicon 2y ago
Are you hiring? (only somewhat sarcastic) I did a 15~ month stint at AWS. Originally I was signed on to be a support engineer for their Linux teams, which sounded great! Absolutely within my skillset and a great way to get to play with new technologies and features at scale that I haven't before (Ansible, etc) After going through all the hoops I get sidelined into the "Containers" team and have to learn Kubernetes, ECS, Fargate etc all effectively from scratch It's all miserable. All of it. Massive Rube Goldberg machines of complexity for the sake of complexity that you need a team to decipher, let alone maintain. Unfortunately it feels like all the Sysadmin jobs have been replaced either by "DevOps" or Cloud Engineers. There's little market left for people who just want to keep boxes inside a datacenter (or on prem) humming along. The ones that do all want that also seem to be all wedded to MSP's unfortunately
BrandoElFollito 2y ago
Installing a docker container (I am referring to your "cloud" point) and maintaining it is orders of magnitude easier that managing a service on a VM. I do IT stuff since 1994 (pro and perso) and I have seen it all (just an expression, I actually saw 2% of the all :)) I have about 30 services at home, they do not require maintenance. They just update on their own (Home Assistant failed once in the last 10 years), there are no dependency hell and I need to only maintain one OS. I like to code and the idea to hace "service as code" or rather "service as yaml" is great (though I hate yaml)
osigurdson 2y ago
Kubernetes allows me to treat hardware more like software and like it very much for this reason.

tryauuum 2y ago

I have same thoughts. the only form of kubernetes I would be willing to try is the one with kata-containers for having all the security of virual machines.

nyclounge 2y ago

Or if you got a static ip and fast up speed, then just port forward 80, 443 and start hosting your self. Even an old Intel Mac Book pro from 2000's with 4 GB of RAM may not be that hot running MacOS, but install Debian with no X. It is running smooth as a whistle, while running several conduit matrix, haraka, zone-mta, ice cast, nginx with no issues. WebRTC/TUN/STUN becomes an issue with the nginx config. May consider looking at pingora. The whole rust -> binary + toml file is super nice to run from system admin perspective.

d3Xt3r 2y ago
> It is running smooth as a whistle ... until you get hit by a DDoS attack. Not much you can do about it unless your ISP offers protection, or you end up going for Cloudflare or the like instead of exposing your IP and ports.
- PittleyDunkin 2y ago
  Folding the second you get dos'd is aa feature, not a liability. Staying up through those is a great way to lose a ton of money basically instantly.
- nyclounge 2y ago
  Or hetzner has also DDoS protect for all their customers. Seems like a lot less rip off and scam than AWS and Cloudflare. Set your TLS a low number, and you can swap whenever you feel like it.

tapoxi 2y ago

This read as "old man yells at cloud" to me. I've managed a few thousand VMs in the past, and I'm extremely grateful for it. An image is built in CI, service declares what it needs, the scheduler just handles shit. I'm paged significantly less and things are predictable and consistent unlike the world of VMs where even your best attempt at configuration management would result in drift, because the CM system is only enforcing a subset of everything that could go wrong. But yes, Kubernetes is configured in YAML, and YAML kind of sucks, but you rarely do that. The thing that changes is your code, and once you've got the boilerplate down CI does the rest.

catdog 2y ago
YAML is fine, esp. compared to the huge collection of often 10x worse config formats you have to deal with in the VM world.
cess11 2y ago
I'd prefer Ansible if I was running VM:s. Did that at a gig, controlled a vCenter cluster and hundreds of machines in it, much nicer experience than Kubernetes style ops. Easier to do ad hoc troubleshooting and logging for one.
- vrighter 2y ago
  until, as happened to us, you're in the middle of an upgrade cycle, with a mix of red-hat 6 and red-hat 8 servers, and ansible decide to require support for the latest available version of python on red-hat 8, which isn't available on red-hat 6, so we have no way of using ansible to manage both sets of servers. The python ecosystem is a cancer.
  - cess11 2y ago
    Sure, and I'm also not a fan of RedHat. We ran Ubuntu and Debian on that gig, the few Python issues we ran into we could fix with some package pinnings and stuff like that.
  - mkesper 2y ago

voidfunc 2y ago

I'm always kind of blown away by experiences like this. Admittedly, I've been using Kubernetes since the early days and I manage an Infra team that operates a couple thousand self-managed Kubernetes clusters so... expert blindness at work. Before that I did everything from golden images to pushing changes via rsync and kicking a script to deploy. Maybe it's because I adopted early and have grown with the technology it all just makes sense? It's not that complicated if you limit yourself to the core stuff. Maybe I need to write a book like "Kubernetes for Greybeards" or something like that. What does fucking kill me in the Kubernetes ecosystem is the amount of add-on crap that is pitched as "necessary". Sidecars... so many sidecars. Please stop. There's way too much vendor garbage surrounding the ecosystem and dev's rarely stop to think about whether they should deploy something when it's easy as dropping in some YAML and letting the cluster magically run it.

theptrk 2y ago
I would pay for the outline of this book.
t-writescode 2y ago
> Admittedly, I've been using Kubernetes since the early days and I manage an Infra team I think this is where the big difference is. If you're leading a team and introduced all good practices from the start, then the k8s and Terraform or whatever config files can never get so very complicated that a Gordian knot isn't created. Perhaps k8s is nice and easy to use - many of the commands certainly are, in my experience. Developers have, over years and decades, learned how to navigate code and hop from definition to definition, climbing the tree and learning the language they're operating in, and most of the languages follow similar-enough patterns that they can crawl around. Configuring a k8s cluster has absolutely none of that knowledge built up; and, reading something that has rough practices is not a good place to learn what it should look like.
- paddy_m 2y ago
  Thank you. I can always xargs grep for a function name, at worst. Dir() in python at a debugger for other things. With YAML, kubernetes and other devops hotness, I frequently can’t even find the relevant scripts/YAML that are executed nor their codebases. This also happens with configuration based packaging setups. Python hatch in particular, but sometimes node/webpack/rollup/vite.

cheriot 2y ago

> I might just be stupid, but it feels like all I ever do with ____ is update and break ____ files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange. I cannot imagine how anyone goes to work and actually enjoys working in ____, though I guess someone must in terms of “law of large numbers”. I'd make a similar statement about the sys admin stuff you already know well. Give me yaml and a rest api any day. I see where you and the article are coming from, though. The article reasonably points out that k8s is heavy for simpler workloads.

stiray 2y ago

I agree, but what pisses me the most is that today higher level abstractions (like cloud, spring boost,...) are hiding lower level functionality so well, that you are literally forced to use obnoxious amounts of time to study documentation (if you are in luck and it is well written), while everything is decorated with new naming of known concepts that was invented by people who didn't know that the concept already exists and has a name or some marketing guy figured out it will sell better with more "cool" name. Like Shakespeare work would be clumsy and half translated to french advertising jargoon and you are forced to read it and make it work on a stage.

datadeft 2y ago
The real benefits of LLMs is to give me a good summary of these documents. Still, using these abstractions is probably not worth it.
- PittleyDunkin 2y ago
  Well, a summary. It's impossible to evaluate the quality of the summary without actually reading the document you're trying to avoid reading.
- stiray 2y ago
  LLMs are a symptom, not a solution. Just another over-hyped bullshit (those downvoting will agree in few years or never as they are the ones riding the hype) that its only concern is to boost company stocks. Google is the proof, their search started to suck immediately when they added AI into concept. It promotes bullshit, while it doesn't hit the really relevant results, even if you specify them into the details. If AI would have any real value, it would never be given out for free and this is the case: they have a solution and they are searching for a problem it solves.

devjab 2y ago

I don’t mind the cloud, but even in enterprise organisations I fail to see the value of a lot of the more complex tools. I’ve anlways worked with Azure because Denmark is basically Microsoft territory in a lot of non-tech organisations because of the synergy between pricing and IT operations staff. I’ve done bicep, terraform and both Kubernetes and the managed (I forgot what azure conteiner apps running on top of what is basically Kubernetes is called). When I can get away with it I always use the Azure CLI through bash scripts in a pipeline however and build directly into Azure App services for contained which is just so much less complicated than what you probably call “cloud shit”. The cool part about the Azure CLI and their app services is that it hasn’t really changed in the past 3 years, and they are almost one size fit any organisation. So all anyone needs to update in the YAML scripts are the variables. By contrast working with Bicep/Terraform, Jenkins and whatever else people use has been absolutely horrible, sometimes requiring full time staff just to keep it updated. I suppose it may be better now that azure co-pilot can probably auto-generate what you need. A complete waste of resources in my opinion. It used to be more expensive, but with the last price hike of 600% on azure container apps it’s usually cheaper. It’s also way more cost efficient in terms of maintaining since it’ll just work after the initial setup pipeline has run. This is the only way I have found that is easier than what it was when organisations ran their own servers. Whether it was in the basement or at some local hardware house (not exactly sure what you call the places where you rent server rack space). Well places like Digital Ocean are even easier but they aren’t used by enterprise. I’m fairly certain I’ve ever worked with an organisation that needed anything more than that since basically nothing in Denmark scales beyond what can run on a couple of servers behind a load balancer. One of the few exceptions is the tax system which sees almost 0 usage except for the couple of weeks where the entire adult population logs in in at the same time. When DevOps teams push back, I tend to remind them that StackOverflow ran on a couple of IIS servers for a while and that they don’t have even 10% of the users. Eventually the business case for Azure will push people back to renting hardware space or jumping to Hetzner and similar. But that’s a different story.

DanielHB 2y ago
Terraform has the same problem as Kubernetes sidecars with terraform providers trying to do magic for you. If you stick to the cloud platform provider it is actually much nicer than using the CLI. Although my experience is with AWS, I find the terraform AWS provider docs better documentation than the official AWS docs for different options. If they don't answer any question I have right away they at least point me where to look for answers in the mess that is AWS docs.

ozim 2y ago

I am running VPSes at our small startup-ish company on IaaS cloud. Every time we get a new guy I have to explain that we are already „in cloud” there is no need to „move to cloud”.

rcleveng 2y ago
Do they mean PAAS vs IAAS when they say "move to cloud"?
- ozim 2y ago
  Mostly business guys don’t know the difference but we are running on local cloud provider and they think of it is not on Azure or AWS it is not in cloud - they understand that we run stuff on servers but they also don’t understand VPS is IaaS. Developers want to use PaaS and also AWS or Azure so they can put it on their resume for the future.
  - bluehatbrit 2y ago
    > Developers want to use PaaS and also AWS or Azure so they can put it on their resume for the future. I think this is a little disingenuous. Developers want to use them because they already know them. The services composing them are also often well documented by the provider. I say all of that as someone trying to move a company away from aws, and over to our own hardware.
    - PittleyDunkin 2y ago
      I want to use them because I'm not paying for it and managing your own hardware is a pain in the ass I'd rather avoid when I could just code instead. Which is not to say that using your own hardware isn't smart, but it is definitely miserable.
      - regularfry 2y ago
        In my experience you swap managing your own hardware for managing intensely obtuse service configurations, pretty much 1:1. That might be preferable but I see a lot of folks approach it like the tradeoff doesn't exist.

misswaterfairy 2y ago

I hate "cloud shit" as well, though specifically that there's a vendor-specific 'app', or terminology, or both, for everything that we've had standard terms for, for decades. I just want a damn network, a couple of virtual machines, and a database. Why does each <cloud provider> have to create different fancy wrappers over everything, that not even their own sales consultants, and even engineers, understand?(1) What I do like about Docker and Kubernetes is that shifting from one cloud provider to another, or even back to on-premises (I'm waiting for the day our organisation says "<cloud-provider> is too damn expensive; those damn management consultants lied to us!!!!") is a lot easier than re-building <cloud provider>'s bespoke shit in <another cloud provider>'s bespoke shit, or back on-premises with real tech (the right option in my opinion for anyone less than a (truly) global presence). I do like the feel of, and being able to touch bare metal, though the 180-proof-ether-based container stuff is nice for quick, flexible, and (sometimes) dirty. Especially for experimenting when the Directors for War and Finance (the same person) say "We don't have the budget!! We're not buying another server/more RAM/another disk/newer CPUs! Get fucked!". The other thing about Docker specifically I like is I can 'shop' around for boilerplate templates that I can then customise without having to screw around manually building/installing them from scratch. And if I fuck it up, I just delete the container and spin up another one from the image. (1) The answer is 'vendor lock-in', kids. (I apologise, I've had a looooooong day today.......)

yungporko 2y ago
agreed, cloud shit can fuck off. they market themselves as the solution to problems you will never have so that you build your shit on their land and they can charge you rent, and most people eat it up and think they're doing them a favour.

fragmede 2y ago

I'll let you in on the joke. The joke is the demand for 100% availability and instant gratification. we're making services where anything less than 4 nines, which is 5 minutes month, is deemed unacceptable. three nines is 10 minutes a week. two nines is 15 minutes a day. there are some things that are important enough that you can't take a coffee break and wait for, but Kubernetes lets you push four nines of availability, no problem. Kubernetes is solving for that level of availability, but my own body doesn't have anything near that level of availability. demanding that from everything and everyone else is what pushes for Kubernetes level of complexity.

kitd 2y ago
Are you the only one waiting on your app if it goes down?
EVa5I7bHFq9mnYK 2y ago
4 nines is a lie. That's not what I experience as a user. I experience 403, 404, 405, 500, or other forms of down, such as difficult captchas, 2fa not working, geolocation blockages etc several times a day.
spacebanana7 2y ago
Most outages these days are also caused by "devops errors" that're much more likely to happen when working with complex infrastructure stacks. So there's real value in keeping things simple.
karmarepellent 2y ago
Its a matter of evaluating what kind of infrastructure your application needs to run on. There are certainly mission critical systems where even a sliver of downtime causes real damage, like lost revenue. If you come to the conclusion that this application and everything it involves better run on k8s for availability reasons, you should probably focus on that and code your application in a k8s-friendly manner. But there are tons of applications that run on over-engineered cloud environments that may or may not involve k8s and probably cost more to operate than they must. I use some tools every day where a daily 15 min downtime would not affect my or my work in the slightest. I am not saying this would be desirable per se. Its just that a lot of people (myself included) are happy to spend an hour of their work day talking to colleagues and drinking coffee, but a 15 min downtime of some tool is seen as an absolute catastrophe.

cess11 2y ago

Once it gets hard to run the needed services for all the applications in the organisation/team on your development machine it'll start to look more attractive to turn to Docker/Podman and the like, and that's when automatic tests and deploys built on disgusting YAML starts to make more sense. I've been at a place where the two main applications were tightly coupled with several support systems and they all were dependent on one or more of Oracle DB, Postgres, Redis, JBoss, Tibco EMS and quite a bit more. Good luck using your development device to boot and run the test suite without containers. Before that team started putting stuff in containers they used the CI/CD environment to run the full set of tests, so they needed to do a PR, get it accepted, maybe wait for someone else's test run to finish, then watch it run, and if something blew, go back to commit, push to PR, &c. all over again. Quite the nuisance. A full test suite had a run time of about an hour too. When I left we'd pushed it to forty minutes on our dev machines. They didn't use raw Kubernetes though, they had RedHat buy-in and used OpenShift, which is a bit more sane. But it's still a YAML nightmare that cuts you with a thousand inscrutable error messages.

teekert 2y ago

Sorry to have to tell you this, but you’re old. Your neural plasticity has gone down and you feel like you have seen it al before. As a result you cling to the old and never feel like you grasp the new. The only reasonable thing to is to acknowledge and accept this and try not let it get in your way. Our generation has seen many things before, but at the same time the world has completely changed and it’s led to the people growing up in it to be different. You and me didn’t fully grasp CPUs anymore. Some people today don’t grasp all the details of the abstractions below K8s anymore and use it when perhaps something simpler (in architecture , not necessarily in use!) could do it better. And yet, they build wonderous things. Without editing php.ini and messing up 2 services to get one working. Do I think K8s is the end all? Certainly not, I agree it’s sometimes overkill. But I bet you’ll like it’s follow-up tech even less. It is the way of things.

tedk-42 2y ago
> Is K8s the end all? Certainly not, I agree it’s sometimes overkill. But I bet you’ll like it’s follow-up tech even less. It is the way of things. I agree with your analysis. People wanna talk up about how good the old days were plugging cables into racks but it's really laborious and can take days to debug that a faulty network switch is the cause of these weird packet drop issues seen sporadically on hot days. Same as people saying 'oh yeah calculators are too complicated, pen and paper is what kids should be learning'. It's the tide of change
- tombert 2y ago
  I reject that comparison, I'm not really resistant to change, I'm resistant the awful bureaucratic crap that k8s and its ilk force you to use. It's not fun, as far as I can tell no one actually understands any of it (young or old), they just copy and past large blocks of YAML from blogs and documentation and then cross their fingers. I'm not saying that plugging in cables and hoping power supplies don't die is "better" in any kind of objective sense, or even subjective sense really, I'm just saying that I hate this cloud cult that has decided that the only way to do anything is to add layers of annoying bureaucratic shit.
creesch 2y ago

bob1029 2y ago

Cloud is a broad spectrum of tools. I think the Goldilocks zone is something right in between bare-ass VMs and FaaS. If I can restart the machine and it magically resolves the underlying hardware fault (presumably by migrating me to a new host), then I am in a happy place. Most of the other problems can be dealt with using modern tooling. I lean aggressively on things like SQLite and self-contained deployments in .NET to reduce complexity at hosting time. When you can deploy your entire solution with a single zip file you would be crazy to go for something like K8s. One other cloud thing that is useful is S3-style services. Clever use of these APIs can provide incredible horizontal scalability for a single instance solution. SQLite on NVMe is very fast if you are offloading your blobs to another provider.

grishka 2y ago

> If I get to tens of millions of users, maybe I’ll worry about it then. Nope, then you'll set up sharded databases and a bunch of application servers behind a load balancer.

raffraffraff 2y ago

I'm sure you can get some of the handy extras that come with a typical kubernetes deployment without the kubernetes, but overall I'll take kubernetes + cloud. Once you've got the hang of it, it's ok. I have a terraform project that deploys clusters with external-dns, external-secrets, cert-manager, metrics, monitoring stack, scalers and FluxCD. From there, pretty much everything else is done via FluxCD (workloads, dashboards, alerts). And while I detest writing helm charts (and sometimes using them, as they can get "stuck" in several ways) they do allow you to wrap up a lot of the kubernetes components into a single package that accepts more-or-less standardized yaml for stuff like resource limits, annotations (eg for granting and AWS role to a service) etc. And FluxCD .postBuild is extremely handy for defining environment vars to apply to more generic template yaml, so we avoid a sprawl. So much so that I am the one-man-band (Sys|Dev|Sec)Ops for our small company, and that doesn't give me panic attacks. The cloud integration part can be hairy but I have terraform patterns that, once worked out, are cookie cutter. With cloud kubernetes, I can imagine starting from scratch, taking a wrong turn and ending up in hell. But I'm exchanging one problem set for another. Having spent years managing fleets of physical and virtual servers, I'm happier and more productive now. I never need to worry about building systems or automation for doing OS build / patching, config management, application packaging and deployment, secrets management, service discovery, external DNS, load balancering, TSL certs etc. Because while those are just "words" now, back then each one was a huge project involving multiple people fighting over "CentOS Vs Ubuntu", "Puppet Vs Ansible", "RPMs Vs docker containers", "Patching Vs swapping AMIs". If you're using Consul and Vault, good luck - you have to integrate all of that into whatever mess you've built, and you'll likely have to write puppet code and scripts to hook it all up together. I lost a chunk of my life writing 'dockerctl' and a bunch of puppet code that deployed it so it could manage docker containers as systemd services. Then building a vault integration for that. It worked great across multiple data centers, but took considerable effort. And in the end it's a unique snowflake used by exactly one company, hardly documented and likely full of undiscovered bugs and race conditions even after all the hard work. The time it took to onboard new engineers was considerable and it took time away from an existing engineer. And we still had certificates expire in production.

datadeft 2y ago

One thing that might help you in this madness is: https://github.com/dhall-lang/dhall-kubernetes https://github.com/dhall-lang/dhall-kubernetes Type safe, fat finger safe representation of your YAMLs is grossly underrated.

friendzis 2y ago

In my limited experience k8s is treated as some magic instead of what it is: resource virtualization platform. Sort one (or two) step above hardware virtualization platforms (e.g. esxi). The fact that vmware can migrate a running vm to a different hardware node is surely powerful feature. Do you want to pay for that with complexity? If you are one infra team serving on-prem deployments with consistent loads you provision things when new projects are started and things break. However, if infra team serves internal product teams it is nice to give certain guarantees to them and limit blast radius how they can affect other teams. This is where kubernetes sit. It's a deployment platform, where the deployable is container image instead of VM image. Slice off a namespace to an internal team and have their deployment blast radius contained within their namespace. Do you need such flexibility? I'm pretty sure that roughly 99% of all teams do not. A static set of VMs provisioned with some container orchestration system is more than enough for them. Loads and deployments are too static for it to matter. >>> But it allows seamless upgrades! Dude or dudette, if you can't properly drain your nodes and migrate sessions now, kubernetes will not save you.

torginus 2y ago

Kubernetes starts sucking at first sight and is kind of the sign of the ailment of modern times. Let me try to explain: First, you encounter the biggest impedance mismatch between cloud and on prem: Kubernetes works with pods, while AWS works with instances as the unit of useful work, so they must map to each other, right? Wrong, first each instance needs to run a Kubernetes node, which duplicates the management infrastructure hosted by AWS, and reduces the support for granularity, like if I need 4 cores for my workload, I start an instance with 4 cores, right?. Not so with k8s, you have to start up a big node, then schedule pods there. Yay, extra complexity and overhead! And it's like when you need 3 carrots for the meal your cooking, but the store only sells it in packs of 8, you have to pay for the ones you don't need, and then figure out how to utilize the waste. I'm not even going to talk about on-prem kubernetes, as I've never seen anyone IRL use it. Another thing is that the need for kubernetes is manifestation of crappy modern dev culture. If you wrote your server in node, Python and Ruby, you're running a single threaded app in the era of even laptops having 32 core CPUs. And individual instances are slow, so you're even more dependent on scaling. So, to make use of the extra CPU power, you're forced to turn to Docker/k8s and scale your infra that way, whereas if you went with something like Go, or god forbid, something as deeply uncool as ASP.NET, you could just put your 32 core server, and you get fast single threaded perf, and perfect and simple multi-threaded utilization out of the box without any of the headaches. Also I've found stuff like rolling updates to be a gimmick. Also a huge disclaimer, I don't think k8s is a fundamentally sucky or stupid thing, it's just I've never seen it used as a beneficial architectural pattern in practice.

erinaceousjones 2y ago
Being nitpicky about Python specifically, Python not necessarily single-threaded; gunicorn gets you a multiprocess HTTP/WSGI server if you configure it to. asyncio and gevent have made it easier to do things actually-concurrently. Judicious use of generator functions lets you stream results back instead of blocking I/O for big chunks. And we have threads. Yeah, the Global Interpreter Lock is still hanging around, it's not the fastest language, but there are ways to produce a Docker image of an API written in Python which can handle thousands of concurrent HTTP requests and actively use all available CPU cores to do intensive computation.

jmb99 2y ago

I have pretty much the exact same opinion, right down to “cloud shit” (I have even used that exact term at work to multiple levels of management, specifically “I’m not working on cloud shit, you know what I like working on and I’ll do pretty much anything else, but no cloud shit”). I have worked on too many projects where all you need is a 4-8 vCPU VM running nginx and some external database, but for some reasons there’s like 30 containers and 45 different repos and 10k lines of terraform, and the fastest requests take 750ms which should be nginx->SQL->return in <10ms. “But it’s autoscaling and load balanced!” That’s great, we have max 10k customers online at once and the most complex requests are “set this field to true” and “get me an indexed list.” This could be hosted on a raspberry pi with better performance. But for some reason this is what people want to do. They would rather spend hours debugging kubernetes, terraform, and docker, and spending 5 digits on cloud every month, to serve what could literally be proxied authenticated DB lookups. We have “hack days” a few times a year, and I’m genuinely debating rewriting the entire “cloud” portion of our current product in gunicorn or something, host it on a $50/month vps, point it at a mirror of our prod db, and see how many orders of magnitude of performance I can knock off in a day. I’ve only managed to convert one “cloud person” to my viewpoint but it was quite entertaining. I was demoing a side project[0] that involved pulling data from ~6 different sources (none hosted by me), concatenating them, deduping them, doing some math, looking up in a different source an image (unique to each displayed item), and then displaying the list of final items with images in a list or a grid. ~5k items. Load time on my fibre connection was 200-250ms, sorting/filtering was <100ms. As I was demoing this, a few people asked about the architecture, and one didn’t believe that it was a 750 line python file (using multiprocessing, admittedly) hosted on an 8 core VPS until I literally showed up. He didn’t believe it was possible to have this kind of performance in a “legacy monolithic” (his words) application. I think it’s so heavily ingrained in most cloud/web developers that this is the only option that they will not even entertain the thought that it can be done another way. [0] This particular project failed for other reasons, and is no longer live.

jiggawatts 2y ago
Speaking of Kubernetes performance: I had a need for fast scale-out for a bulk testing exercise. The gist of it was that I had to run Selenium tests with six different browsers against something like 13,000 sites in a hurry for a state government. I tried Kubernetes, because there's a distributed Selenium runner for it that can spin up different browsers in individual pods, even running Windows and Linux at the same time! Very cool. Except... Performance was woeful. It took forever to spin up the pods, but even once things had warmed up everything just ran in slow motion. Data was slow to collect (single-digit kilobits!), and I even saw a few timeout failures within the cluster. I gave up and simply provisioned a 120 vCPU / 600 GB memory cloud server with spot pricing for $2/hour and ran everything locally with scripts. I ended up scanning a decent chunk of my country's internet in 15 minutes. I was genuinely worried that I'd get put on some sort of "list" for "attacking" government sites. I even randomized the read order to avoid hammering any one site too hard. Kubernetes sounds "fun to tinker with", but it's actually a productivity vampire that sucks up engineer time. It's the Factorio of cloud hosting.

mrweasel 2y ago

I wouldn't throw "cloud shit" in the same bucket as Kubernetes. Professional cloud services are mostly great, but super expensive. Kubernetes is interesting, because it basically takes everything you know and sort of pushes it down the stack (or up, depending on your viewpoint). To some extend I get the impression that the idea was: Wouldn't it be great if we took all the infrastructure stuff, and just merged the whole thing into one tool, which you can configure using a completely unsuitable markup language. The answer is that "No, that would in fact not be great". For me the issue is that Kubernetes is overused. You can not really argue that it's not useful or has its place, but that place is much much small than the Internet wants us to believe. It's one of the few services where I feel like "Call us" would be an appropriate sales method. The article is correct, you probably don't need Kubernetes. It's a amazing piece of technology, but it's not for the majority of us. It is and should be viewed as a niche product.

nixpulvis 2y ago
What do you think its niche is?
- mrweasel 2y ago
  Very large "single product" services, the focus being on "very large". It would also be relevant in the cases where your product is made up of a large number of micro-services, though if you have more than 50 services to support your main product I'd be interested in knowing why you have that many services. There might be completely sane reasons, but it is a special case. Mostly a question of scale to me, I'd guess that the majority (80-90%) of people running Kubernetes don't have large enough scale that it makes sense to take on the extra complexity. Most Kubernetes installations I've seen runs on VMs, three for control plane and 2 - 5 for worker node and I don't think the extra layer is a good trade off for a "better" deployment tool. If you do use Kubernetes as a deployment tool, then I can certainly understand that. It is a reasonably well-known, and somewhat standardised interface and there's not a lot of good alternatives for VMs and bare metal. Personally I'd just much rather see better deployment tools being developed, rather than just picking Kubernetes because Helm charts are a thing. You'd also need to have a rather dynamic workload, in the sense that some of your services is need a lot of capacity at one point in time, while other need the capacity at another time. If you have constant load, then why? It's like Oracle's Exadata servers, it's a product that has its place, but the list of potential buyers isn't that long.

kortilla 2y ago

All of these anecdotes seem to come from people who don’t bother to try to learn kubernetes. > YAML files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange. This is terrible behavior. Its not any different from yanking out pam modules because you’re getting SSH auth failures caused by a bad permission on an SSH key. > If I get to tens of millions of users, maybe I’ll worry about it then. K8s isn’t there for 10s of millions of users. It’s there so you’re not dependent on some bespoke VM state. It also allows you to do code review on infra changes like port numbers being exposed, etc. Separately, your VM likely isn’t coming from any standard build pipeline so now a vulnerability patch is a login to the machine and an update, which hopefully leaves it in the same state as VMs created new… Oh, and assuming you don’t want to take downtime on every update, you’ll want a few replicas and load balancing across them (or active/passive HA at a minimum). Good luck representing that as reviewable code as well if you are running VMs. The people that don’t understand the value prop of infra as code orchestration systems like k8s tend to work in environments where “maintenance downtime” is acceptable and there are only one or two people that actually adjust the configurations.

secondcoming 2y ago
Just because you're using VMs doesn't mean you're now dealing with state. It's 100% possible to have stateless VMs running in an auto-scaling instance group (in GCP speak, I forget what AWS calls them)
- everfrustrated 2y ago
  In the beginning AWS didn't even support state on their VMs! All VMs were ephemeral with no state persistence when terminated. They later introduced EBS to allow for the more classic enterprise IT use cases.
- kortilla 2y ago
  Once you have the tools to manage all of that, you effectively have kubernetes. Container vs VM is largely irrelevant to what the op is complaining about when it comes to k8s. People that don’t like k8s tend to be fine with docker. It’s usually that they don’t like declarative state or thinking in selectors and other abstractions.

PittleyDunkin 2y ago

Isn't using VMs or servers just cloud shit in a different form? How does that fix anything? Orchestrating software (or hardware) is a pain in the ass whether it's in your closet or someone else's.

andyjohnson0 2y ago

Kubernetes was developed by Google, and it seems to me it's a classic example of a technology that can work well in a resource-rich (people, money) environment. But in an averge business it becomes a time/money sink quite quickly. Way too many businesses cargo-cult themselves into thinking that they need FAANG-class infra, even though they haven't got the same scale or the same level of resourcing. Devs and ops people love it because they get to cosplay and also get the right words on their CV. If you're not Google-scale then, as you say, a few VMs or physical boxes are all you need for most systems. But its not sexy, so the business needs people who don't mind that.

xorcist 2y ago
K8s is absolutely not Google-scale. Not even close. The scheduler is not built for it.

Woeps 2y ago

I don't mind K8s, and it can be usefull if you just stay to the core essentials. But I do agree that there seems to be the sentiment:"Everybody has to like it!" And I mostly think that this is because our collective bias for everything Big Tech does is good. While often it just depends. Just because Google does X or Y doesn't mean it will work for everybody else.

wordofx 2y ago

I’m confused. Do you hate cloud or the container shit? Cloud is awesome. Containers is dumb as hell for 99% of the applications hacker news kids work on.

tombert 2y ago
I don't know what "cloud" means. I don't really like Kubernetes, and I don't really like the AWS and GCP systems for doing everything. The fact that WGU offers a bachelors degree in Cloud Computing with focuses on AWS or Azure should be telling. [1] Containers don't bother me that much on their own. I just feel like with k8s and its ilk I end up spending so much time futzing with weird YAML and trying to remember differences between stateful sets and services and fighting with stuff because I missed the thing that mounts a disk so my app will break in three hours. [1] https://www.wgu.edu/online-it-degrees/cloud-computing-bachelors-program.html https://www.wgu.edu/online-it-degrees/cloud-computing-bachel...

chromanoid 2y ago

IMO the actual experience you want is something as simple as you can do with those old-school simple PHP-based "serverless" web hosters. You just upload your stuff and maybe configure a database URL and that's it. Regarding AWS, function.zip Lambda + DynamoDB + S3 + SQS is basically this at "enterprise-grade". Now you have the in-between left, where you want enterprise-grade (availability, scaling and fault tolerance) but with a catch (like lower costs, more control, data sovereignty or things that are not as serverless as you want, e.g. search engine etc.). In these cases you will run into many trade-off decisions, that may lead you to K8s, cloud specific stacks and also simple VM ClickOps / ShellOps setups. As long as you can still be on the pet instead of cattle range of problems, K8s is not what you want. But as soon as you want cattle, reproducible throw-away online environments etc. "just have a VM or a physical server" will become a "build your own private cloud solution" that may or may not become a bad private cloud / K8s.

antihero 2y ago

I think a good compromise is building your applications in a containerised manner so that you can simply run them with docker-compose, but then if it turns out you need some heavy scale, it's merely a case of switching the underlying infrastructure. That said, my experience has been fairly different. Running microk8s on a VPS with some containers and an ingress just seems like a nice way to spin them up as pods and manage them. It really doesn't feel that complicated. Once you integrate with cloud providers and you get more complex systems, sure, it gets more complex. I much prefer the container paradigm for separating parts of apps out and managing dependencies and reproducibility. Having a bunch of raw servers with all the bullshit and drift and mutability that comes along with it feels far more like a headache to me than the abstraction of k8s or docker. If you aren't deploying stuff with GitOps/IaC in 2024 I fear for anyone that has to manage or maintain it. 100% reproducibility and declarative infrastructure are wonderful things I could never come back from.

pdimitar 2y ago
I love the idea of declarative infrastructure but I'm never considering a bunch of YAML files as a good practice. Stuff like k9s / microk8s / k3s are clutches and workarounds and I hope we all see it. If they figure to use an actual programming language or just start using much smaller amount of files than they currently do then I'd be the first to learn k8s. Before that, nope. I love the idea but the implementation makes me want to slit my wrists.

gigatexal 2y ago

The yaml and the k8s model can be grokked it just takes a willingness to. That being said there are abstractions terraform, pulumi, others. But my go to is always that most companies will never get to the point where k8s is required — most companies never get to that scale. A well maintained docker compose setup gets you a long way.

Deutschlandds 2y ago

I find it always interesting and weird to read so dimetrical points to kubernetes. 1. Cloud for me is a lot better than what we had before: Before i had to create a ticket for our internal it department, have huge cross-charges (like 500$ for a server, instead of 50), had to wait for a few weeks and than get lectured that installing basic default tools on that suse based server would take a week and add additional cross-charges onto it. Their reasoning? Oh we do backup and upgrades... With cloud, i click a server in a Minute for less money, i upgrade it myself and have snapshots as a basic backup workflow which actually is reliable and works. Then k8s came along and let me be clear: my k8s setup is big, so its definitly worth it but tx to my experience with a bigger k8s setup, my very small ones are also working very very well. I get, out of the box, HA, network policies, snapshotting through my selected storage setup, Infrastructure as code etc. Instead of having shitty shell scripts, ansible setup and co, i only write a little bit of yaml, check it into my git system and roll it out with ansible. Absolut no brainer. And the auto healing solved real issues: Out of memroy? just restart the pod. Out of disk? Just recreate it. Logging and metrics just works out of the box thanks to the prometheus based monitoring stack. Starting with one server, yeah why not but you know you are not doing it right if you are the only one who can set it up and recover it. But if you don't have anyone with expertise, i would also not just start with k8s. If my startup is a pure application + db thingy, i would go with any app platform out there. But we have real servers because we do stuff with data and need sometimes performance, performance is expensive if you run it in the cloud.

gizzlon 2y ago
> shitty shell scripts, ansible setup and co, i only write a little bit of yaml Why are the shell scripts shity but the yaml not? When I look at those yaml files I always throw up just a little :P Also, have you tried Cloud Run?
- Deutschlandds 2y ago
  Yaml is declarative, you tell k8s what you want and how it looks. For shell scripts, try proper error handling. You start doing some catch hooks, you have issues cehcking error codes of different tools, debugging is hard too. In one infra project we swtiched from shell scripts to golang just to have a lot more control/stability of our scripts.

Havoc 2y ago

“let me install everything myself” doesn’t generalise well and gets messy even if you IaC it. There is a reason k8s gang keeps going on about “cattle not pets”. The starting assumptions and goals are fundamentally different vs “give me a physical server” Both have their place I think so not really one is right other is wrong

immibis 2y ago
And people tend to vastly underestimate the power of a single server. These days you can get a terabyte of RAM, 96 cores and dual 10G Ethernet for a low-to-mid 4-digit price (used). Do you need the cloud? Some do, but often your highest conceivable scale fits on one or two servers. Stack Exchange famously ran this way for a long time (until recently when they've been bought by large investors and they're going full cloud and AIshit).
- Havoc 2y ago
  Scaling wasn’t really what I was getting at there per se. Said generalise but that’s admittedly quite fuzzy. Meant it more in the abstraction layer and standardisation sense. Installing stuff straight on server is very messy especially if it’s lots of different providers with their own dependencies. So you need to do some form of containers or VMs to isolate them. At which point you need some sort of tooling around that. And deal with failures etc. Before you know it you’ve reinvented k8s except with less standardization and more duck tape. So it think there is a strong case for a k8s cluster but being mindful to keep it as simple as possible within that paradigm. Ie k8s but just the basic building blocks
tombert 2y ago
Everyone says this, but do we actually have data to back that up? I feel like I spend so much time working around CloudSQL for postgres support in GCP at work, to a point where I'm not actually sure I'm saving a ton of time over running and managing it myself. That's probably not true, I'm sure there are edge cases that I'm not accounting for, but I'm a little tired of everyone acting like AWS and GCP and Azure are the "set it and forget it" thing that they advertise.

deleted 2y ago

[deleted]

zelphirkalt 2y ago

I am sure the consultants and others peddling Kubernetes like it, since they are getting the paycheck for bringing it on, or cementing their position, as they are the ones with the expert knowledge of the system then, if anyone is at all, after something has been migrated to Kubernetes.

bananapub 2y ago

> I have set up about a dozen rack mount servers in my life, yes, k8s and co are silly for trivially tiny systems like this.

randomtoast 2y ago

Kubernetes adds an extra layer of abstraction on top of existing ones. When the stack of abstractions grows too large, it can start to feel sluggish and unwieldy. What if we had something like Kubernetes but at the hardware level? Imagine a single Linux installation running across multiple servers, where resources are seamlessly pooled and managed. In htop, you could visualize all CPU cores, with each core labeled by its corresponding node. Now, consider starting a container with Podman: the container would execute on the CPU cores of one node. If you start another container, it could run on the cores of a different node. This approach would essentially transform Linux into an operating system capable of spanning a distributed cluster of nodes. To achieve this, the operating system wouldn’t need to be entirely reinvented—it could simply be Linux, enhanced with the necessary kernel modifications to enable such distributed functionality. This could provide the simplicity and efficiency of a unified OS while leveraging the power of a distributed system. Or maybe it's a pipe dream.

CursedSilicon 2y ago

Are you hiring? (only somewhat sarcastic) I did a 15~ month stint at AWS. Originally I was signed on to be a support engineer for their Linux teams, which sounded great! Absolutely within my skillset and a great way to get to play with new technologies and features at scale that I haven't before (Ansible, etc) After going through all the hoops I get sidelined into the "Containers" team and have to learn Kubernetes, ECS, Fargate etc all effectively from scratch It's all miserable. All of it. Massive Rube Goldberg machines of complexity for the sake of complexity that you need a team to decipher, let alone maintain. Unfortunately it feels like all the Sysadmin jobs have been replaced either by "DevOps" or Cloud Engineers. There's little market left for people who just want to keep boxes inside a datacenter (or on prem) humming along. The ones that do all want that also seem to be all wedded to MSP's unfortunately

BrandoElFollito 2y ago

Installing a docker container (I am referring to your "cloud" point) and maintaining it is orders of magnitude easier that managing a service on a VM. I do IT stuff since 1994 (pro and perso) and I have seen it all (just an expression, I actually saw 2% of the all :)) I have about 30 services at home, they do not require maintenance. They just update on their own (Home Assistant failed once in the last 10 years), there are no dependency hell and I need to only maintain one OS. I like to code and the idea to hace "service as code" or rather "service as yaml" is great (though I hate yaml)

osigurdson 2y ago

Kubernetes allows me to treat hardware more like software and like it very much for this reason.