13 ms·

I Didn't Need Kubernetes, and You Probably Don't Either

• OtomotO 2y ago
  Right. Depending on my client's needs we do it oldschool and just rent a beefy server. Using your brain to actually assess a situation and decide without any emotional or monetary attachment to a specific solution actually works like a charm most of the time. I also had customers who run their own cloud based on k8s. And I heard some people have customers that are on a public cloud ;-) Choose the right solution for the problem at hand.
• lkrubner 2y ago
  Interesting that the mania for over-investment in devops is beginning to abate. Here on Hacker News I was a steady critic of both Docker and Kubernetes, going to at least 2017, but most of these posts were unpopular. I have to go back to 2019 to find one that sparked a conversation: https://news.ycombinator.com/item?id=20371961 https://news.ycombinator.com/item?id=20371961 The stuff I posted about Kubernetes did not draw a conversation, but I was simply documenting what I was seeing: vast over-investment in devops even at tiny startups that were just getting going and could have easily dumped everything on a single server, exactly as we used to do things back in 2005.
  • pclmulqdq 2y ago
    The attraction of this stuff is mostly the ability to keep your infrastructure configurations as code. However, I have previously checked in my systemd cofig files for projects and set up a script to pull them on new systems. It's not clear that docker-compose or even kubernetes* is that much more complicated if you are only running 3 things. * if you are an experienced user
    • honkycat 2y ago
      Having done both: running a small Kubernetes cluster is simpler than managing a bunch of systemd files.
      • worldsayshi 2y ago
        Yeah this is my impression as well which makes me not understand the k8s hate.
        • pclmulqdq 2y ago
          The complexity of k8s comes the moment you need to hold state of some kind. Now instead of one systemd entry, we have to worry about persistent volume claims and other such nonsense. When you are doing things that are completely stateless, it's simpler than systemd.
          • p_l 2y ago
            If you need to care about state with systemd you still have the "nonsense" of persistent volume claims, they are just something you keep in notes somewhere, in my experience usually in heads of the sysadmins or an excel sheet or a text file that tries to track which server has what data connected how.
            • pclmulqdq 2y ago
              Understand that in the hypothetical system we are discussing, there are something like 1-2 servers. In that case the "volume claim" is just "it's a file on the obvious filesystem" and does not actually need to be spelled out they way you need to spell it out in k8s. The file path you give in environment variables is where the most up-to-date version of the volume claim is. And that file is free to expand to hundreds of GB without bothering you.
              • p_l 2y ago
                Things get iffier when you start doing things like running multiple instances of something (maybe you're sticking two test environments for your developers), or suddenly you grew a bit or no longer fit on the server and start migrating around. The complexity of PVCs in my experience isn't really that big compared to this, possibly lower, and I did stuff both ways.
  • OtomotO 2y ago
    It's just the hype moving on. Every generation has to make similar mistakes again and again. I am sure if we had the opportunity and the hype was there we would've used k8s in 2005 as well. The same thing is true for e.g. JavaScript on the frontend. I am currently migrating a project from React to HTMX. Suddenly there is no build step anymore. Some people were like: "That's possible?" Yes, yes it is and it turns out for that project it increases stability and makes everything less complex while adding the exact same business value. Does that mean that React is always the wrong choice? Well, yes, React sucks, but solutions like React? No! It depends on what you need, on the project! Just as a carpenter doesn't use a hammer to saw, we as a profession should strive to use the right tool for the right job. (Albeit it's less clear than for the carpenter, granted)
    • augbog 2y ago
      It's actually kinda hilarious how RSC (React Server Components) is pretty much going back to what PHP was but yeah proves your point as hype moves on people begin to realize why certain things were good vs not
    • ajayvk 2y ago
      Along those lines, I am building https://github.com/claceio/clace https://github.com/claceio/clace for teams to deploy internal tools. It provides a Cloud Run type interface to run containers, including scaling down to zero. It implements an application server than runs containerized apps. Since HTMX was mentioned, Clace also makes it easy to build Hypermedia driven apps.
      • MortyWaves 2y ago
        Would you be open to non Python support as well? This tool seems useful, very useful in fact, but I mainly use .NET (which yes can run very well in containers).
        • ajayvk 2y ago
          Starlark (python like config language) is used to configure Clace. For containerized apps, python frameworks are supported without a Dockerfile being required. All other languages currently require a user provided Dockerfile, the `container` spec can be used. I do plan to add specs for other languages. New specs have to be added here https://github.com/claceio/appspecs https://github.com/claceio/appspecs. New specs can be created locally also in the config, see https://clace.io/docs/develop/#building-apps-from-spec https://clace.io/docs/develop/#building-apps-from-spec
  • valenterry 2y ago
    So, let's say you want to deploy server instances. Let's keep it simple and say you want to have 2 instances running. You want to have zero-downtime-deployment. And you want to have these 2 instances be able to access configuration (that contains secrets). You want load balancing, with the option to integrate an external load balancer. And, last, you want to be able to run this setup both locally and also on at least 2 cloud providers. (EDIT: I meant to be able to run it on 2 cloud providers. Meaning, one at a time, not both at the same time. The idea is that it's easy to migrate if necessary) This is certainly a small subset of what kubernetes offers, but I'm curious, what would be your goto-solution for those requirements?
    • caseyohara 2y ago
      I think you are proving the point; there are very, very few applications that need to run on two cloud providers. If you do, sure, use Kubernetes if that makes your job easier. For the other 99% of applications, it’s overkill. Apart from that requirement, all of this is very doable with EC2 instances behind an ALB, each running nginx as a reverse proxy to an application server with hot restarting (e.g. Puma) launched with a systemd unit.
      • osigurdson 2y ago
        To me that sounds harder than just using EKS. Also, other people are more likely to understand how it works, can run it in other environments (e.g. locally), etc.
      • valenterry 2y ago
        Sorry, that was a misunderstanding. I meant that I want to be able to run it on two cloud providers, but one at a time is fine. It just means that it would be easy to migrate/switch over if necessary.
      • globular-toast 2y ago
        Hmm, let's see, so you've got to know: EC2, ALB, Nginx, Puma, Systemd, then presumably something like Terraform and Ansible to deploy those configs, or write a custom set of bash scripts. And all of that and you're tied to one cloud provider. Or, instead of reinventing the same wheels for Nth time, I could just use a set of abstractions that work for 99% of network services out there, on any cloud or bare metal. That set of abstractions is k8s.
  • honkycat 2y ago
    Start-ups that don't need to scale will quickly go away, because how else are you going to make a profit? How have you been going since 2005 and still not understand the economics of software?
    • ndriscoll 2y ago
      CPUs are ~300x more powerful and storage offers ~10,000x more IOPS than 2005 hardware. More efficient server code exists today. You can scale very far on one server. If you were bootstrapping a startup, you could probably plan to use a pair of gaming PCs until at least the first 1-10M users.
      • shakiXBT 2y ago
        10 million users on a pair of gaming PCs is ridiculous. What's your product, a website that tells the current time?
        • ndriscoll 2y ago
          How many requests do you expect users actually do? Especially if you're serving a B2B market; not everything is centered around addiction/"engagement". My 8 year old PC can do over 10k page requests/second for a reddit or myspace clone (without getting into caching). A modern high end gaming PC should be around 10x more capable (in terms of both CPU and storage IOPS). The limit in terms of needing to upgrade to "unusual" hardware for a PC would likely be the NIC. Networking is one place where typical consumer gear is stuck in 2005. Webapps might make it hard to tell, but a modern computer (or even an old computer like mine) is mindbogglingly fast.
    • arlcode 2y ago
      Just to make it clear: There are a million use cases that don't involve scaling fast. For example B2B businesses where you have very few but extremely high value customers for specialized use cases. Another one is building bully hardware. Your software infrastructure does not need to grow any faster than your shop floor is building it. Whether you want to call that a "startup" is up for debate (and mostly semanticist if you ask me) but at one point they were all a zero employee company and needed to survive their first 5 years. In general you won't find their products on the app store.
  • sobellian 2y ago
    I've worked at a few tiny startups, and I've both manually administered a single server and run small k8s clusters. k8s is way easier. I think I've spent 1, maybe 2 hours on devops this year. It's not a full-time job, it's not a part-time job, it's not even an unpaid internship. Perhaps at a bigger company with more resources and odd requirements...
    • nicce 2y ago
      But how much this costs extra? Sounds like you are using cloud-provided k8s.
      • sobellian 2y ago
        EKS is priced at $876 / yr / cluster at current rates. Negligible for me personally, it's much less than either our EC2 or RDS costs.
        • fer 2y ago
          Yeah, using EKS isn't the same thing as "administering k8s", unless I misread you above. Actual administration is already done for you, it's batteries included, turn-key, and integrated with everything AWS. A job ago we had our own k8s cluster in our own DC, and it required a couple of teams to keep running and reasonably integrated with everything else in the rest of the company. It was probably cheaper overall than cloud given the compute capacity we had, but also probably not by much given the amount of people dedicated to it. Even my 3-node k3s at home requires more attention than what you described.
          • sobellian 2y ago
            You did misread me, I never said I administered k8s. The quoted phrase does not exist :)
      • p_l 2y ago
        I currently use k8s to control bunch of servers. The amount of work/cost of using k8s for handling them in comparison to doing it "old style" is probably negative by now.
  • santoshalper 2y ago
    As an industry, we spent so much time sharpening our saw that we nearly forgot to cut down the tree.
  • rozap 2y ago
    ZIRP is over.
  • harrall 2y ago
    People gravely miss-understand containerization and Docker. All it lets you do is put shell commands into a text file and be able to run it self-contained anywhere. What is there to hate? You still use the same local filesystem, the same host networking, still rsync your data dir, still use the same external MySQL server even if you want -- nothing has changed. You do NOT need a load balancer, a control plane, networked storage, Kubernetes or any of that. You ADD ON those things when you want them like you add on optional heated seats to your car.
    • skydhash 2y ago
      Why would you want to run it anywhere. People mostly select an OS and just update that. It may be great when distributing applications for others to host, but not when it’s the only strategy. I have to reverse engineer dockerfiles when the developer wouldn’t provide a proper documentation.
      • ftmch 2y ago
        OS upgrades are a pain. Even just package updates could break everything. Having everything in containers makes migrating to another system much easier.
  • sunshine-o 2y ago
    Kubernetes, as an industry standard that a lot of people complain about is just a sitting duck waiting to be disrupted. Anybody who doesn't have the money, time or engineering resources will jump on whatever appear as a decent alternative. My intuition is that alternative already exist but I can't see it... A bit like Spring emerged as an alternative to J2EE or what HTMX is to React & co. Is it k3s or something more radical? Is it on a chinese Github?
    • ftmch 2y ago
      I wish Docker Swarm would get more attention. It could be the perfect Kubernetes lightweight alternative. Instead it seems like it could get deprecated any day now.
• dhorthy 2y ago
  i mean control loops are good but you don't need hundreds of them
• gavindean90 2y ago
  K8s always seems like the tool that people choose to avoid cloud vendor lock in but there is something to be said for C k8s lock in as well as the article points out. If you end up with exotic networking or file system mounts you can just be stuck maintaining k8s forever and some updates aren’t so stable so you have to be more vigilant that windows updates.
  • osigurdson 2y ago
    I don't think it makes sense to conflate vendor lock-in with taking a dependency on a given technology. Do we then have "Linux lock-in" and "Postgres lock-in"? The term "lock-in" shouldn't be stretched to cover this concept imo.
    • gavindean90 2y ago
      It’s not quite the same thing as a commercial vendor lock in but it’s close. You can have Postgres lock in as much as Wordpress has MySql lock in. I agree that you have less Linux lock in but Docker still requires a Linux kernel everywhere it goes. BSD need not apply.
• osigurdson 2y ago
  The "You Probably Don't Either" is a little presumptuous. Many projects probably don't need cloud run either. Certainly, many projects shouldn't even be started in the first place.
  • stitched2gethr 2y ago
    I work with Kubernetes enough that I would answer to the title "kubernetes developer" and I would recommend you don't use kubernetes. In the same way I would recommend you don't use a car when you can walk. Your friend lives 1/8 miles away. You go to see them every day so why wouldn't you drive? Well, cars are expensive and you should avoid them if you don't need them. There are a TON of downsides to driving a car 1/4 of a mile every day. And there are a TON of benefits to using a car to drive 25 miles every day. I hate to quash a good debate but this all falls under the predictable but often forgotten "it depends". Usually do you need kubernetes == do you have a lot of shit to run.
• deleted 2y ago
  [deleted]
• semitones 2y ago
  Kubernetes has a steep learning curve, and certainly a lot of complexity, but when used appropriately for the right case, by god it's glorious
  • threeseed 2y ago
    Kubernetes has a proportional learning curve. If you're used to managing platforms e.g. networking, load balancers, security etc. then it's intuitive and easy. If you're used to everything being managed for you then it will feel steep.
    • alienchow 2y ago
      That's pretty much it. I think the main issue nowadays is that companies think full stack engineering means OG(FE BE DB) + CICD + Infra + security compliance + SRE. If a team of 5-10 SWEs have to do all of that while only graded on feature releases, k8s would massively suck. I also agree that experienced platform/infra engineers tend to whine less about k8s.
    • ikiris 2y ago
      Nah the difference between managing k8 and the system it was based on is VASTLY different. K8 is much harder than it needs to be because there wasn't tooling for a long time to manage it well. Going from google internal to K8 is incredibly painful.
    • t-writescode 2y ago
      I think this is only true if the original k8s cluster you're operating against was written by an expert and laid out as such. If you're entering into k8s land with someone else's very complicated mess across hundreds of files, you're going to be in for a bad time. A big problem, I feel, is that if you don't have an expert design the k8s system from the start, it's just going to be a horrible time; and, many people, when they're asked to set up a k8s setup for their startup or whatever, aren't already experts, so the thing that's produced is not maintainable. And then everyone is cursed.
      • 2y ago
• sneak 2y ago
  Except that all of this subjects you and all of your workloads to warrantless US government surveillance due to running in the big public clouds. I personally don’t want the federal government being able to peek into my files and data at any time, even though I’ve done nothing wrong. It’s the innocent people who have most to lose from government intrusion. It seems insane to me to just throw up one’s hands and store private data in Google or Amazon clouds, especially when not doing so is so much cheaper.
  • shlomo_z 2y ago
    I've never heard of this. Can you share a link of what you are referring to? I care about this.
    • mitjam 2y ago
      Here you are: https://www.brennancenter.org/our-work/analysis-opinion/senate-approves-massive-expansion-government-surveillance-power-brennan https://www.brennancenter.org/our-work/analysis-opinion/sena...
      • shlomo_z 2y ago
        Wow. I guess self hosting is a good idea.
• ChrisArchitect 2y ago
  Related: Dear friend, you have built a Kubernetes https://news.ycombinator.com/item?id=42226005 https://news.ycombinator.com/item?id=42226005
  • doctorpangloss 2y ago
    People don’t want solutions, they want holistic experiences. Kubernetes feels bad and a pile of scripts feels good. Proudly declaring how much you buy into feelings feels even better!
    • Timber-6539 2y ago
      Those pile of scripts are the solutions on which Kubernetes was built on. And the fact that they never break is why greybeards still use them over the complexity of shiny new things.
      • p_l 2y ago
        The script break so hard is why one of my first times i felt a "grey beard moment" (with actual grey beard even) was telling a junior the "running ansible from their laptop is not acceptable solution"
• hinkley 2y ago
  I’m having to learn kubernetes to improve my job hunt. It’s really too much for most projects. Cosplaying doesn’t make you money.
• honkycat 2y ago
  are we really still doing this lol? >> Kubernetes is feature-rich, yet these “enterprise” capabilities turned even simple tasks into protracted processes. I don't agree. After learning the basics I would never go back. It doesn't turn simple tasks into a long process. It massively simplifies a ton functionality. And you really only need to learn 4 or 5 new concepts to get it off the ground. If you have a simple website you don't need Kubernetes, but 99% of devs are working in medium sized shops where they have multiple teams working across multiple functionalities and Kubernetes helps this out. Karpenter is not hard to set up at all. It solves the problem about over-provisioning out of the box and has for almost 5 years. It's like writing an article: "I didn't need redis, and you probably don't either" and then talking about how Redis isn't good for relational data.
  • osigurdson 2y ago
    The "...You probably don't either" part is where the argument loses all of its weight. How do they know what I or anyone else needs?
• threeseed 2y ago
  Kubernetes lock-in: bad. Google CloudRun, Database, PubSub, Cloud Storage, VPC, IAM, Artifact Registry etc lock-in: good.
  • deleted 2y ago
    [deleted]
• devops99 2y ago
  These anti-Kubernetes articles are a major signal that the competency crisis is very real.
• deleted 2y ago
  [deleted]
• gatnoodle 2y ago
  "In practice, few companies switch providers unless politics are involved, as the differences between major cloud services are minimal." This is not always true.
• ants_everywhere 2y ago
  People talk about Kubernetes as container orchestration, but I think that's kind of backwards. Kubernetes is a tool for creating computer clusters. Hence the name "Borg" (Kubernetes's grandpa) referring to assimilating heterogeneous hardware into a collective entity. Containers are an implementation detail. Do you need a computer cluster? If so k8s is pretty great. If you don't care about redundancy and can get all the compute you need out of a single machine, then you may not need a cluster. Once you're using containers on a bunch of VMs in different geographical regions, then you effectively have hacked together a virtual cluster. You can get by without k8s. You just have to write a lot of glue code to manage VMs, networking, load balancing, etc on the cloud provider you use. The overhead of that is probably larger than just learning Kubernetes in the long run, but it's reasonable to take on that technical debt if you're just trying to move fast and aren't concerned about the long run.
  • politelemon 2y ago
    I like to describe it similarly, but as a way of building platforms.
  • Spivak 2y ago
    This has got to be the most out there k8s take I've read in a while. k8s doesn't save you from learning your cloud providers infrastructure, you have to learn k8s in addition to your cloud provider's infrastructure. It's all ALBs, ASGs, Security Groups, EBS Volumbes and IAM policy underneath and k8s, while very clever, isn't so clever as to abstract much of any of it away from you. On EKS you get to enjoy more odd limitations with your nodes than EC2 would give you on its own. You're already building on a cluster, your cloud provider's hypervisor. They'll literally build virtual compute of any size and shape for you on demand out of heterogeneous hardware and the security guarantees are much stronger than colocated containers on k8s nodes. There are quite a few steps between single server and k8s.
    • psini 2y ago
      You can self host Kubernetes on "dumb" VMs from Hetzner or OVH.
• czhu12 2y ago
  I'm not sure google cloud run can be considered a fair comparison to Kubernetes. It would be like saying AWS Lambda is a lot easier to use than EC2. I've used both Kubernetes and GCR at the current company I cofounded, and theres pros and cons to both. (Team of about 10 engineers) GCR was simple to run simple workloads, but, an out of the box Postgres database can't just handle unlimited connections and so connecting to it from GCR without having a DB connection proxy like PG bouncer risks exhausting the connection pool. For a traditional web app at any moderate scale, you typically need some fine grained control over per process, per server and per DB connection pools, which you'd lose with GCR. Also, to take advantage of GCR's fine grained CPU pricing, you'd have to have an app that boots extremely quickly, so it can be turned off during periods of inactivity, and rescheduled when a request comes in. Most of our heaviest workloads run on Kubernetes for those reasons. The other thing thats changed since this author probably explored Kubernetes is that there are a ton of providers now that offer a Kubernetes control plane for no cost. The ones that I know of are Digital Ocean and Linode, where the pricing for a Kubernetes cluster is the same as their droplet pricing for the same resources. That didn't use to be the case. [1] The cheapest you can get is a $12 / month, fully featured cluster on Linode. I've been building, in my spare time, a platform that tries to make Kubernetes more usable for single developers: https://canine.sh https://canine.sh, based on my learnings that the good parts of Kubernetes are actually quite simple to use and manage. [1] Digital oceans pricing page references its free control plane offering https://www.digitalocean.com/pricing https://www.digitalocean.com/pricing
  • igor47 2y ago
    Why are GCR and pgbouncer incompatible? Could you run a pgbouncer instance in GCR?
    • czhu12 2y ago
      I’m not an expert, but from what I understand, the standard set up is like: 4x(Web processes) -> 1x(pgbouncer) -> database This ensures that the pgbouncer instance is effectively multiplexing all the connections across your whole fleet. In each individual web process, you can have another shared connection pool. This is how we set it up
• jeswin 2y ago
  Docker swarm would have worked for 98.5% of all users (how k8s won over swarm should be a case study). And kamal, or some thing like it, would work for 88.25% of all users.
• paxys 2y ago
  > Kubernetes comes with substantial infrastructure costs that go beyond DevOps and management time. The high cost arises from needing to provision a bare-bones cluster with redundant management nodes. That's your problem right there. You really don't want to be setting up and managing a cluster from scratch for anything less than a datacenter-scale operation. If you are already on a cloud provider just use their managed Kubernetes offering instead. It will come with a free control plane and abstract away most of the painful parts for you (like etcd, networking, load balancing, ACLs, node provisioning, kubelets, proxies). That way you just bring your own nodes/VMs and can still enjoy the deployment standardization and other powerful features without the operational burden.
  • dikei 2y ago
    Even for on-prem scenario, I'd rather maintain a K8S control plane and let developer teams manage their own apps deployment in their own little namespace, than provisioning a bunch of new VMs each time a team need some services deployed.
    • spockz 2y ago
      I can imagine. Do you have complete automation setup around maintaining the cluster? We are now on-prem using “pet” clusters with namespace as a service automated on it. This causes all kinds of issues with different workloads with different performance characteristics and requirements. They also share ingress and egress nodes so impact on those has a large blast radius. This leads to more rules and requirements. Having dedicated and managed clusters where everyone can determine their sizing and granularity of workloads to deploy to which cluster is paradise compared to that.
      • solatic 2y ago
        > This causes all kinds of issues with different workloads with different performance characteristics and requirements. Most of these issues can be fixed by setting resource requests equal to limits and using integer CPU values to guarantee QoS. You should also have an interface with developers explaining which nodes in your datacenter have which characteristics, using node labels and taints, and force developers to pick specific node groups as such by specifying node affinity and tolerations, by not bringing online nodes without taints. > They also share ingress and egress nodes so impact on those has a large blast radius. This is true regardless of whether or not you use Kubernetes.
• politelemon 2y ago
  If you're looking to run a few containers you may also want to look at docker swarm itself. You h get some of the benefits of orchestration and a small manageable overhead. And it's just part of docker.
• tracerbulletx 2y ago
  Cool you left Kubernetes for a more locked in abstraction around Kubernetes like automation?
• JohnMakin 2y ago
  I don't understand how these posts exist when much of my consulting and career for the last few years has been on companies that have basically set up a bare bones out of the manual EKS/GCP solution and just essentially let it sit for 3+ years untouched until it got to a crisis. That to me as a systems engineer is nuts and a testament to how good this stuff is when you get it even kind of right. Of course, I'm referring to managed systems. Doing Kubernetes from scratch I would not dream of doing.
• throwaway984393 2y ago
  [dead]
• ldehaan 2y ago
  [dead]
• mitjam 2y ago
  I believe CloudRun is based on KNative which runs on Kubernetes. Thus, you’re still running on Kubernetes, it’s just abstracted away from you.
  • nosefrog 2y ago
    Nope.
    • p_l 2y ago
      It's not as much mentioned in the docs these days, and Google might have built a more optimized version, but it's still using knative APIs and originally was essentially KNative deployed on GKE just managed by Google.
      • nosefrog 2y ago
        Wasn't it always the same tech as app engine? I was there when they first released it.
• ofrzeta 2y ago
  so, what do you all think about CloudFoundry? :)
• solatic 2y ago
  Cloud Run is fine if you're a small startup and you're thinking about your monthly bill in three-figure or even four-figure terms. Like most serverless solutions, it does not permit you to control egress traffic. There are no firewall controls exposed to you, so you can't configure something along the lines of "I know my service needs to connect to a database, that's permitted, all other egress attempts are forbidden", which is a foundational component of security architecture that understands that getting attacked is a matter of time and security is something you build in layers. EDIT: apparently I'm wrong on Cloud Run not being deployable within a VPC! See below. GCP and other cloud providers have plenty of storage products that only work inside a VPC. Cloud SQL. Memorystore. MongoDB Atlas (excluding the expensive and unscalable serverless option). Your engineers are probably going to want to use one or some of them. Eventually you will need a VPC. You will need to deploy compute inside the VPC. Managed Kubernetes solutions make that much easier. But 90% of startups fail, so 95% of startups will fail before they get to this point. YMMV.
  • bspammer 2y ago
    I’m surprised Cloud Run doesn’t let you do this. You can put an AWS lambda in a VPC no problem.
  • jedi3335 2y ago
    Cloud Run has had network egress control for a while: https://cloud.google.com/run/docs/configuring/vpc-direct-vpc https://cloud.google.com/run/docs/configuring/vpc-direct-vpc
    • solatic 2y ago
      Nice, I didn't know about this, it wasn't available last time I checked. With that said... there are so many limitations on that list, that seriously, I can't imagine it would really be so much easier than Kubernetes.
  • p_l 2y ago
    kubernetes is how I keep compute costs in 2-3 digits :V
• oron 2y ago
  I just use a single k3s install on a single bare metal from Hetzner or OVH, works like charm, very clean deployments, much more stable than docker-compose and 1/10 of the cost of AWS or similar.
  • usrme 2y ago
    Do you have a write-up about this that you have to share, even if it's someone else's? I'd be curious to try this out.
    • fernandotakai 2y ago
      i was actually playing with hetzner and k3s over the weekend and found this https://github.com/vitobotta/hetzner-k3s https://github.com/vitobotta/hetzner-k3s to be super useful.
    • globular-toast 2y ago
      I've done this but on EC2. What would you like to know? Installing K3s on a single node is trivial and at that point you have a fully functional K8s cluster and API. I have an infrastructure layer that I apply to all clusters that includes things like cert-manager, an ingress controller and associated secrets. This is all cluster-independent stuff. Then some cluster-dependent stuff like storage controllers etc. I use flux to keep this stuff under version control and automatically reconciled. From there you just deploy your app with standard manifests or however you want to do it (helm, kubectl, flux, whatever). It all works wonderfully. The one downside is all the various controllers do eat up a fair amount of CPU cycles and memory. But it's not too bad.
  • p_l 2y ago
    Doing the same, grabbed a reasonably cheap Ryzen (zen2) server with 64GB ECC and 4x NVMe SSDs (2x 512G + 2x 1024G). Runs pretty much this stack: "Infrastructure": - NixOS with ZFS-on-Linux for as 2 mirrors on the NVMes - k3s (k8s 1.31) - openebs-zfs provisioner (2 storage classes, one normal and one optimized for postgres) - cnpg (cloud native postgres) operator for handling databases - k3s' built-in traefik for ingress - tailscale operator for remote access to cluster control plane and traefik dashboard - External DNS controler to automate DNS - Certmanager to handle LetsEncrypt - Grafana cloud stack for monitoring. (metrics, logs, tracing) Deployed stuff: - Essentially 4 tenants right now - 2x Keycloak + Postgres (2 diff. tenants) - 2x headscale instances with postgres (2 diff. tenants, connected to keycloak for SSO) - 1 Gitea with Postgres and memcached (for 1 tenant) - 3 postfix instances providing simple email forwarding to sendgrid (3 diff. tenants) - 2x dashy as homepage behind SSO for end users (2 tenants) - 1x Zitadel with Postgres (1 tenant, going to migrate keycloaks to it as shared service) - Youtrack server (1 tenant) - Nextcloud with postgres and redis (1 tenant) - tailscale-based proxy to bridge gitea and some machines that have issues getting through broken networks Plus few random things that are musings on future deployments for now. The server is barely loaded and I can easily clone services around (in fact a lot of the services above? instantiated from jsonnet templates). Deploying some stuff was more annoying than doing it by hand from shell (specifically nextcloud) but now I have replicable setup, for example if I decide to move from host to host. Biggest downtime ever was dealing with not well documented systemd-boot behaviour which caused the server to revert to older configuration and not apply newer ones.
• tombert 2y ago
  I’ve come to the conclusion that I hate “cloud shit”, and a small part of me is convinced that literally no one actually likes it, and everyone is playing a joke on me. I have set up about a dozen rack mount servers in my life, installing basically every flavor of Unix and Linux and message busses under the sun in the process, but I still get confused by all the Kubectl commands and GCP integration with it. I might just be stupid, but it feels like all I ever do with Kubernetes is update and break YAML files, and then spend a day fixing them by copy-pasting increasingly-convoluted things on stackexchange. I cannot imagine how anyone goes to work and actually enjoys working in Kubernetes, though I guess someone must in terms of “law of large numbers”. If I ever start a company, I am going to work my damndest to avoid “cloud integration crap” as possible. Just have a VM or a physical server and let me install everything myself. If I get to tens of millions of users, maybe I’ll worry about it then.
  • tryauuum 2y ago
    I have same thoughts. the only form of kubernetes I would be willing to try is the one with kata-containers for having all the security of virual machines.
  • nyclounge 2y ago
    Or if you got a static ip and fast up speed, then just port forward 80, 443 and start hosting your self. Even an old Intel Mac Book pro from 2000's with 4 GB of RAM may not be that hot running MacOS, but install Debian with no X. It is running smooth as a whistle, while running several conduit matrix, haraka, zone-mta, ice cast, nginx with no issues. WebRTC/TUN/STUN becomes an issue with the nginx config. May consider looking at pingora. The whole rust -> binary + toml file is super nice to run from system admin perspective.
    • d3Xt3r 2y ago
      > It is running smooth as a whistle ... until you get hit by a DDoS attack. Not much you can do about it unless your ISP offers protection, or you end up going for Cloudflare or the like instead of exposing your IP and ports.
• coding123 2y ago
  Cloud Run and K8S are not in the same space. One is to make the infra generic, Cloud Run ONLY works on GCP.
  • mitjam 2y ago
    Yes, but it's based on KNative (actually it's a managed KNative service[1]) which is based on K8s. You can get a dev experience similar to CloudRun with KNative on K8s. [1]: https://cloud.google.com/blog/products/serverless/knative-based-cloud-run-services-are-ga?hl=en https://cloud.google.com/blog/products/serverless/knative-ba...
• lmm 2y ago
  So how does this person link up the different parts that go into deploying a service? Like, yes, you can have a managed database, and you can have a managed application deployment (via cloud run), and you can have a pub-sub messaging queue, and you can have a domain name. But where's the part where you tie these all together and say that service A is these and service B is those? Just manually?
• hi_hi 2y ago
  I'm running kubernetes (actually k3s with helm, but that counts, right) on a ludicrously old and underpowered Ubuntu thinclient thats about 10 years old - https://www.parkytowers.me.uk/thin/hp/t620/ https://www.parkytowers.me.uk/thin/hp/t620/ I didn't _need_ to, and it was a learning curve to setup that had me crying into my whisky some nights, but its been rock solidly running my various media server and development services for the past few years with no issues. Sure, its basically a fancy wrapper around a bunch of docker containers, and I use hardly any of the features which k8s brings to the party, but your cold hard logic won't win over the warm and fuzzy feelings I get knowing I did something stupid and it works!
• seabrookmx 2y ago
  We dabbled with Cloud Run and Cloud Functions (which as of v2 are just a thin layer over Cloud Run anyways). While they worked fine for HTTP workloads, we wanted to use them to consume from Pub/Sub and unfortunately the "EventArc" integrations are all HTTP-push based. This means there's no back pressure, so if you want the subscription to buffer incoming requests while your daemons work away there's no graceful way to do this. The push subscription will just ramp up attempting to DoS your Cloud Run service. GKE is more initial overhead (helm and all that junk) but using a vanilla, horizontally scaled Kubernetes deployment with a _pull_ subscription solves this problem completely. For us, Cloud Run is just a bit too opinionated and GKE Autopilot seems to be the sweet spot.
  • gizzlon 2y ago
    > The push subscription will just ramp up attempting to DoS your Cloud Run service Interesting. My assumption would be that Cloud Run should quickly* spin up more containers to handle the spike and then spin them down again. So there would be no need for back pressure? Guess it depends on the scale? How big of a spike are we talking about? :) *Let's say a few seconds
    • seabrookmx 2y ago
      You typically have it configured with a cap on instances for cost reasons. Even if you don't, if you have other services (like a database) downstream, you might not want it to scale infinitely as then you're simply DoS'ing the DB instead of the Cloud Run service. Backpressure is really important for the resiliency of any distributed system, IMO.
• wetpaste 2y ago
  RE: slow autoscaling Maybe the cloud companies could do something here by always keeping a small subset of machines online and ready to join the cluster. Provided there is some compromise in what the configuration is for the end user. I guess it doesn't solve image pulling. Pre-warming nodes is an annoying problem to solve. Best solution I've been able to come up with is: Spegel (lightweight p2p image caching) + Karpenter (dynamic node autoscaling) + pods with low priority to hold onto some extra nodes. It's not perfect though
  • p_l 2y ago
    1. Do some capacity planning 2. Apply appropriate changes to application resources (like parameters for spreading pods around) 3. Add descheduler[1] or similar tool to force redistribution of pods 4. Configure your cluster autoscaling params according to values from step (1) and have it autoscale before nodes are too heavily loaded.
• PohaJalebi 2y ago
  I like how the author says that Kubernetes has a vendor lock in, yet suggests a GCP managed service as their preferred alternative.
  • dpeckett 2y ago
    That probably runs on Kubernetes (or Borg) under the hood.
    • p_l 2y ago
      Cloud Run explicitly uses KNative APIs... which are kubernetes objects.
• ribadeo 2y ago
  One can avoid container orchestration by avoiding the trend of containerizing your app. It wastes system resources, provides half-baked replicas of OS services, reduces overall security while simultaneously making networking a total pitas. Your cloud provider is already divvying up a racked server into your VPS's, via a hypervisor, then you install an OS on your pretend computer. While i can see how containerized apps provide a streamlined devops solution for rare hard to configure software that needs to run on Acorn OS 0.2.3 only, it should never be the deployment solution for a public facing production web service. Horses for courses.
• siliconc0w 2y ago
  The downside to cloud run is you don't get a disk. If I could get a pd attached at runtime to each instance, it'd be a lot more compelling.
• philbo 2y ago
  We migrated to Cloud Run at work last year and there were some gotchas that other people might want to be aware of: 1. Long running TCP connections. By default, Cloud Run terminates inbound TCP connections after 5 minutes. If you're doing anything that uses a long-running connection (e.g. a websocket), you'll want to change that setting otherwise you will have weird bugs in production that nobody can reproduce in local. The upper limit on connections is 1 hour, so you will need some kind of reconnection logic on clients if you're running longer than that. Ref: https://cloud.google.com/run/docs/configuring/request-timeout https://cloud.google.com/run/docs/configuring/request-timeou... 2. First/second generation. Cloud Run has 2 separate execution environments that come with tradeoffs. First generation emulates Linux (imperfectly) and has faster cold starts. Second generation runs on actual Linux and has faster CPU and faster network throughput. If you don't specify a choice, it defaults to first generation. Ref: https://cloud.google.com/run/docs/about-execution-environments https://cloud.google.com/run/docs/about-execution-environmen... 3. Autoscaling. Cloud Run autoscales at 60% CPU and you can't change that parameter. You'll want to monitor your instance count closely to make sure you're not scaling too much or too little. For us it turned out to be more useful to restrict scaling on request count, which you can control in settings. Ref: https://cloud.google.com/run/docs/about-instance-autoscaling https://cloud.google.com/run/docs/about-instance-autoscaling
• rcleveng 2y ago
  To me the most important quote in the whole writeup is this "Our new stack is boring.". When you are creating a solution to a problem, boring is good. Strive to be boring. Only be exciting when you must.
• thefz 2y ago
  5 years ago everyone and their uncle would swear that kubernetes was the new way of doing things and by not jumping on it you were missing out and potentially harming your career.
• figmert 2y ago
  This article kinda reads like "I didn't need HA, and you probably don't either". HA isn't necessary, but if you want a reliable system that will be online without being woken up at 3am (assuming you even have alerts at that point), you're better off with HA. Similarly, you don't need Kubernetes, but if you want something that makes developer's life's easier, makes it easy to use a singular API, has many, many integrations and tooling, then you're better off with K8s. Sure, you can go with VMs, but now you have to scale and manage your application on a per VM level instead of per container. You have to think about a lot of cloud specific services, network policies, IAMs, I don't know what else, scaling. I guess what I'm saying, you always have the option of writing in Assembly, but why would you when you can have a higher level language that abstracts most of it away. Yes, the maintenance burden on the devops/platform team is higher, but it's so much easier for users of the platform to use said platform.
• madjam002 2y ago
  I don't get these recent anti-Kubernetes posts, yes if you're deploying a simple app then there are alternatives which are easier, but as your app starts to get more complex then suddenly you'll be wishing you had the Kubernetes API. I'd use Kubernetes even if I was spinning up a single VM and installing k3s on it. It's a universal deployment target. Spinning up a cluster isn't the easiest thing, but I don't understand how a lot of the complaints around this come from sysadmin-type people who replace Kubernetes with spinning up VMs instead. The main complexity I've found from managing a cluster is the boring sysadmin stuff, PKI, firewall rules, scripts to automate. Kubernetes itself is pretty rock solid. A lot of cluster failure modes still result in your app still working. Etcd can be a pain but if you want you can even replace that with a SQL database for moderate sized clusters if that's more your thing (easier to manage HA control plane then too if you've already got a HA SQL server). Or yes just use a managed Kubernetes cluster.
  • noname44 2y ago
    lol, even if you have complex apps there are always easier solutions than Kubernetes. It is evident that you have never run such an app and are just talking about it. Otherwise, you would know the issues you would encounter with every update due to breaking changes. Not to mention that you need a high level of expertise and a dedicated team, which costs far more than running an app on Fargate. Recommending a managed Kubernetes cluster is nonsense, as it goes against the whole purpose of Kubernetes itself.
    • madjam002 2y ago
      I've been running apps on Kubernetes clusters for the past 6 years and the only thing that really comes to mind that was a breaking change was when the ingress class resource type was introduced. Everything else has been incremental. Maybe I'm forgetting something. What's wrong with recommending a managed cluster? I wouldn't use one but it is certainly an option for teams that don't want to spin up a cluster from scratch, although it comes with its own set of tradeoffs. My project at the moment is definitely easier thanks to Kubernetes as pods are spun up dynamically and I've migrated to a different cloud provider and since migrated to a mix of dedicated servers and autoscaled VMs, all of which was easy due to the common deployment target rather than building on top of a cloud provider specific service.
• retinaros 2y ago
  The hell of k8s for le are the libs in version 0.12536 and all the frequent updates and compatibility versions you need to maintain between them. Why would anyone want to do that in prod. It feels like at some point the tech needs to mature in something more stable and its not coming
• Imustaskforhelp 2y ago
  Interesting , I was reading the author's comments in the hackernews post about https://www.gitpod.io/blog/we-are-leaving-kubernetes https://www.gitpod.io/blog/we-are-leaving-kubernetes And I was there also curious about it , and had posted some questions , though now I am not sure if all of those were interested. But yeah , I also agree with this sentiment , personally I would much rather optimize my code in hopes that it can run on single machine (vps like hetzner) instead of the dread of kubernetes Though I also want to see kubernetes , so maybe some day I am going to experiment it for the sake of experimentation with something like kubectl or other minimalist single computer based approach to learn new things. I also feel like I can't benchmark things unless done through kubernetes like anton putra , so I am not sure.
• ptman 2y ago
  So what is the self-hosted solution for running something like cloud run on your own hardware? Multiple servers and optional redundancy so not dokku.
  • josegonzalez 2y ago
    Dokku runs on multiple servers by abstracting other schedulers (kubernetes, nomad). You don't really need to deal with the scheduler side (other than setting it up on the new servers in your cluster, which is a single command we document). Disclaimer: I am the Dokku maintainer
• jusonchan81 2y ago
  I have to admit that I’m inspired to get rid of k8s for a couple of situations based on all the recent posts. There are some clusters that are not required and costs way too much to run compared to a single VM.
• Panik 2y ago
  Nice solution. Just use vendor lock-in, so you can't use your own shit later.
• torginus 2y ago
  By the way how much malice can we attribute to cloud vendors? >Moreover, Kubernetes’s slow autoscaling meant I had to over-provision services to ensure availability, paying for unused resources rather than scaling based on demand. A typical Linux instance on AWS starts up in about 8 seconds from the asking to start to command line, so lets double that. You could start up and shut down instances in 15 seconds. Why the hell do you need to overprovision instances then, or leave empty ones running? I don't see any other reasonable explanation than its in the best interest of cloud vendors to not have short lived instances for the purposes of load balancing, as well as make you consume as much CPU time as possible, even if you don't need it.
  • p_l 2y ago
    TL;DR someone didn't learn to plan capacity.
    • torginus 2y ago
      I don't like that enshittification of technology has created an entire profession around babying pretend servers - if the cloud lived up to the promise of being an operating system for a global server farm, capacity scaling would consist of a new computer being allocated in ~1ms, said computer downloading my 100MB binary in ~10ms and loading it into memory and jumping to the first instruction in ~0ms. And I'm being generous with the timings.You could literally spin up a server in the time it takes to process a request. If I write a program and decide to start a new thread, I don't need a dedicated ThreadOps professional moralizing about how I should've configured my app in the yaml file to use 2 threads instead of 1. I have long held the belief that the clowning around that happens in IT nowadays is the opposite of engineering.
• rnts08 2y ago
  The worst thing about the whole kubernetes-cult and the like is that IaC and CM was supposed to help us reduce configuration, to make it less prone to failure and easier to manage. But the truth is that we ran service based architecture, network meshes and containers with bash just fine before cloud everything, usually with less effort than it is to do literally anything today. Sure you had to know how to set up network bonding and how to tune your systems. Very, very few people and businesses _needs_ kubernetes or it's cousins. Most just need a decent system administrator.
  • boingo 2y ago
    I agree with you, but can offer a counterpoint. A system administrator might setup a server in a certain way, and if they leave the replacement has a hard time discovering what they did. With IaC, all decisions are templated in the code and the replacement has full insight into the state of the machine.
  • p_l 2y ago
    As I hope decent system administrator... turns out I find running minimal kubernetes setup (k3s is great there) more sensible and cheaper than managing the server in classic way.
  • pas 2y ago
    Good for you, my bash on balls was unfortunately not able to solve service based meshtainer architecture :( I remember how kernel module params for ALG/bonding/teaming had to be figured out for each fancy NIC/driver, it was fun, but definitely not great. Of course this is mostly solved if you pay the cloud premium. Most businesses need a product guy/gal, a website, then a developer, then later one VM or Heroku or something. And maybe if there's at least a few customers, yeah, it makes sense to think about ops (as in business ops), and then eventually sure, engineering needs to scale to solve the challenges, and that might mean getting a sysadmin.
• buro9 2y ago
  I still just use VPS devices and some colocated hardware. I'm constantly told "I am not the typical customer, most do use K8s". Side projects and volunteer things I run include 470 websites serving about 320K registered users and about 500K-1M monthly guest users... on about 15 VPS devices or small servers. It's just classic dynamic websites, HTML is the output of a server (no single page web apps), lots of caching for guest users, reasonable complexity (load balancers ahead of web front ends ahead of API back ends ahead of databases). Things that look like microservices exist... it's just API calls, and they come back through the front door, it's easy to reason about. Monitoring is mostly Prometheus node exporter, it turns out that CPU + Memory + Disk IOPS + Network IOPS is +90% of what you need... some HTTP logs or profiles are the last 10%. It's just simple... this is run in my spare time, effort is under an hour per month. (and no it's not monetised, not everything has to be)
• lakomen 2y ago
  Preach, however all fucking job ads want proficiency in k8s. So screwed if you do, screwed if you don't
• vergessenmir 2y ago
  it's baffling to me that realising that your application deployment requirements don't fit k8s means that k8s is not fit for purpose. Note that the alternative isn't a return to the management of VMs, which we all agree none of us ever want to return to, but is a cloud service which hides the concerns that k8s makes explicit. It's a bit short-sighted (granted not intentionally) to talk about k8s not being fit for purpose without specifying the workloads it's not suitable for and at what scale of operations. Often times when these discussions come up it's usually reduced to a stateless web app being the justification for illustrating the shortfalls of k8s as a solution. Personally, I'm not a big fan of k8s because it can be complex but the alternatives tend to be more complex. I can guarantee, your solution over time will converge on operational knowledge trapped in the minds of your team members , runbooks and custom scripts that express an abstraction that k8s implements perfectly well.
• netdevphoenix 2y ago
  There seems to be a trend of people ditching Kube, I wonder why. There are currently two trending articles about this right now on this website
• daitangio 2y ago
  I created [misterio](https://github.com/daitangio/misterio https://github.com/daitangio/misterio) to exactly solve the same issue described in the article. So yes, if you have a simple setup you can use a "web" of docker servers. But if your needs are just a bit more complex, or you have a mix of different technology (rubyonrails, python flask, java spring boot) and you want to standardize the communication, the security, the performance tracking and so on, K8s is the way. The problem is the microservice architecture: the complexity is pushed to the infrastructure by the microservices, and K8s is a super-generic way to solve that complexity. It is like unpacking (in a exploding-kitten mode) an application server on the your Intranet.
  • jdsleppy 2y ago
    Nice, that is my current approach on my latest project (referring to what your linked repo does). Inlining it here for others: copy a .env and a docker-compose.yml, run `docker compose "$@"` on the instance via ssh. That lets me deploy, run one-off commands, and tail logs easily.
• stuaxo 2y ago
  For a lot of things it's not needed. There is a huge tail of small websites where if it comes down someone can fix it and that's fine.
• isoprophlex 2y ago
  > Complexity Overload Wait... so you're saying, I can rent a couple of beefy boxes instead, and live a life where I don't need to know what TolerateDuringExecutionButNotDuringScheduling does?!
• anal_reactor 2y ago
  At my company we're running EKS Fargate. We spend very little time actually managing clusters, and it really boils down to "just running some containers". I don't understand why the hate, the solution really works for my use case.
• openplatypus 2y ago
  You can start using Kubernetes with one YAML file and docker image. If you want to skip devops work of setting up cluster, go K8s managed. All major cloud providers give you that option. So, you replaced opensource, standard cloud solution such as K8s with proprietary cloud platform by Google. I mean, good for you but for many of us this sounds backwards. As a bootstrapped founder, going with K8s was the best decision ever: deployment, scaling, custom resources, jobs, cron, etc, all that stuff comes included and it costs writing a YAML file. Do you have to learn something to use K8s. Yes, like with everything. Is it difficult? No more difficult than any other solution out there.
• rqtwteye 2y ago
  I think Kubernetes config is a perfect target for AI. Conceptually k8s is not that complex but setting up and managing all YAML files with tons of boilerplate is very hard if you don’t do it every day. An AI should be the perfect tool to analyze the files and make changes.
  • Havoc 2y ago
    Busy using a fair bit of A.I. to produce yams manifests. It still hallucinates a fair bit and doesn’t connect the dots well between the various parts of you colour outside the lines even slightly Works but feels substantially weaker than on coding tasks. Not sure whether that’s lack of training material or the lack of execution flow code has.
  • miyuru 2y ago
    Hard no. There is already people who uses k8s that does not understand how it works or even know how the app works inside k8s. Adding AI to it is just asking for a disaster down the line.
• m0llusk 2y ago
  So this is an ad for a Google product? What happens when Google gets bored and drops it?
  • p_l 2y ago
    Well, since it's cloud run... ... you spin knative on kubernetes cluster and copy over the actually-k8s-manifests over :P
• nixpulvis 2y ago
  Never once has anyone come close to the developer experience of Heroku and RoR circa 2014.
• deleted 2y ago
  [deleted]
• rbanffy 2y ago
  I have concluded that most (perhaps all) cloud offerings of Kubernetes clusters are a prime example of "enshitification". You take something that should be easy and trivial, then add layers of complexity and vendor-specific components, slightly different configuration options, lots of plug-ins you probably won't ever need (but you need to know which ones you will, and how to decide which is which), and so on. The result is that, when you are finished, you made something that runs only on the specific provider you selected.
• sn9 2y ago
  OP I'm on firefox 131 on Ubuntu and your site is basically unreadable. Dark background and the text is the faintest of shades lighter. I don't mean this in a nitpicky way. Normally I'm fine with most sites. I have to highlight the text to read it.
  • beaugunderson 2y ago
    Chrome on macOS is the same, bailed when I couldn't read the text.
• fhke 2y ago
  >The high cost arises from needing to provision a bare-bones cluster with redundant management nodes. ?! How else are you managing your infrastructure besides having redundancy in the control plane? Even if it’s a chef or puppet server, or even sysadmin Dave running scripts on his laptop, you should still have redundancy. > Moreover, Kubernetes’s slow autoscaling meant I had to over-provision services to ensure availability, paying for unused resources rather than scaling based on demand. ?????? Slow compared to what exactly? Anecdotally, k8s with karpenter is significantly faster to scale than auto scaling groups.
• hackburg 2y ago
  [dead]
• prakashn27 2y ago
  Serverless is the best thing that happened for cloud. People argue that it is expensive for zillions of users but I have never reached that scale