5 ms·
Kubernetes and back – Why I don't run distributed systems
- Havoc 3y agoI thought the DB isn’t supposed to go in the cluster?
- danpalmer 3y agoDepends on what you're using your cluster for. When you're big enough it's worth just treating the database like any other job and running it on the cluster, but this can be tricky to get right. For small K8s deployments many people choose to outsource the database to their cloud provider and have it separate from the cluster as there are fewer things that can go wrong.
- MrSkyNet 3y agoAnd why did you thought this? With an operator, you get HA setup with switch over etc. basically for free. I would prefer relativly cheap managed db over hosting it myself but its absolutly fine.
- samcat116 3y agoSeems like all the issues described here are with stateful application and running HA mongo. I don’t think these performance issues are really a result of K8s itself. I do agree backing stuff up in this situation isn’t easy. I think that’s why a lot of folks punt their stateful services to managed services.
- ldjkfkdsjnv 3y agoPeople are getting confused, alot of open source software released by major players (Kubernetes, Next.JS (which is really just FBs take on react SSR), Microservice architectures), have no place in a company smaller than 100 people. But they use them anyways, and it ends up costing a ton of money for almost no business benefit.
- dartos 3y agoVercel, netlify, AWS, and gke have great marketing
- blackoil 3y agoThese posts come up so frequently, people keep falling for buzzwords they don't understand and then start blaming the tech. Distributed systems are all cool and fun but use them as a last option when you have exhausted all other choices. > I took a moment to calculate the cost of hosting my app on the K8s offerings of a Hyperscaler like AWS, Google or Azure just to figure out that this won’t be an option for me. Clear signal that you can't justify the overheads. Cloud services are costly but in most cases you'll pay similar cost in terms of time and in some cases lot more if you pick something complex that you don't know about.
- finaard 3y ago> Cloud services are costly but in most cases you'll pay similar cost in terms of time and in some cases lot more if you pick something complex that you don't know about. Even just something like EC2 instances is costly - and a surprising amount of stuff in the cloud is just containers or EC2 instances, before redundancy even. Adding in specialized services makes things even more costly. Problem is that nowadays it's hard to find a team that knows how to properly handle hardware - if you manage to get a handful of people that know how to do that cloud is only sensible for very few edge cases. And as the article states - in many cases you probably don't even need all that fancy from K8s. I can for the cost of using K8s cloud stuff just put a second copy of our hardware in a different DC (or maybe even a third one), and just fail between DCs if needed. Way simpler setup, and excludes a lot of failure cases I'd catch with a K8s setup.
- pdimitar 3y ago"You are holding it wrong", huh? From the homepage https://kubernetes.io/ https://kubernetes.io/: "Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications." Do you see "not recommended for smaller-scale applications" anywhere? Including on the entire home page? Looking for "small", "big" and "large" also yields nothing.
- riku_iki 3y ago> use them as a last option when you have exhausted all other choices. if one wants to run service with high availability (no downtime if server died), what are other choices then?..
- danpalmer 3y agoK8s requires either a (small) ops team to run it or using a hosted cloud solution. It's powerful and pays off when you have this, but if you can't afford to pay for the cloud hosting or hiring a team to do it, you're not going to get the benefits, you're not the target market. Many products have a minimum size at which they are effective, no one is buying an Oxide rack for their homelab or side project, no startup going through YC is buying an IBM mainframe, because they are both obviously nowhere near the minimum effective size/complexity/spend. K8s is the same, and yet it seems to trip up so many people.
- alde 3y agoRunning a self-hosted Mongo cluster is tedious and complicated task by itself. Mongo Inc isn’t incentivized to make it easy and user friendly because they have a cloud offering for you. In addition to that Kubernetes isn’t really tailored to stateful workloads, so there are all sorts of gotchas and surprises there.
- htrp 3y agoThe question is whether the potential revenue growth justifies the growing technical complexity.
- arkh 3y ago> “JUST PUT IT ON A SERVER!!1” For all your data: yes. It means persistent filesystems and databases. Kubernetes will still be perfect to manage your running apps, but data is "the rest of the owl" of most devops / cd proponents and tools. "I should be able to rollback to a previous working state", well I hope your last deployment did not change your database schema.
- MrSkyNet 3y agoLocalPV supports ZFS Snapshots. MongoDB is shitsoftware anyway. I use k8s in a very small setup (3 worker nodes, 3 ctrl planes) in my startup. But i'm working professionally with k8s for 6 years. My setup is relative big but also relative simple and most of it is also IaC. I don't need to backup my setup and i don't need to document how i setup everything. I have blue/green deployment, multiply enviorrnments (dev, test, prod) configured 100% equal, a good monitoring stack + tracing, s3 replication, postgresql ha and offside backup. i can shut down a node and nothing happens. But you know, if k8s doesn't fit your time and experience and benefit estimation, don't use it. The upfront cost of k8s is bigger though.
- deleted 3y ago[deleted]
- santoshalper 3y agoWhy on earth did the author even believe he needed K8s to serve 500 concurrent users? You could do that on a couple basic web servers behind a load balancer 15+ years ago. Hell, you could probably run the whole app, including the database on a single server if you don't care about availability.
- MrSkyNet 3y agoFor me its about IaC, automatisation. Building my app, deploying a container image and building a helm chart is not a lot of work if you have experience and the initial additional work is only happening once. After that, you have your whole app definition in code, seting up a second or third env takes 5 minutes. I really should write a blog article about my stack and the advantges of it (git, kubernetes, argocd, helm, ...)
- pram 3y agoI’ve read such insanely doctrinaire opinions here on HN like you must NEVER!!! ssh into a system, and instead have some rube-goldberg distributed contrivance for deployment and debugging and anything else. I presume this attitude is how we got to people running their phpbb blog in a 5 node kubernetes cluster. A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime. Your hobby project isn’t Google.
- moondev 3y ago> A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime. Your hobby project isn’t Google. 73.6% of All Statistics Are Made up https://www.datacenterdynamics.com/en/opinions/ovhclouds-data-center-fire-one-year-on-what-do-we-know/ https://www.datacenterdynamics.com/en/opinions/ovhclouds-dat...
- Borg3 3y ago18:37:01 up 1618 days, 6:40, 6 users, load average: 0.00, 0.00, 0.00 Just go away and burn ;)
- riku_iki 3y agosurvivalship bias..
- profstasiak 3y agothe original comment was about the possibility of achieving 100% uptime, not that it's an expected outcome > A dedicated server from OVH running some processes under lxc and systemd can achieve 100% yearly uptime.
- moondev 3y agoWhat is point exactly of saying something "might" happen 100% of the time?
- Borg3 3y agoDistributed systems are cool, but people I think got carried away with complexity. Hardest part here is active-active replication, but this is needed for very few who need true 100% availability and big scaling. For majority of people active-backup setups are perfectly fine. You can also run frontend-backend mode where frontend takes care of caching, passing write to backend nodes. I run couple of systems like this and lives good. Failure of frontend node is absolutly no brainer. Failure of backend node with active-backup is also easy. You just lose last not commited and not synced transactions.
- charles_f 3y ago> All I’m saying is: Choose your weapons carefully. That's the right moral of the story to take away. > Today, I’m back with a single machine that handles the peak 500 concurrent users And this is what tells me k8s is probably not the right tool for the job. Something that runs on a single machine should absolutely run on a single machine... Unless if there's a pre-existing, funded and operated cluster you can tap into. No abstraction comes for free, as the author mentioned you need to figure routing, backup, etc. You are paying a premium for the fact that you want to make that somewhat generic enough to fit with multiple use cases. The abstraction doesn't materialize for the operator of the cluster. It's there for the customers of that cluster - the devs who can now avoid to configure whatever the cluster is already handling.
- mbonca 3y agoWe ran our project on Google's K8s for $76 a month for quite a while. They provide one free Master. There is no way, EVER, that I would host K8s myself for the size of the projects I usually work on. But having a panic in one of my services at 1am, and K8s auto-restarting the pod is amazing. Namespaces even let us run three separate environments for that same $76. We have a couple thousand users per day who are in the app all day, so not a large project. We use other services like GCP Cloud Storage, Pub/Sub, PostgreSQL, etc. that require next to no setup. We are up to 3 nodes in our small cluster and still costing us less than $350 a month. K8s is repetitive. You create one YAML for a service and the rest are similar. Rolling releases are a simple script. As small as our company and product is, I think K8s has been one of our best decisions. Some things like MongoDB can quickly triple your costs though. We were able to pick technologies that kept the cost really low and stack maintainable.
- b7we5b7a 3y agoI run kubernetes as... a standalone node. I run it on a dedicated server with two disks on btrfs RAID1, with a subvolume for each pod that needs it (hostPath). Not the minikube version, not the kubespray version: the kubeadm version, installed by hand and minimally tweaked (NodePorts 0-65535), with dual-stack networking support. I have ~3 tenants, 5-15 pods each, and managing this configuration with docker was a steadily increasing pain for the last few years: - I had docker-compose split-files for each service for each tenant to keep a consistent state, so I used a simple bash script to generate a really long docker-compose line to bring everything together, networking was a real pain - Dual-stack support in docker does not exist/work in compose v3 to this day, but v2 has it - Querying the configuration and state of a container was an exercise in debugging the output - For certificates, I had to run letsencrypt/certbot externally (2 tenants with separate accounts for wildcard certs), but use traefik in docker-compose (the syntax pain was on top of the rest, of course) Kubernetes simplified all the pain points above: - I still have the service split-files, but I don't care about network addresses and links between pods anymore, just their names - better isolation between all the architecture points, and clearer definition and integration - IPv4&IPv6 native just working - "describe" gets me all the information I need in a pretty format, "get type name -o yaml" if I need the configuration - standard nginx ingress + cert-manager, and I can have multiple tenants without resorting to hacks DBs are regular containers in the pod that requires them, just like any other container. Daily subvolume snapshots are small and effective to make sure the data is safe. Kubernetes is a much better orchestrator than docker compose, availability is a plus if you really need it. You don't need to build an HA control plane plus 3 workers if you have no use for it.