6 ms·
We cut costs by 70% by moving from GCP and CockroachDB to Hetzner and PostgreSQL
- hklijlyh 2y ago[flagged]
- threeseed 2y agoNot really. You didn't factor in the time it takes to manage k3s yourself versus GKE which is managed.
- unixhero 2y agoMaybe it is a very simple setuo
- CactusBlue 2y agoProbably. But it was only a single command to get up a k3s node - not much difference in effort for a relatively low-workload, low-critical service.
- threeseed 2y agoIt's probably only a few clicks to get a GKE cluster. The point is that it's not really a fair comparison.
- jitl 2y agoSelf management could be expensive if it’s actually going to burn hours of dev time. At teensy scale is k3s really gonna take much “management” after the bring-up? I don’t think so. For a teensy team, it seems totally worth it to take a runway from 12 months to 40 months for the price of a few afternoon’s work.
- CactusBlue 2y agoThis. Just upgrading this made me (a fresh university grad with minimum savings) basically and my startup "default alive" so that the startup can be bootstrapped
- jauntywundrkind 2y agoThe wins here are absurd, & the resistance to them remains as staunchly resolutely inconsiderate as ever. I've done Lets Encrypt by hand. I've had rotation not work as expected. I've done postgres by hand. I've spent work weeks setting up and testing backups. I've dealt with fallout from another person's backups scripts not working. I've worked with teams that took weeks to setup replicas. I've been on teams that have spent quarters building and improving ci/cd systems. Folks will never ever shut up about how you might not need Kubernetes, but damn this post documented an extremely short path through a couple kind of annoying to DIY things that just bloody work and at a much much much higher level of productization than the bullshit bespoke scripts your own team would put together. Traeffik load balancer with l7 routing? Already running. Let's Encrypt? Works, reliably. Internal DNS? Out of the box. Postgres? Incredibly well managed. Ci/cd? A nearly Heroku like beautiful world of gitops. These pieces all snap in to the general framing with ease & grace; this whole post is a complete DIY infrastructure stack that one could do in an afternoon, and even better the output is predictable & understandable & legible by another practitioner, which the "real men just do it themselves" attitude doesn't cover. The bullshit asymetry principle feels soooo strong here. Spreading fear uncertainty & doubt, yucking people's yum, is going to waste an incredible amount of time.
- papichulo2023 2y agoDont you have it out of the box with some distros?
- jitl 2y ago48.08 USD per month on a single-node GKE cluster w/ 2 cores 4 gigs -> 15.34 USD per month Hetzner VPS w/ 4 cores 8 gigs.
- lyu07282 2y ago48 seems really low to me, I've never seen a k8s cloud cluster for less then 100-150 a month on the very low end, also doesn't take egress into account how do you even run a k8s cluster "on 2 cores 4 gigs", don't you need at least 3 nodes?
- jitl 2y agoYou need 3 nodes for a high availability control plane if you’re managing the Kubernetes control plane yourself. With GKE, the control plane is totally managed by Google Cloud, things like etcd etc are not running on any of your instances. It’s fine to have just 1 compute node (like the before state), or to put all the k8s control plane on a single node (like the after state) if you don’t need high availability.
- deathanatos 2y agoIt would have to be a zonal cluster, I think. A regional cluster would have 3 nodes, minimum (ish¹). (Unless maybe if you can do a two-zone regional cluster? But I'm not sure why you'd ever do that, though. I don't have much experience w/ the zonal ones, since typically we want our service spread across multiple failure domains. OP seems to be in the bootstrap phase, so the cost savings might be > the reliability at their present scale, to them, which I can respect.) ¹you can have a zero node regional cluster, but it isn't going to do much work.
- shrubble 2y agoI predict hard times for GCP as they will get squeezed by Amazon and Azure, plus the kind of belt-tightening we see here.
- CactusBlue 2y agoGCP is expensive even compared to Amazon and Azure. The only reason I really used it was because it was the cloud that I was used to (when employer was covering for the cost), and I kinda stuck with it.
- papichulo2023 2y agoI predict a very hard time to CTOs once shareholders realize how money they are wasting on the cloud.
- paxys 2y agoIt's wild to me that we are talking about a site running on a single 2 core VPS costing $15/mo and the developer didn't stop to consider that maybe they don't need Kubernetes for this. Like...just run your damn services on the box like we've been doing since web hosting first became a thing.
- CactusBlue 2y agoFor me, the main benefit of Kubernetes is declarative deployments. You can have the entire deployment automated through this and GitHub actions, without much fiddling with the production environment manually. I know, it's overkill and I don't like it that much either, but it's the industry standard.
- ben_jones 2y ago> but it's the industry standard. I strongly disagree with this statement. Even at the most bubbly of Silicon Valley events you will not see kubernetes for a smaller deployment be an industry standard. It’s ok to make architectural mistakes, or otherwise over engineer a thing. We all do it. Let’s not standardize it. And to your credit thank you for publishing this example as a learning opportunity to others.
- CactusBlue 2y agoI mean as in what the big (and small) clouds support. I haven't seen any clouds supporting Hashicorp Nomad or Docker Swarm!
- SJC_Hacker 2y agoYou shoudl be able to run any of those things on any cloud. A box with sshd is a box with sshd.
- jitl 2y agok3s is a teensy sqlite based orchestrator, something tons of devs run on their laptops for local dev. I think those are all single box clusters, why is it crazy to run it on a server but not a laptop?
- riwsky 2y agoI hear that you and your band have sold your guitars and bought turntables. I hear that you and your band have sold your turntables and bought guitars.
- eduction 2y agoI want to make a Yaz record
- erlenmayer 2y ago[dead]
- tayo42 2y agoI should write a blog about how I saved the same amount in absolute terms by turning off the stuff i forgot is running in the cloud. Side note: the anime girls all over is kind strange. Like the header is giant cartoon tits? Why?
- CactusBlue 2y agoI use AI-generated anime girls for article covers because I'm a weeb and it makes the rather-thick wall of text slightly more bearable.
- deleted 2y ago[deleted]
- rbtprograms 2y agoYea, the obivously AI-generated big-titty anime woman header was super out of place.
- mycentstoo 2y agoAs an SRE who’s moved between a few companies recently, I’ve noticed a trend towards simpler infrastructure options rather than k8s. Most places I interviewed at used AWS with just fleets of EC2s managed through ASG and instance refresh. Deployments are no downtime. It’s pretty nice and if you want to run with docker on the host instances you can but you can also just install the application using systemd.
- secondcoming 2y agoThis is what we do for GCE. Our CI builds a system image and then that's deployed across our datacentres. Worked perfectly for years, but still people want to switch to kubernetes. What can you do?
- mycentstoo 2y agoOh definitely that happens and from my experience I don’t think there’s a lot that can be done unless there’s strong faith in engineering leadership to push back on the complexity.
- jauntywundrkind 2y agoHow do you build ec2's? Funny thing is, everyone will have their own answer. Few will look alike. It's maybe simpler, but also bespoke. There's different definitions of simpler. I've rarely felt like folk's have a great grasp when they cobble together a bunch of options. Things that maybe one engineer started as a simple project 5 years ago grow and sprawl. They rarely stay simple. What this team gets out of the box here is not to be trivialized. You havent started talking to the ci/cd that ArgoCD deploys. How are load balancers setup & managed, and at what price (Traeffik here is basically free)? Well managed Postgres with backups & replication, Lets Encrypt... handled. Is it really simpler to keep DIY'ing bits and bobs? Are you confident your simple start will work reliably & won't need improvement? What is better about your pile "simple" bespoke solutions, and why not have a platform that integrates varying concerns with consistent management patterns & autonomic operators?
- 2y ago
- JB_Dev 2y agoThe time investment here doesn’t really make sense to me. That 70% savings was actually only $33 a month. Was it really worth spending that dev time and effort for such a small return rather than the product itself? And yea now you get to manage your own cluster… thats an ongoing cost too, not free.
- beoberha 2y agoYeah feels a bit disingenuous to title the article “70%”. While correct, it elicits the thought of a much more interesting migration than what actually happened.
- motoxpro 2y agoThis is what I always think. People will spend $10,000 on engineering time (2 engineers working ~1.5 weeks) to save $500, that then turns into a COST later because they have to spend more on maintenance.
- xnx 2y agoNow they get to write an article about it.
- monkaiju 2y agoFun article! I had a great time playing with k3s a while back and this makes me want to get back to it
- suyash 2y agoKubernetes - is that thing still around?
- beoberha 2y agoSeems strange to have used CockroachDB but not leverage any of the distributed nature of it? I was looking forward to some interesting database problems in this article, but it was quite anticlimactic.
- CactusBlue 2y agoThere were also some performance issues regarding the read/writes compared to plain PostgreSQL, but these were relatively minor compared to issues with the extensions and tooling
- secondcoming 2y agoThanks for the Japanese cartoon porn warning! /s
- nukeyoular 2y ago[dead]
- nathants 2y agohetzner egress is x100 cheaper. it’s a good reason to switch.