4 ms·

I also recently left Kubernetes. It was a huge waste of time and money. I've replaced it with just a series of services on Google Cloud Run and then using Goo

I also recently left Kubernetes. It was a huge waste of time and money. I've replaced it with just a series of services on Google Cloud Run and then using Google's Cloud Run Tasks services for longer running tasks. The infrastructure now incredibly understandable and simple and cost effective. Kubernetes cost us >$million in both DevOps time and actually Google Cloud costs unnecessarily, and even worse it cost us time to market. Stay off of Kubernetes as long as you can in your company, unless you are basically forced onto it. You should view it as an unnecessary evil that comes with massive downsides in terms of complexity and cost.

elcomet 2y ago
Aren't you afraid of being now stuck with GCP?
- bhouston 2y ago
  It is just a bunch of docker containers. Some run in tasks and some run as auto-scaling services. Would probably take a week to switch to AWS as there are equivalent managed services there. But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is nothing much to be gained by moving from one to the other unless politics are involved (e.g. someone high up wants a specific provider.)
  - spwa4 2y ago
    How does the orchestration work? How do you share storage? How do the docker containers know how to find each other? How does security work? I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.
    - bhouston 2y ago
      > How does the orchestration work? Github Actions CI. Take this and make a few more dependencies and a matrix strategy and you are good to go: https://github.com/bhouston/template-typescript-monorepo/blob/main/.github/workflows/ci.yml https://github.com/bhouston/template-typescript-monorepo/blo... For dev environments, you can add post-fixes to the services based on branches. > How do you share storage? I use managed DBs and Cloud Storage for shared storage. I think that provisioning your own SSDs/HDs to the cloud is indicative of an anti-pattern in your architecture. > How do the docker containers know how to find each other? I try to avoid too much communication between services directly, rather try to go through pub-sub or similar. But you can set up each service with a domain name and access them that way. With https://web3dsurvey.com https://web3dsurvey.com, I have an api on https://api.web3dsurvey.com https://api.web3dsurvey.com and then a review environment (connected to the main branch) with https://preview.web3dsurvey.com https://preview.web3dsurvey.com / https://api.preview.web3dsurvey.com https://api.preview.web3dsurvey.com. > How does security work? You can configure Cloud Run services to be internal only and not to accept outside connections. Otherwise one can just use JWT or whatever is normal on your routes in your web server.
      - dthg 2y ago
        > But you can set up each service with a domain name and access them that way. Are you using Cloud Run domain mappings for this or something else? I have been converging on a similar stack, but trying to avoid using a load balancer in an effort to keep fixed costs low.
        bhouston 2y ago
        Yup domain mappings for now. There is some label support in Cloud Run but I haven’t explored it yet. You can also get the automatic domain name for a service via the cloud run tools. Yeah I definitely want to also avoid a load balancer or gateway or end points as well for cost purposes.
- sofixa 2y ago
  One of Cloud Run's main advantages is that it's literally just telling it how to run containers. You could run those same containers in OpenFaaS, Lambda, etc relatively easily.
rglover 2y ago
What stack are you deploying?
- bhouston 2y ago
  Stuff like this, just at larger scale: https://github.com/bhouston/template-typescript-monorepo https://github.com/bhouston/template-typescript-monorepo This is my living template of best practices.
  - rglover 2y ago
    I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.
    - Imustaskforhelp 2y ago
      yeh I have same thoughts , also if possible , bun can also reduce memory usage in very very basic scenarios https://www.youtube.com/watch?v=yJmyYosyDDM https://www.youtube.com/watch?v=yJmyYosyDDM Or just https://github.com/mightymoud/sidekick https://github.com/mightymoud/sidekick or coolify or dokku or dockify , like there are million of such things , oh just remembered kamala deploy from DHH and docker swarm IIRC (though people have seemed to forget docker swarm !) I like this idea very much !
    - Imustaskforhelp 2y ago
      there was some recent HN post which showed that they didn't even use docker but rather there was some other mechanism and it was so so simple , I really enjoyed that article
    - bhouston 2y ago
      > I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts. I appreciate the offer! But it is not as robust and it is more expensive and misses a lot of benefits. Back in the 1990s I did FTP my website to a VPS after I graduated from Geocities. Google Cloud charges based on CPU used. Thus my servers have no traffic, they cost less than a $1/month. If they have traffic, they are still cost effective. https://web3dsurvey.com https://web3dsurvey.com has about 500,000 hits per month and it costs me $4/month to run both the Remix web server and the Fastify API server. Details here: https://x.com/benhouston3d/status/1840811854911668641 https://x.com/benhouston3d/status/1840811854911668641 Also it will autoscale under load. Thus when one of my posts was briefly the top story on Hacker News last month, Google Cloud Run added more instances to my server to handle the load (because I do not run my personal site behind a CDN, it cost too much, I prefer to pay $1/month for hosting.) Also deploying Docker containers that build on Github Actions CI in a few minutes is a great automated experience. I do also use Google services like Cloud Storage, Firestore, BigQuery etc. And it is easier to just run it on GCP infrastructure for speed. I also have to version various tools that get installed in the docker like Blender, Chromium, etc. This is the perfect use case for Docker. I feel this is pretty close to optimal. Fast, cheap, scalable, automated and robust.
candiddevmike 2y ago
You know that Cloud Run is effectively a Kubernetes PaaS, right?
- richards 2y ago
  Google employee here. Not the case. Cloud Run doesn't run on Kubernetes. It supports the Knative interface which is an OSS project for Kubernetes-based serverless. But Cloud Run is a fully managed service that sits directly atop Borg (https://cloud.google.com/run/docs/securing/security https://cloud.google.com/run/docs/securing/security).
  - dboreham 2y ago
    Parent said "effectively", which it appears you confirm.
    - bhouston 2y ago
      But it skips out on the Kubernetes part, which is the important part. :)
- bhouston 2y ago
  > You know that Cloud Run is a Kubernetes PaaS, right? Yup. Isn't it Knative Serving or a home grown Google alternative to it? https://knative.dev/docs/serving/ https://knative.dev/docs/serving/ The key is I am not managing Kubernetes and I am not paying for it - it is a fool's errand, and incredibly rarely needed. Who cares what is underneath the simple Cloud Run developer UX? What matters for me is cost, simplicity, speed and understandability. You get that with Cloud Run, and you don't with Kubernetes.
  - davewritescode 2y ago
    I haven't looked at Cloud Run pricing but running Kubernetes in the cloud is pretty cheap these days and my experience with solutions like Cloud Run in the past is that they end up becoming expensive. Kubernetes can be as complex or as expensive as you'd like but it's also fairly possible to run a pretty bulletproof simple Kube cluster.
deleted 2y ago
[deleted]
kbolino 2y ago
As far as I can tell, there actually is no AWS equivalent to GCP Cloud Run. The closest equivalents I know of are ECS on Fargate, which is more like managed Kubernetes except without Kubernetes compatibility or modern features, or AppRunner, which is closer in concept but also sorely lacking in comparable features.
Imustaskforhelp 2y ago
wow very very interesting. I think we can discuss about it on hours. 1.) What would you think of things like hetzner / linode / digitalocean (if stable work exists) 2.) What do you think of https://sst.dev/ https://sst.dev/ or https://encore.dev/ https://encore.dev/ ? (They support rather easier migration) 3.) Could you please indicate the split of that 1 million $ in devops time and google cloud costs unnecessarily & were there some outliers (like oh our intern didn't add this specific variable and this misconfigured cloud and wasted 10k on gcloud oops! or was it , that bandwidth causes this much more in gcloud (I don't think latter to be the case though)) Looking forward to chatting with you!

elcomet 2y ago

Aren't you afraid of being now stuck with GCP?

bhouston 2y ago
It is just a bunch of docker containers. Some run in tasks and some run as auto-scaling services. Would probably take a week to switch to AWS as there are equivalent managed services there. But this is really a spurious concern. I myself used to care about it years ago. But in practice, rarely do people switch between cloud providers because the incremental benefits are minor, they are nearly equivalent, there is nothing much to be gained by moving from one to the other unless politics are involved (e.g. someone high up wants a specific provider.)
- spwa4 2y ago
  How does the orchestration work? How do you share storage? How do the docker containers know how to find each other? How does security work? I feel like Kubernetes' downfall, for me, is the number of "enterprise" features it (got convinced into) supporting and enterprise features doing what they do best: turning the simplest of operations into a disaster.
  - bhouston 2y ago
    > How does the orchestration work? Github Actions CI. Take this and make a few more dependencies and a matrix strategy and you are good to go: https://github.com/bhouston/template-typescript-monorepo/blob/main/.github/workflows/ci.yml https://github.com/bhouston/template-typescript-monorepo/blo... For dev environments, you can add post-fixes to the services based on branches. > How do you share storage? I use managed DBs and Cloud Storage for shared storage. I think that provisioning your own SSDs/HDs to the cloud is indicative of an anti-pattern in your architecture. > How do the docker containers know how to find each other? I try to avoid too much communication between services directly, rather try to go through pub-sub or similar. But you can set up each service with a domain name and access them that way. With https://web3dsurvey.com https://web3dsurvey.com, I have an api on https://api.web3dsurvey.com https://api.web3dsurvey.com and then a review environment (connected to the main branch) with https://preview.web3dsurvey.com https://preview.web3dsurvey.com / https://api.preview.web3dsurvey.com https://api.preview.web3dsurvey.com. > How does security work? You can configure Cloud Run services to be internal only and not to accept outside connections. Otherwise one can just use JWT or whatever is normal on your routes in your web server.
    - dthg 2y ago
      > But you can set up each service with a domain name and access them that way. Are you using Cloud Run domain mappings for this or something else? I have been converging on a similar stack, but trying to avoid using a load balancer in an effort to keep fixed costs low.
      - bhouston 2y ago
        Yup domain mappings for now. There is some label support in Cloud Run but I haven’t explored it yet. You can also get the automatic domain name for a service via the cloud run tools. Yeah I definitely want to also avoid a load balancer or gateway or end points as well for cost purposes.
sofixa 2y ago
One of Cloud Run's main advantages is that it's literally just telling it how to run containers. You could run those same containers in OpenFaaS, Lambda, etc relatively easily.

rglover 2y ago

What stack are you deploying?

bhouston 2y ago
Stuff like this, just at larger scale: https://github.com/bhouston/template-typescript-monorepo https://github.com/bhouston/template-typescript-monorepo This is my living template of best practices.
- rglover 2y ago
  I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts.
  - Imustaskforhelp 2y ago
    yeh I have same thoughts , also if possible , bun can also reduce memory usage in very very basic scenarios https://www.youtube.com/watch?v=yJmyYosyDDM https://www.youtube.com/watch?v=yJmyYosyDDM Or just https://github.com/mightymoud/sidekick https://github.com/mightymoud/sidekick or coolify or dokku or dockify , like there are million of such things , oh just remembered kamala deploy from DHH and docker swarm IIRC (though people have seemed to forget docker swarm !) I like this idea very much !
  - Imustaskforhelp 2y ago
    there was some recent HN post which showed that they didn't even use docker but rather there was some other mechanism and it was so so simple , I really enjoyed that article
  - bhouston 2y ago
    > I'd investigate getting a build out to Node.js (looks like you already have this) and then just doing a simple SCP of the build to a VPS. From there, just use a systemd script to handle startup/restart on errors. For logging, something like the Winston package does the trick. If you want some guidance, shoot me an email (in profile). You can run most stuff for peanuts. I appreciate the offer! But it is not as robust and it is more expensive and misses a lot of benefits. Back in the 1990s I did FTP my website to a VPS after I graduated from Geocities. Google Cloud charges based on CPU used. Thus my servers have no traffic, they cost less than a $1/month. If they have traffic, they are still cost effective. https://web3dsurvey.com https://web3dsurvey.com has about 500,000 hits per month and it costs me $4/month to run both the Remix web server and the Fastify API server. Details here: https://x.com/benhouston3d/status/1840811854911668641 https://x.com/benhouston3d/status/1840811854911668641 Also it will autoscale under load. Thus when one of my posts was briefly the top story on Hacker News last month, Google Cloud Run added more instances to my server to handle the load (because I do not run my personal site behind a CDN, it cost too much, I prefer to pay $1/month for hosting.) Also deploying Docker containers that build on Github Actions CI in a few minutes is a great automated experience. I do also use Google services like Cloud Storage, Firestore, BigQuery etc. And it is easier to just run it on GCP infrastructure for speed. I also have to version various tools that get installed in the docker like Blender, Chromium, etc. This is the perfect use case for Docker. I feel this is pretty close to optimal. Fast, cheap, scalable, automated and robust.

candiddevmike 2y ago

You know that Cloud Run is effectively a Kubernetes PaaS, right?

richards 2y ago
Google employee here. Not the case. Cloud Run doesn't run on Kubernetes. It supports the Knative interface which is an OSS project for Kubernetes-based serverless. But Cloud Run is a fully managed service that sits directly atop Borg (https://cloud.google.com/run/docs/securing/security https://cloud.google.com/run/docs/securing/security).
- dboreham 2y ago
  Parent said "effectively", which it appears you confirm.
  - bhouston 2y ago
    But it skips out on the Kubernetes part, which is the important part. :)
bhouston 2y ago
> You know that Cloud Run is a Kubernetes PaaS, right? Yup. Isn't it Knative Serving or a home grown Google alternative to it? https://knative.dev/docs/serving/ https://knative.dev/docs/serving/ The key is I am not managing Kubernetes and I am not paying for it - it is a fool's errand, and incredibly rarely needed. Who cares what is underneath the simple Cloud Run developer UX? What matters for me is cost, simplicity, speed and understandability. You get that with Cloud Run, and you don't with Kubernetes.
- davewritescode 2y ago
  I haven't looked at Cloud Run pricing but running Kubernetes in the cloud is pretty cheap these days and my experience with solutions like Cloud Run in the past is that they end up becoming expensive. Kubernetes can be as complex or as expensive as you'd like but it's also fairly possible to run a pretty bulletproof simple Kube cluster.

deleted 2y ago

[deleted]

kbolino 2y ago

As far as I can tell, there actually is no AWS equivalent to GCP Cloud Run. The closest equivalents I know of are ECS on Fargate, which is more like managed Kubernetes except without Kubernetes compatibility or modern features, or AppRunner, which is closer in concept but also sorely lacking in comparable features.

Imustaskforhelp 2y ago

wow very very interesting. I think we can discuss about it on hours. 1.) What would you think of things like hetzner / linode / digitalocean (if stable work exists) 2.) What do you think of https://sst.dev/ https://sst.dev/ or https://encore.dev/ https://encore.dev/ ? (They support rather easier migration) 3.) Could you please indicate the split of that 1 million $ in devops time and google cloud costs unnecessarily & were there some outliers (like oh our intern didn't add this specific variable and this misconfigured cloud and wasted 10k on gcloud oops! or was it , that bandwidth causes this much more in gcloud (I don't think latter to be the case though)) Looking forward to chatting with you!