24 ms·

Thankfully we've never had the need for such complexity and are happy with our current GitHub Actions > Docker Compose > GCR > SSH solution [1] we're using to d

by mythz 3y ago

Thankfully we've never had the need for such complexity and are happy with our current GitHub Actions > Docker Compose > GCR > SSH solution [1] we're using to deploy 50+ Docker Containers. Requires no infrastructure dependencies, stateless deployment scripts checked into the same Repo as Project and after GitHub Organization is setup (4 secrets) and deployment server has Docker compose + nginx-proxy installed, deploying an App only requires 1 GitHub Action Secret, as such it doesn't get any simpler for us and we'll look to continue to use this approach for as long as we can. [1] https://servicestack.net/posts/kubernetes_not_required https://servicestack.net/posts/kubernetes_not_required

seabrookmx 3y ago
I used to do something similar at a previous company and this works well if you don't have to worry about scaling. YAGNI principal and all that. When you run hundreds of containers for different workloads, k8s bin packing and autoscaling (both on the pod and node level) tips the balance in my experience.
- mythz 3y ago
  Yeah if we ever need to autoscale then I can see Kubernetes being useful, but I'd be surprised if this a problem most companies face. Even when working at StackOverflow (serving 1B+ pages, 55TB /mo [1]) did we need any autoscaling solution, it ran great on a handful of fixed servers. Although they were fairly beefy bare metal servers which I'd suspect would require significantly more VMs if it was to run on the Cloud. [1] https://stackexchange.com/performance https://stackexchange.com/performance
  - swozey 3y ago
    I was a k8s contrib since 2015, version 1.1. I even worked at Rancher and Google Cloud. If you don't need absolutely granular control over a PAAS/SAAS (complex networking w/ circuit breaking yadda yadda, deep stack tracing, vms controlled by k8s (kubevirt etc), multi-tenancy in cpu or gpu) you don't need k8s and will absolutely flourish using a container solution like ECS. Use fargate and arm64 containers and you will save an absolute fortune. I dropped our AWS bill from $350k/mo to around $250k converting our largest apps to arm from x86. GKE is IMO the best k8s solution PAAS wise that exists, but quite frankly few companies need that much control and granularity in their infrastructure. My entire infrastructure now is AWS ECS and it autoscales and I literally never, ever, ever have had to troubleshoot it outside of my own configuration mishaps. I NEVER get on call alerts. I'm the Staff SRE at my corp.
    - walth 3y ago
      Can completely agree. We went k8s several years ago and we battle the complexity continually. Want to change the pod spec to alter a variable under Vitess? That will be restarting several thousands of pods. DNS lookups failing as IPVS tuple collision? API server memory balooning to OOM due to watch/list cm storm? It all basically boils down to several FTE servicing K8s instead of building our platform. And on call is a nightmare.

by mythz 3y ago

seabrookmx 3y ago

I used to do something similar at a previous company and this works well if you don't have to worry about scaling. YAGNI principal and all that. When you run hundreds of containers for different workloads, k8s bin packing and autoscaling (both on the pod and node level) tips the balance in my experience.

mythz 3y ago
Yeah if we ever need to autoscale then I can see Kubernetes being useful, but I'd be surprised if this a problem most companies face. Even when working at StackOverflow (serving 1B+ pages, 55TB /mo [1]) did we need any autoscaling solution, it ran great on a handful of fixed servers. Although they were fairly beefy bare metal servers which I'd suspect would require significantly more VMs if it was to run on the Cloud. [1] https://stackexchange.com/performance https://stackexchange.com/performance
- swozey 3y ago
  I was a k8s contrib since 2015, version 1.1. I even worked at Rancher and Google Cloud. If you don't need absolutely granular control over a PAAS/SAAS (complex networking w/ circuit breaking yadda yadda, deep stack tracing, vms controlled by k8s (kubevirt etc), multi-tenancy in cpu or gpu) you don't need k8s and will absolutely flourish using a container solution like ECS. Use fargate and arm64 containers and you will save an absolute fortune. I dropped our AWS bill from $350k/mo to around $250k converting our largest apps to arm from x86. GKE is IMO the best k8s solution PAAS wise that exists, but quite frankly few companies need that much control and granularity in their infrastructure. My entire infrastructure now is AWS ECS and it autoscales and I literally never, ever, ever have had to troubleshoot it outside of my own configuration mishaps. I NEVER get on call alerts. I'm the Staff SRE at my corp.
  - walth 3y ago
    Can completely agree. We went k8s several years ago and we battle the complexity continually. Want to change the pod spec to alter a variable under Vitess? That will be restarting several thousands of pods. DNS lookups failing as IPVS tuple collision? API server memory balooning to OOM due to watch/list cm storm? It all basically boils down to several FTE servicing K8s instead of building our platform. And on call is a nightmare.