12 ms·

I see a lot of comments to the effect that this is a complicated setup, especially for one person. I really think most people need to re-evaluate their stacks w

I see a lot of comments to the effect that this is a complicated setup, especially for one person. I really think most people need to re-evaluate their stacks without all of the marketing of cloud providers and gigascalers. This setup is very similar to the startup that I run. We have used k8s from day one, despite plenty of warnings when we did, but it has been rock solid for 3 years. We also decided to run Postgres and Redis on k8s using Patroni + Sentinel and the LVM storage class. This means we get highly performant local storage on each node and we push HA down to the application. Was there a learning curve? Yes, it took a solid week to figure out what we were doing. And we've had the odd issue here and there, but our uptime has exceeded what we would have had with a cookie cutter AWS setup, and our costs have been a fraction (including human capital costs).

malux85 2y ago
I am solo builder and runner of https://atomictesselator.com https://atomictesselator.com and I have used k8s from day one. I also agree with you, it's been rock solid, and the ability to horizontally scale so easily is great. I dont even know which physical machines in my rack the jobs/pods are running on, I just have a high level dashboard (lens) and I can see the CPU+GPU utilization hit 100% as everything self balances, it's great, this has been very helpful scaling my GPU workload which has increased a lot recently
- FlyingSnake 2y ago
  Looks like the setup didn’t survive a death hug from HN, the site’s not loading for me.
  - azornathogron 2y ago
    It's just a typo in the URL. It should be https://atomictessellator.com https://atomictessellator.com (two Ls)
- deleted 2y ago
  [deleted]
marcinzm 2y ago
Same. The simple fact that every PR can deploy a full stack, including RDS and managed Redis if desired, automatically in it's own namespace with proper DNS pointing to services is a massive win for us. As in, if you label the PR then it all happens automatically and then it all shuts down automatically when the PR is closed.
- williamdclt 2y ago
  Do you have any resource to how to set that up? I’m interested!
  - marcinzm 2y ago
    Should make a blog post but in short: - Every service is deployed via a Helm chart and using containers - GitHub actions build the container and deploy the helm chart Some of the details that matter: - ACK is used to create AWS services (RDS, Redis, etc.) via Helm charts (we also have a container option for helm charts as it's faster and less expensive) - External Secrets is used to create secrets in the new namespace and also do things like generate RDS passwords - ExternalDNS creates DNS entries in Route53 from the Ingress objects - Namespace name is generated automatically from the branch name - Docker images use the git hash for the tag Some things that are choices: - Monorepo although each service aims to be as self-contained as possible. - Docker context is the git root as this allows for a service to include shared libraries from the repo when creating a container. This is for case where we break the previous rule.
CuriouslyC 2y ago
K8s is overkill though. Google cloud run gives you the good bits with less work and it's still reasonably priced. Azure container apps in theory should be similarly good, but it's kind of buggy and poorly engineered.
- qeternity 2y ago
  Why do you say it's overkill? Why do people think k8s (or k3s) is some impossible monster? Have you deployed and managed k8s before? What did you find more complicated than learning the ins and outs of the cloud variants?
  - CuriouslyC 2y ago
    I'm not saying it's an impossible monster but I can setup and deploy a project from scratch with cloud run in ~5 gcloud cli commands while being somewhat protected from my own ignorance.
    - mitjam 2y ago
      Caution: Working with CloudRun with ignorance on any level can lead to very large bills: https://news.ycombinator.com/item?id=25372336 https://news.ycombinator.com/item?id=25372336 - The article discussed there is a good cautionary tale: https://archive.is/D5qc8 https://archive.is/D5qc8 - I prefer to explore and experiment on a less scalable solution which is usually also less scalable in terms of costs.
  - dinkleberg 2y ago
    In my experience, people who think k8s is really hard to user are either: a) someone who had k8s forced on them without getting a chance to learn it (or had no interest in the first place) and was stuck with its complexity, or b) has never actually used k8s and is just parroting what they've heard. The former situation does suck. K8s is amazing and once you understand it, it is easy to work with. But if you haven't learned the concepts and core resources, it will definitely appear as a black box with a ton of complexity. But I think for many of us who have used Kubernetes a lot, it is a no-brainer in a lot of situations as it doesn't matter which cloud provider you're using (for the most part), you get a common and familiar interface.
FlyingSnake 2y ago
For a startup that was generating revenue and got acquired, this is par for the course. What are people expecting instead?

malux85 2y ago

I am solo builder and runner of https://atomictesselator.com https://atomictesselator.com and I have used k8s from day one. I also agree with you, it's been rock solid, and the ability to horizontally scale so easily is great. I dont even know which physical machines in my rack the jobs/pods are running on, I just have a high level dashboard (lens) and I can see the CPU+GPU utilization hit 100% as everything self balances, it's great, this has been very helpful scaling my GPU workload which has increased a lot recently

FlyingSnake 2y ago
Looks like the setup didn’t survive a death hug from HN, the site’s not loading for me.
- azornathogron 2y ago
  It's just a typo in the URL. It should be https://atomictessellator.com https://atomictessellator.com (two Ls)
deleted 2y ago
[deleted]

marcinzm 2y ago

Same. The simple fact that every PR can deploy a full stack, including RDS and managed Redis if desired, automatically in it's own namespace with proper DNS pointing to services is a massive win for us. As in, if you label the PR then it all happens automatically and then it all shuts down automatically when the PR is closed.

williamdclt 2y ago
Do you have any resource to how to set that up? I’m interested!
- marcinzm 2y ago
  Should make a blog post but in short: - Every service is deployed via a Helm chart and using containers - GitHub actions build the container and deploy the helm chart Some of the details that matter: - ACK is used to create AWS services (RDS, Redis, etc.) via Helm charts (we also have a container option for helm charts as it's faster and less expensive) - External Secrets is used to create secrets in the new namespace and also do things like generate RDS passwords - ExternalDNS creates DNS entries in Route53 from the Ingress objects - Namespace name is generated automatically from the branch name - Docker images use the git hash for the tag Some things that are choices: - Monorepo although each service aims to be as self-contained as possible. - Docker context is the git root as this allows for a service to include shared libraries from the repo when creating a container. This is for case where we break the previous rule.

CuriouslyC 2y ago

K8s is overkill though. Google cloud run gives you the good bits with less work and it's still reasonably priced. Azure container apps in theory should be similarly good, but it's kind of buggy and poorly engineered.

qeternity 2y ago
Why do you say it's overkill? Why do people think k8s (or k3s) is some impossible monster? Have you deployed and managed k8s before? What did you find more complicated than learning the ins and outs of the cloud variants?
- CuriouslyC 2y ago
  I'm not saying it's an impossible monster but I can setup and deploy a project from scratch with cloud run in ~5 gcloud cli commands while being somewhat protected from my own ignorance.
  - mitjam 2y ago
    Caution: Working with CloudRun with ignorance on any level can lead to very large bills: https://news.ycombinator.com/item?id=25372336 https://news.ycombinator.com/item?id=25372336 - The article discussed there is a good cautionary tale: https://archive.is/D5qc8 https://archive.is/D5qc8 - I prefer to explore and experiment on a less scalable solution which is usually also less scalable in terms of costs.
- dinkleberg 2y ago
  In my experience, people who think k8s is really hard to user are either: a) someone who had k8s forced on them without getting a chance to learn it (or had no interest in the first place) and was stuck with its complexity, or b) has never actually used k8s and is just parroting what they've heard. The former situation does suck. K8s is amazing and once you understand it, it is easy to work with. But if you haven't learned the concepts and core resources, it will definitely appear as a black box with a ton of complexity. But I think for many of us who have used Kubernetes a lot, it is a no-brainer in a lot of situations as it doesn't matter which cloud provider you're using (for the most part), you get a common and familiar interface.

FlyingSnake 2y ago

For a startup that was generating revenue and got acquired, this is par for the course. What are people expecting instead?