4 ms·

Basically doing this for a small startup - there are some complexities around autoscaling task queues with gpus and whatnot, but the heart of it is on a single

Basically doing this for a small startup - there are some complexities around autoscaling task queues with gpus and whatnot, but the heart of it is on a single VM (nginx, webapp, postgres, redis). We're b2b, so there's very little traffic anyway. The additional benefit is devs can run all the same stuff on a Linux laptop (or Linux VM on some other platform) - and everyone can have their own VM in the cloud if they like to demo or test stuff using all the same setup. Bootstrapping a new system is checking in their ssh key and running a shell script. Easy to debug, not complex or expensive, and we could vertically scale it all quite a ways before needing to scale horizontally. It's not for everyone, but seed stage and earlier - totally appropriate imo.

teaearlgraycold 2y ago
I did this type of setup but without even redis. Postgres can do anything.
- mattbillenstein 2y ago
  True, I use it mainly for a few convenience things - holding ephemeral monitoring data, distributed locks, redis streams for some pub/sub stuff, sorted sets can be handy - things I could do in Postgres, but are a bit simpler in Redis.
normie3000 2y ago
I love the simplicity of this approach. In your setup, how do you track config and updates of your VMs?
- clvx 2y ago
  I like this but one of the issues with this approach is if no Docker images like traditional configuration management tool, you are going for a world of pain. Docker and Docker images have tons of best practices already defined for plenty of use cases. If it's already containerized; then, jumping to any orchestrator that supports OCI images is more about adjusting the business to a new set of operations.
- mdaniel 2y ago
  Like everything, it's context dependent, but wowzers my life has improved so much since I got on board the Flatcar or Bottlerocket train of immutable OS. Flatcar (née CoreOS) does ship with docker but is still mostly a general purpose OS but Bottlerocket is about as "cloud native" as it comes, shipping with kubelet and even the host processes run in containers. For my purposes (being a k8s fanboy) that's just perfect since it's one less bootstrapping step I need to take on my own Both are Apache 2 and the Flatcar folks are excellent to work with https://github.com/flatcar/Flatcar#readme https://github.com/flatcar/Flatcar#readme https://github.com/bottlerocket-os#bottlerocket https://github.com/bottlerocket-os#bottlerocket
  - thebeardisred 2y ago
    As usual, I'm stoked to see I'm not the only one using Flatcar. :)
  - mattbillenstein 2y ago
    Sure, but again, complexity - stuff people have to learn/maintain/upgrade, etc. ymmv Running and configuring VMs isn't hard to do correctly, it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system.
    - mdaniel 2y ago
      > it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system. Yup, and I'm glad your experience has been different from mine but mine has been that tired and stressed people are anything but disciplined, so nipping a few "I'll just apt-get ..." in the bud goes a long way. So does Reverse Uptime (or its friend, Chaos Engineering)
- mattbillenstein 2y ago
  I have a custom deployment system which idempotently configures an Ubuntu LTS VM. All the config templates are checked into source control. I don't configure anything by hand - it's either handled in this thing or via a small user-data script run at provisioning time.
mdaniel 2y ago
> Bootstrapping a new system is checking in their ssh key and running a shell script. If it interests you, both major git hosts (and possibly all of them) have and endpoint to map a username to their already registered ssh keys: https://github.com/mdaniel.keys https://github.com/mdaniel.keys https://gitlab.com/mdaniel.keys https://gitlab.com/mdaniel.keys It's one level of indirection away from "check in a public key" in that the user can rotate their own keys without needing git churn Also, and I recognize this is departing quite a bit from what you were describing, ssh key leases are absolutely awesome because it addresses the offboarding scenario much better than having to reconcile evicting those same keys: https://github.com/hashicorp/vault/blob/v1.12.11/website/content/docs/secrets/ssh/signed-ssh-certificates.mdx https://github.com/hashicorp/vault/blob/v1.12.11/website/con... and while digging up that link I also discovered that Vault will allegedly do single-use passwords, too <https://github.com/hashicorp/vault/blob/v1.12.11/website/content/docs/secrets/ssh/one-time-ssh-passwords.mdx https://github.com/hashicorp/vault/blob/v1.12.11/website/con...>, but since I am firmly in the "PasswordLogin no" camp, caveat emptor with that one
- mattbillenstein 2y ago
  Yeah, I've used the github ssh key thing before, but never heard of key leases - will take a look. Thx!

teaearlgraycold 2y ago

I did this type of setup but without even redis. Postgres can do anything.

mattbillenstein 2y ago
True, I use it mainly for a few convenience things - holding ephemeral monitoring data, distributed locks, redis streams for some pub/sub stuff, sorted sets can be handy - things I could do in Postgres, but are a bit simpler in Redis.

normie3000 2y ago

I love the simplicity of this approach. In your setup, how do you track config and updates of your VMs?

clvx 2y ago
I like this but one of the issues with this approach is if no Docker images like traditional configuration management tool, you are going for a world of pain. Docker and Docker images have tons of best practices already defined for plenty of use cases. If it's already containerized; then, jumping to any orchestrator that supports OCI images is more about adjusting the business to a new set of operations.
mdaniel 2y ago
Like everything, it's context dependent, but wowzers my life has improved so much since I got on board the Flatcar or Bottlerocket train of immutable OS. Flatcar (née CoreOS) does ship with docker but is still mostly a general purpose OS but Bottlerocket is about as "cloud native" as it comes, shipping with kubelet and even the host processes run in containers. For my purposes (being a k8s fanboy) that's just perfect since it's one less bootstrapping step I need to take on my own Both are Apache 2 and the Flatcar folks are excellent to work with https://github.com/flatcar/Flatcar#readme https://github.com/flatcar/Flatcar#readme https://github.com/bottlerocket-os#bottlerocket https://github.com/bottlerocket-os#bottlerocket
- thebeardisred 2y ago
  As usual, I'm stoked to see I'm not the only one using Flatcar. :)
- mattbillenstein 2y ago
  Sure, but again, complexity - stuff people have to learn/maintain/upgrade, etc. ymmv Running and configuring VMs isn't hard to do correctly, it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system.
  - mdaniel 2y ago
    > it just takes discipline to never "hack it in the moment" - or if you do, can that change in your config system. Yup, and I'm glad your experience has been different from mine but mine has been that tired and stressed people are anything but disciplined, so nipping a few "I'll just apt-get ..." in the bud goes a long way. So does Reverse Uptime (or its friend, Chaos Engineering)
mattbillenstein 2y ago
I have a custom deployment system which idempotently configures an Ubuntu LTS VM. All the config templates are checked into source control. I don't configure anything by hand - it's either handled in this thing or via a small user-data script run at provisioning time.

mdaniel 2y ago

> Bootstrapping a new system is checking in their ssh key and running a shell script. If it interests you, both major git hosts (and possibly all of them) have and endpoint to map a username to their already registered ssh keys: https://github.com/mdaniel.keys https://github.com/mdaniel.keys https://gitlab.com/mdaniel.keys https://gitlab.com/mdaniel.keys It's one level of indirection away from "check in a public key" in that the user can rotate their own keys without needing git churn Also, and I recognize this is departing quite a bit from what you were describing, ssh key leases are absolutely awesome because it addresses the offboarding scenario much better than having to reconcile evicting those same keys: https://github.com/hashicorp/vault/blob/v1.12.11/website/content/docs/secrets/ssh/signed-ssh-certificates.mdx https://github.com/hashicorp/vault/blob/v1.12.11/website/con... and while digging up that link I also discovered that Vault will allegedly do single-use passwords, too <https://github.com/hashicorp/vault/blob/v1.12.11/website/content/docs/secrets/ssh/one-time-ssh-passwords.mdx https://github.com/hashicorp/vault/blob/v1.12.11/website/con...>, but since I am firmly in the "PasswordLogin no" camp, caveat emptor with that one

mattbillenstein 2y ago
Yeah, I've used the github ssh key thing before, but never heard of key leases - will take a look. Thx!