3 ms·
Yes, I manually create and configure in the sense that I tweak a number in Terraform templates and manually run the ansible playbook for each new server. It's t
by erikrothoff 6y ago
Yes, I manually create and configure in the sense that I tweak a number in Terraform templates and manually run the ansible playbook for each new server. It's taken a lot of time to get to that level (I think keeping a setup of bash scripts would suffice in our case...)
We run a read-replica on every database, so in case a hardware error occurs on the main database we can manually switch it over. It might mean up to an hour of downtime if the worst happens. Some data loss is OK and can be solved with manual customer support most of the time. It's also a lot cost effective than working towards a 100% SLA.
Keeping the read-replicas alive is plenty pain enough! I can't imagine automating everything to auto-heal itself. (Sounds super fun though)
Codifying the setup for auto-scaling would be a massive undertaking. Each new change then requires destroying VMs and bringing up new ones. That would then require a k8s-like layer of infrastructure for secrets, DNS, service discovery, not relying on ephemeral storage (which is a lot faster than volumes/block storage).
I really love doing ops/devops, and would love to have the perfect setup which is 100% automatic and scaleable. Even now I have to stop myself from spending too much time scriptifying things that can just be run manually.
- saargrin 6y agoand hour of downtime and some data loss are not metrics acceptable to most businesses i know of and what if you have 10 customers joining every day? still gonna be running that ansible manually?
- erikrothoff 6y agoI guess you assumed 1 customer = 1 new server? For enterprise purposes where data siloes are important a different approach definitely makes sense. We have 300+ new users per day, so our manual system scales well.