3 ms·
I think this outage was made worse by them not being properly in a big cloud provider. In a cloud provider, having a few people working simultaneously on spinn
by londons_explore 5y ago
I think this outage was made worse by them not being properly in a big cloud provider.
In a cloud provider, having a few people working simultaneously on spinning up instances with different potential fixes, running different tests, and then directing all traffic to the first one that works properly is a viable path to a solution.
When you have your own hardware, you can really only try one thing at a time.
- KronisLV 5y ago> When you have your own hardware, you can really only try one thing at a time. How so? What would prevent you from hiring 5-10 people for Ops heavy stuff and getting a bit more hardware resources and doing those things in staging environments with load tests and whatnot? I mean, isn't that how you should do things, regardless of where your infra and software is?
- londons_explore 5y agoIf you own your own hardware, for a given service you probably have enough hardware for the production workload, plus maybe 50% more for dev, test, staging, experiments, etc. All those other environments will probably be scaled down versions. Sure, they can be used in an emergency situation, but they can't withstand the full production load, and anyway they're likely on a separate physical hardware and network (usually you want good isolation between production and test environments). If you use AWS, then you probably on average use the same day to day, but in an emergency you can spin up 5 versions of full production scale to test 5 things at once, and just edit a configfile to direct production traffic to any.