3 ms·
Personally, given the multi-day outage, I think I would just restore everything to a separate system, and then only point the affected customers to this new sys
by ksala_ 5y ago
Personally, given the multi-day outage, I think I would just restore everything to a separate system, and then only point the affected customers to this new system. Take the hit of having two fully separated system initially, and work on reconciling them after without having to worry about the on-going outage.
I wonder if they're not doing this due to some tech limitations, to avoid taking the financial cost of running two systems, or to avoid having to reconcile the systems.
- bigtones 5y agoThat's a really good idea.
- mandevil 5y agoAt a big multi-tenancy company I used to work at, the problem would have been the accessory machines: we had something like 15-20 different machines around the main DB and API machines, running cron jobs, terminating SSL connections, load balancing, sending alerts to us and customer emails out, etc. And while the backing up and failing over on DB and API machines was a well documented, thoroughly tested process... the other machines were all custom jobs that were very poorly documented, with who knows what scripts running on them, that might or might not be important. Trying to replicate all of that during an emergency would have been a challenge. For just this sort of problem, we actually had three DB servers running all the time: active, passive, and hour behind with the ability to break hour behind's copying of the write-ahead log of active as the DBA's secret weapon for just this problem. If all customers had accidentally lost an hours worth of data it would have been embarrassing, but much less than completely shutting out hundreds of paying customers for two weeks, I think?