3 ms·
Here's one example from my experience: There are many MySQL tools which are able to automate the process of failing over between a master and a slave. Github i
by falcolas 11y ago
Here's one example from my experience:
There are many MySQL tools which are able to automate the process of failing over between a master and a slave. Github initially chose a solution which would trigger this failover automatically, when a master was unresponsive for a particular amount of time. One bad query came in, and the tool did it's job by failing over from the master to the slave. Here's the problem - the query was automatically retried against the slave, causing it to fail over back to the master. Repeatedly.
The solution? Page when MySQL became unresponsive, let a human initiate the failover, and let the tools handle the failover itself (which is a very complex dance to perform manually). I think they eventually moved to a model where it would fail over once per few hours, and alert if the condition occurred again.
- MikeTLive 11y agoThe same questions a human will ask can be encoded into your admin automation. It is only as powerful or accurate as the process allows. If you don't already have a procedural flowchart build one. Have the human document the questions asked and answers followed.
- falcolas 11y agoOnly if you can foresee the problems and solutions to encode into the system. If you miss one, the system will fall down or make the wrong decision.