3 ms·
Self-healing system: increase cluster size and replace servers randomly. It works because it was a problem of threads occasionally entering an infinite loop but
by alecco 2y ago
Self-healing system: increase cluster size and replace servers randomly. It works because it was a problem of threads occasionally entering an infinite loop but not corrupting data. And the whole system can tolerate these kind of whole server crashes. IMHO an unusual combination of preconditions.
It's not explained why they couldn't write a monitor script instead to find servers having the issue and only killing those.
- 4star3star 2y agoI think they just needed a quick and dirty solution that was good enough for a few days. They figured that for 1% failure per hour, they needed to kill x processes every y minutes to keep ahead of the failures. I'm sure it would be much more efficient but also more complicated to try to target the specific failures, and the "good enough" solution was acceptable.