2 ms·
> also important to rate-limit remediations and repairs as a safety net to prevent bugs in the code from mass draining and unprovisioning, which can result in
by londons_explore 5y ago
> also important to rate-limit remediations and repairs as a safety net to prevent bugs in the code from mass draining and unprovisioning, which can result in service outages if not handled properly.
This is the most important bit for someone reimplementing this...
Never let automation 'run wild' - always have a maximum number of machines per second it can act on, and a maximum percentage of the fleet unhealthy to allow it to continue taking things out of service.
- WJW 5y agoI agree wholeheartedly. Computers can do the wrong thing at incredible speed sometimes. This reminds me of the anecdote that nuclear submarine designers knowingly don't automate some things because if the automation would fail then it would cause the loss of the entire sub quickly enough that the situation is beyond salvage by the time humans can react. Instead, they have an extensive checklist and do all procedures with 2 people and a third over an audio line. Even if it does go wrong, it will probably go wrong slowly enough that a concerted damage control effort from the rest of the crew can contain the problem.