Degraded modes of operation is one example of how to visualize masked errors. Another is to trigger an alarm on fallbacks. As a general reflection, many distri

Degraded modes of operation is one example of how to visualize masked errors. Another is to trigger an alarm on fallbacks. As a general reflection, many distributed system leave out the cause of their changes and only log actions. Instead of logging "new membership, new members are b,c,d" you are better of logging "node a has not responded to heartbeat in the last 30 seconds, considering it faulty". Following such a principle makes it much easier to spot masked bugs, since you can reason about the behaviour much better. Aggregating logs to a central location and being able to analyze global behaviour in retrospect is also a great feature.