Pitfalls off the top of my head: Detecting servers going down is not very simple. They don't just go down and become completely unresponsive. Sometimes they bec

Pitfalls off the top of my head: Detecting servers going down is not very simple. They don't just go down and become completely unresponsive. Sometimes they become slow, overloaded, generate incorrect responses, sometimes there are connectivity issues, packets get dropped. Sometimes connectivity issues are not globally visible, but affect only some of your users. The number of affected users might increase over time too without you detecting anything. All of the problems apply to monitoring nodes too. The simplest approach that more or less works is to use a single monitoring node close to your users and only make decisions when majority of servers are responsive. Once some server has more than one problem over a few tries, pull its record. Then put it back once it's been free of problems for some time. When updating DNS records automatically increase serial number of the zone. Make sure monitoring node retrieves that serial number from DNS servers on start, to survive own crashes.