#6 -- you should check out nagios. We have many thousands of checks running on all aspects of our infrastructure. Allows for email, text messages, etc notificat

#6 -- you should check out nagios. We have many thousands of checks running on all aspects of our infrastructure. Allows for email, text messages, etc notifications on OK, WARN, and CRITICAL events. There are plenty of plugins already available but if you want something custom create it... it's free and open-source. For example, we have checks to see if there are fan, drive, power supply, etc, failures within systems. We also monitor main power feeds, UPS, ventilation, etc on site infrastructure. You can also monitor/graph/check history of websites and their load times. EDIT: you might also want to check out ganglia from the OS side. This doesn't allow notifications (as far as I know) like nagios but you can instantly see what your resources are being used on in larger machine clusters.