I put computing clusters on big UPS's to help keep them running during temporary power glitches. Make sure to put all the related equipment on UPS's also (swit

I put computing clusters on big UPS's to help keep them running during temporary power glitches. Make sure to put all the related equipment on UPS's also (switches, routers, hubs, KVM, monitor). Access control and scheduling computing time is important also, you don't want multiple people running computations at the same time because you might run out of RAM for example and in many experiments knowing the run time on a fully utilized CPU is important. Batch scripts can help; if you are running a series of computations that are not dependent on each other run them from a script, then if some fail it is easy to comment out the others and rerun the ones that failed. There are cluster management software programs that can automate much of this for you. Security measures are important, you don't want ransomware to hit during a run. Write good code for your computation that has sensible error handling so the computation can recover from errors, or at least give you data on what caused them. It can be difficult to debug long running software, it is similar to debugging server software. Sometimes you need the ability to only start generating logs after X hours of running or when the computation reaches a particular stage or to capture state continuously but only dump recent state history when an error occurs. Reliable air conditioning and power are essential. Physical access control to the hardware keeps anyone from bumping the hardware or accidentally unplugging it or otherwise messing with it. Sometimes it may make sense to have some spare machines that a task can migrate to if there is a hardware failure. Critical management functions you might run on more than one machine so that there is always at least one machine alive to control the systems. There are ways to lock circuit breakers so that service people can not turn them off (or on) by mistake without a key. Write your code to remove dependencies on an internet connection (e.g., use IP addresses instead of DNS, keep data local, have a local DNS server, send error reports using a local mail server that will keep retrying.) The computer location should keep insects, rodents, animals, mold, and water at bay. Vibration from nearby mechanical systems can be an issue (e.g., construction going on in a nearby room. Make sure facilities people have to get permission from you.) Remote access can let you check that things are running smoothly, a dashboard showing a variety of measures can be useful, though these are also a security risk. Keep automatic upgrades/updates turned off, only manually upgrade/update. Stop unnecessary services from running. Once you start thinking about it you'll find lots of things that could potentially go wrong; the world is extremely perverse and sneaky in this respect. If you find yourself running long term tasks over many years your list of mitigation measures will always be growing. Maybe you didn't consider the RF environment but a new local cell tower or radio tower starts messing with your systems. A pile driver a half mile away at a construction site can create strong vibrations and impulses in your computers. A solar storm can increase RF noise and reduce the quality of the AC power. Earthquakes happen. Med labs can have radiation sources. Fire is destructive, fire suppression systems can destroy your computers even if there is no fire. Someone may fire a gun through the walls if the public can get close to your computer room. Wireless networking is a security risk.