Well, I can manage a system running hundreds or thousands of heterogeneous stateless nodes consistently from an image essentially simply managed "by hand" (mayb

Well, I can manage a system running hundreds or thousands of heterogeneous stateless nodes consistently from an image essentially simply managed "by hand" (maybe with judicious help from things like package auto-updates). Otherwise I could add something like Ansible with its configuration -- probably rather larger -- similarly managed by hand. Obviously the latter increases complexity and scope for problems; observationally it also decreases reliability and availability. Substantial experience says the simple way is the best to manage HPC systems, in particular, and it generalizes, like to managing teaching "clusters" with few resources[1]. There are minimal trades-off from just using things you already know -- package management, revision control, and initrd/pivot_root -- against using tools which are distinctly non-trivial, but make you look good for solving the extra problems and justify more manpower. I've used a cluster-building system from a maintainer and contributors with Ansible expertise and been unpleasantly surprised, if not as much as by several cluster vendors' products. Of course you automate things like monitoring, alerts, and some problem mitigation, for which simple configuration management frees effort. Accidental and essential complexity, someone once talked about. 1. http://workshop.openafs.org/afsbpw09/thu_2_1.html http://workshop.openafs.org/afsbpw09/thu_2_1.html