For those who don't want to read all 15 pages, this can be summed up as "automate using the Unix philosophy." If you are not familiar with the Unix philosophy,

For those who don't want to read all 15 pages, this can be summed up as "automate using the Unix philosophy." If you are not familiar with the Unix philosophy, here you go: https://en.wikipedia.org/wiki/Unix_philosophy https://en.wikipedia.org/wiki/Unix_philosophy Allow me to answer the original question in a simpler way. Q: Dear Peter: A few years ago we automated a major process in our system administration team. Now the system is impossible to debug. Nobody remembers the old manual process and the automation is beyond what any of us can understand. We feel like we've painted ourselves into a corner. Is all operations automation doomed to be this way? A: So you wanted to go to the moon, and you hired a blind retiring man to build a rocket ship with no blueprints using parts from a junkyard. Don't do that. Instead, build lots of stairs. Lots and lots of stairs. Operations automation is different than typical software engineering because it isn't a finished product. You could kind of compare it to a constantly evolving organism. It needs constant care and attention and special handling, or it will get sick and fall over. This is due to non-indempotent operations, entropy, and changing requirements. An operations automation tool is never really complete. Probably modified over time by multiple people, most of whom no longer work at your company, and the changes were never documented, and probably were mostly made up of edge-case exceptions to allow some team to get something done for a weekend deadline. And now you need to change or debug it without breaking anything. When you write automation tools or processes, they need to be so dead freaking simple that just looking at the source code explains everything about it. It needs to be incredibly simple to tie together with other tools, so when it becomes too horrible to maintain any longer, it can be rewritten in a weekend. And it needs to be fault-tolerant, in the sense that it can fail and continue working anyway. If you have to write a last-minute exception to fix something that will only last a week, you can literally fork the original automation task and make a new one called "THIS-IS-A-TEMP-HACK-fork-of-automation-task.pl", just to make sure nobody confuses it with the normal, not-hacked-up task. -- I once had to overhaul an old Ops tool that would simply generate a config file used to control the kickstart, post-install setup, and configuration management of thousands of servers. It was a 5,000 line Perl program with no subroutines. I re-wrote it in Perl in a modular fashion, using advanced OO techniques and all the fancyness I could cram into it to cover every corner case of how it would be used in the future. Then I rewrote it as lots of tiny shell scripts. Guess which one ended up working out better? At the end of the day, all you need is to use the simplest and most direct method to automate little things, and use the same approach for everything you touch. It will grow and change over time, but as long as you are consistent with making small, S I M P L E, modular things that do two things well (the original task, and compatibility with other programs), you should be fine. Oh - and if a given automated task takes longer than 20 seconds to explain to someone, it's too fucking complicated.