I'm a big fan of Pallet for infrastructure management ( https://github.com/pallet/pallet https://github.com/pallet/pallet ). That's what we used for all our inf

I'm a big fan of Pallet for infrastructure management ( https://github.com/pallet/pallet https://github.com/pallet/pallet ). That's what we used for all our infrastructure on AWS at BackType, and my team has continued to use it to manage our machines within the Twitter datacenter. Pallet has a high learning curve, but it's worth it. Sam wrote the pallet-hadoop tool which can spin up Hadoop clusters at the click of a button ( https://github.com/pallet/pallet-hadoop https://github.com/pallet/pallet-hadoop ). Although if you're on AWS you're better off just using EMR. You don't need to use Java. I do everything in Clojure (using Cascalog and Storm's Clojure DSL).