Graphite was the biggest changes I had to adjust to when I moved to my current job and will be the biggest thing I push on all future jobs. We have stupid bash

Graphite was the biggest changes I had to adjust to when I moved to my current job and will be the biggest thing I push on all future jobs. We have stupid bash scripts that run on cron every minute. The scripts collect all sorts of system data. Things like load average, iowait, data in, data out. They also collect all sorts of info about our custom network daemon. They take all of this info and using netcat, echo and sometimes statsd shove this all into graphite. Having all of this info gives us several advantages. When we notice an issue we are able to track it back in time. We notice an increase in CPU usage at the same time as a sharp increase in data in? Our daemon preformed a bunch of "foo" actions at the same time, well obviously something about "foo" action is killing the box. In addition to historical data we also can do A/B testing from an operations perspective. Take a couple of boxes split into two groups, tune sysctl "y" to x value on all of group A boxes. Turn up the same load and see how sysctl "y" affects performance of the box. Because graphite has such a simple api for importing data we can easily expand our scripts push them to the boxes and we have new sources of data. Little fuss.