I manage a 2.5 TB/day Splunk cluster at my current employer and can offer a few tips for making Splunk less painful to manage: - Make frequent visits to answer

I manage a 2.5 TB/day Splunk cluster at my current employer and can offer a few tips for making Splunk less painful to manage: - Make frequent visits to answers.splunk.com. It has a very active community, and I've frequently been able to type "how do I do X in Splunk" into Google and found multiple answers on Splunk Answers. - Deployment Server. Make friends with it. In a perfect world, it should hold your configurations for all Indexers, Heavy Forwarders, and Forwarders. If you're having to populate $SPLUNK_HOME/etc/system/local/ yourself, you're doing it wrong. - Make friends with the "splunk btool config_file_name list --debug" command. That makes it dead simple to know which configuration options a Splunk install is running. Append "| grep -v system/default" on the end of that command to filter out the defaults and you'll more easily see which of your options are being used. - If you have the cash, attend Splunk Conf and load your schedule up with presentations. It's worth every penny. Hope that helps.