Once Splunk was about to break the bank, we abandoned it and started looking for something in the open-source world. We've toyed with and pretty much failed us

Once Splunk was about to break the bank, we abandoned it and started looking for something in the open-source world. We've toyed with and pretty much failed using Graylog2. Although it has been coming along steadily in features and stability, we just found that the interface-although pretty-was not intuitive to us: lots of links and multi-click scenarios to get to what you want; and creating filters and streams was difficult and prone to failure. After watching a couple of very compelling presentations by Jordan Sissel (Logstash founder), we decided to test it out. Once I realized that creating a filter (Grok rocks!) that searched for a term and reorganized the log to my liking only took a couple hours, I was sold. Another selling point for us was that Logstash has over 2 dozen ways to suck logs in, including the usual suspects - syslog, files, tcp, udp and *mq. You can also perform a bunch of log parsing on the client (i.e. the servers with the logs) before sending them to your central ELK server/cluster. At the end of the day, there is nothing magical about any of these systems. You alone know your logs best and have to figure out how to read/parse/search them. Our switch to Logstash from Graylog2 was our failing, not Graylog2's.