In support of ELK, I spent years glancing at all sorts of different options (fluentd, greylog, ELK, Splunk, logstash, loggly, ...). I had set up Elasticsearch

In support of ELK, I spent years glancing at all sorts of different options (fluentd, greylog, ELK, Splunk, logstash, loggly, ...). I had set up Elasticsearch but never got the parts working for log ingestion and groking. I got them loading but no groking. This was a "personal time" project at work and I just didn't have the time. Then we decided to wanted Apache logs to be centralized, so I set up an rsyslog server and fed the logs to that. Once I had that component running, I set up another ELK system and figured out the "groking", ended up using liblognorm. A few weeks later our Elasticsearch cluster had some problem and people had come to rely on it, so I started shoring up the deployment. Going from experiment mode to something more production ready. The stack I ended up with was rsyslog on the clients, shipping logs to the rsyslog server using RELP. The rsyslog server using omelasticsearch to send the logs to Elasticsearch, and liblognorm module in rsyslog to process the logs. liblognorm is kind of a pain in the ass, but it was built into rsyslog. If I were to do it again, I'd spend a bit more time figuring out if I could use grok instead.