We're parsing 2k lines/sec with logstash using regexes. We scaled it out to 6 logstash processes across 2 nodes. They pull off a shared redis queue and then ins

We're parsing 2k lines/sec with logstash using regexes. We scaled it out to 6 logstash processes across 2 nodes. They pull off a shared redis queue and then insert the results directly into elasticsearch. (That said, I'd like to configure our apache logs to just output the json logstash expects.) Graphite has a very simple and powerful JSON API [1]. Any graph URL can include &format=json and you'll get back the raw JSON values for the datapoints. I haven't used OpenTSDB yet - I'm curious how its API is better. You get to choose the levels of summarization. If you want to keep 1 second intervals for a year, you're welcome to do so. [1] http://graphite.readthedocs.org/en/0.9.12/render_api.html http://graphite.readthedocs.org/en/0.9.12/render_api.html