3 ms·

It's always nice to see new products in the DevOps space, but be careful not to re-invent the wheel if you do this kind of stuff as the open source world is com

It's always nice to see new products in the DevOps space, but be careful not to re-invent the wheel if you do this kind of stuff as the open source world is coming on leaps and bounds. LogStash, ElasticSearch and Kibana are a great open source stack for log management. StatsD and Graphite are nice tools for metric tracking and visualization. There are lots of open source dashboard offerings which combined with a bit of scripting can get you far. You are also spoiled for choice with SAAS monitoring stuff such as NewRelic and Server Density, even if the OP isn't a fan of cloud based tools.

KyleBrandt 13y ago
I did an experiment with logstash, elasticsearch, and Kibana for our HAProxy logs. The default with logstash was to store each field, and then the whole text so stuff go quite large for our web logs. Also the Kibana interface was pretty buggy. Parsing our logs (~2k entries) a second doesn't work well with Regex so in our version we do a bunch of substring stuff. I'm excited about Kibana/ES for the rest of our logs over with their recent hire. When I looked at StatsD and Graphite last time I didn't really see an API. I really like the model of data being queryable and returning nice serialized format like json (like OpenTSDB does). I'm also not that fond of the "many files" model and the automatic data summarization as it ages (it does save space, but makes forecasting difficult as it can skew data).
- zwily 13y ago
  We're parsing 2k lines/sec with logstash using regexes. We scaled it out to 6 logstash processes across 2 nodes. They pull off a shared redis queue and then insert the results directly into elasticsearch. (That said, I'd like to configure our apache logs to just output the json logstash expects.) Graphite has a very simple and powerful JSON API [1]. Any graph URL can include &format=json and you'll get back the raw JSON values for the datapoints. I haven't used OpenTSDB yet - I'm curious how its API is better. You get to choose the levels of summarization. If you want to keep 1 second intervals for a year, you're welcome to do so. [1] http://graphite.readthedocs.org/en/0.9.12/render_api.html http://graphite.readthedocs.org/en/0.9.12/render_api.html

KyleBrandt 13y ago

I did an experiment with logstash, elasticsearch, and Kibana for our HAProxy logs. The default with logstash was to store each field, and then the whole text so stuff go quite large for our web logs. Also the Kibana interface was pretty buggy. Parsing our logs (~2k entries) a second doesn't work well with Regex so in our version we do a bunch of substring stuff. I'm excited about Kibana/ES for the rest of our logs over with their recent hire. When I looked at StatsD and Graphite last time I didn't really see an API. I really like the model of data being queryable and returning nice serialized format like json (like OpenTSDB does). I'm also not that fond of the "many files" model and the automatic data summarization as it ages (it does save space, but makes forecasting difficult as it can skew data).

zwily 13y ago
We're parsing 2k lines/sec with logstash using regexes. We scaled it out to 6 logstash processes across 2 nodes. They pull off a shared redis queue and then insert the results directly into elasticsearch. (That said, I'd like to configure our apache logs to just output the json logstash expects.) Graphite has a very simple and powerful JSON API [1]. Any graph URL can include &format=json and you'll get back the raw JSON values for the datapoints. I haven't used OpenTSDB yet - I'm curious how its API is better. You get to choose the levels of summarization. If you want to keep 1 second intervals for a year, you're welcome to do so. [1] http://graphite.readthedocs.org/en/0.9.12/render_api.html http://graphite.readthedocs.org/en/0.9.12/render_api.html