8 ms·

What does a modern metrics stack look like? There are so many words... Graphfana, Kibana, StatsD, Graphite, etc. Which bits should I choose and attach together?

Karunamon 11y ago
I've got to advise against Kibana/Elasticsearch until the company gets more mature. I was 100% on board with them up until the redesign. Not so much for the design (oh look it's white instead of black, who cares), but the way they've handled it subsequently. The 3.x -> 4.x transition for Kibana left a product that was missing really basic features (like, the ability to set graph colors for one - a bug/feature request that's been open since last january). As it stands, upgrading from Kibana 3 to Kibana 4 is a step backwards. You lose functionality rather than gaining it. They also decided to require a major version bump to Elasticsearch (to 2.x) with a point release of Kibana. I used to be super optimistic about the Elastic guys, but some of these decisions are just head-scratchingly awful.
- meowface 11y ago
  After many years of using Splunk at my job, switching to ELK for personal project use was quite a disappointment for me. Of course, ELK is free and Splunk is very expensive, but I was still surprised at the gap. This might be derailing the thread a bit, but is there any log management platform like ELK or Splunk that has an expressive and versatile query language like Splunk's? My biggest issue with ELK is that analytics is mostly expected to be done through Kibana's GUI, while with Splunk you can craft terse queries to do almost any sort of transformation and visualization imaginable. I don't like how ELK is so GUI-oriented.
  - glial 11y ago
    Not sure what kind of queries you want to do, but Elasticsearch has a fairly extensive Query DSL that'll let you do all sorts of aggregations: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... There is a Python implementation that makes creating complex queries pretty easy: https://github.com/elastic/elasticsearch-dsl-py https://github.com/elastic/elasticsearch-dsl-py I agree with the criticisms of Kibana, but I have had no problems querying Elasticsearch directly. It also supports scripted queries if the built-in aggregations aren't enough. Of course, then you have to build your own visualizations with the results...
    - meowface 11y ago
      I'm not sure I'd consider that a DSL. It's unwieldy to write a full JSON object for every one-off query I want to do. Using one of their examples, this: { "query": { "filtered": { "query": { "bool": { "must": [{"match": {"title": "python"}}], "must_not": [{"match": {"description": "beta"}}] } }, "filter": {"term": {"category": "search"}} } }, "aggs" : { "per_tag": { "terms": {"field": "tags"}, "aggs": { "max_lines": {"max": {"field": "lines"}} } } } } would be the following Splunk query: title=python description!=beta | stats max(lines) by tags It would be nice if there was some kind of query compiler that could generate ES JSON from an expressive query language.
      - illumen 11y ago
        The linked python dsl does pretty well. Not as nice as your reduced dsl but. s = Search(using=client, index="my-index") \ .filter("term", category="search") \ .query("match", title="python") \ .query(~Q("match", description="beta"))
      - deleted 11y ago
        [deleted]
      - michaelcampbell 11y ago
        https://github.com/NLPchina/elasticsearch-sql/ https://github.com/NLPchina/elasticsearch-sql/ does exactly that with a reasonably good subset of SQL and ES query language mixed in. It operates in 2 modes; in one it runs the query, in another it spits back out what the equivalent ES JSON query is. We use this as a quick prototyping tool and modify the ES query as needed, as most of us here still "think" in SQL for a lot of things.
  - krakensden 11y ago
    I know, right? I recently did a pilot project with a Splunk competitor called Jut, that was totally query language oriented. It was great, hugely refreshing after Kibana. They're planning on going open source, too, so maybe the language will even survive.
- frik 11y ago
  Kibana is in incredible heavy weight in comparision to other dashboards like e.g. Grafana. Kibana loads like 4MB+ of JS and data. It is designed for intranet usage where your Elastic Search cluster is near you.
- realharo 11y ago
  There are also some serious unresolved issues in the current versions of the ELK stack bits, such as https://github.com/elastic/kibana/issues/5170 https://github.com/elastic/kibana/issues/5170, https://github.com/elastic/kibana/issues/1961 https://github.com/elastic/kibana/issues/1961, https://github.com/elastic/logstash/issues/3440 https://github.com/elastic/logstash/issues/3440. Even with all that though, it's still an incredibly useful tool for log analysis. It's a much different scenario than what one would use statsd, graphite, etc. for.
- fapjacks 11y ago
  Kibana 4.x is atrocious. In addition to generally being as heavy as a ton of bricks.
  - 23david 11y ago
    I've been using the latest Grafana release as an alternative to Kibana for visualizations. It now supports Elasticsearch as a db backend. Seems much faster and more stable than Kibana 4.
    - Karunamon 11y ago
      Belated thanks for pointing this out! Grafana is now getting some serious play here, and it would get more if I could figure out how to make the term searches work :)
sagichmal 11y ago
Prometheus is really the answer here. http://prometheus.io http://prometheus.io
- 5h 11y ago
  > What does a modern metrics stack look like? There are so many words... Graphfana, Kibana, StatsD, Graphite, Prometheus etc. Which bits should I choose and attach together?
  - bbrazil 11y ago
    Kibana is more logging, but Promethues covers the StatsD and Graphite requirements out of the box. Graphfana is one visualisation option to go with that (http://www.robustperception.io/setting-up-grafana-for-prometheus/ http://www.robustperception.io/setting-up-grafana-for-promet...).
- XorNot 11y ago
  Prometheus with grafana for visualization. I've got some ideas about trying to run automated correlation queries for peaks and the like but they're probably above my skill level. I have had some decent luck with sending syslog into mtail and counting generic word events like 'error' and 'warning' as a way to do "something might be wrong" alerting.
Mahn 11y ago
Grafana + InfluxDB are the simplest to get up and running. Literally two binaries that you download and run, that's it. Can't beat the simplicity of this set up and you can get a lot done with it, if you are just starting your metrics infrastructure I can recommend it.

Karunamon 11y ago

I've got to advise against Kibana/Elasticsearch until the company gets more mature. I was 100% on board with them up until the redesign. Not so much for the design (oh look it's white instead of black, who cares), but the way they've handled it subsequently. The 3.x -> 4.x transition for Kibana left a product that was missing really basic features (like, the ability to set graph colors for one - a bug/feature request that's been open since last january). As it stands, upgrading from Kibana 3 to Kibana 4 is a step backwards. You lose functionality rather than gaining it. They also decided to require a major version bump to Elasticsearch (to 2.x) with a point release of Kibana. I used to be super optimistic about the Elastic guys, but some of these decisions are just head-scratchingly awful.

meowface 11y ago
After many years of using Splunk at my job, switching to ELK for personal project use was quite a disappointment for me. Of course, ELK is free and Splunk is very expensive, but I was still surprised at the gap. This might be derailing the thread a bit, but is there any log management platform like ELK or Splunk that has an expressive and versatile query language like Splunk's? My biggest issue with ELK is that analytics is mostly expected to be done through Kibana's GUI, while with Splunk you can craft terse queries to do almost any sort of transformation and visualization imaginable. I don't like how ELK is so GUI-oriented.
- glial 11y ago
  Not sure what kind of queries you want to do, but Elasticsearch has a fairly extensive Query DSL that'll let you do all sorts of aggregations: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... There is a Python implementation that makes creating complex queries pretty easy: https://github.com/elastic/elasticsearch-dsl-py https://github.com/elastic/elasticsearch-dsl-py I agree with the criticisms of Kibana, but I have had no problems querying Elasticsearch directly. It also supports scripted queries if the built-in aggregations aren't enough. Of course, then you have to build your own visualizations with the results...
  - meowface 11y ago
    I'm not sure I'd consider that a DSL. It's unwieldy to write a full JSON object for every one-off query I want to do. Using one of their examples, this: { "query": { "filtered": { "query": { "bool": { "must": [{"match": {"title": "python"}}], "must_not": [{"match": {"description": "beta"}}] } }, "filter": {"term": {"category": "search"}} } }, "aggs" : { "per_tag": { "terms": {"field": "tags"}, "aggs": { "max_lines": {"max": {"field": "lines"}} } } } } would be the following Splunk query: title=python description!=beta | stats max(lines) by tags It would be nice if there was some kind of query compiler that could generate ES JSON from an expressive query language.
    - illumen 11y ago
      The linked python dsl does pretty well. Not as nice as your reduced dsl but. s = Search(using=client, index="my-index") \ .filter("term", category="search") \ .query("match", title="python") \ .query(~Q("match", description="beta"))
    - deleted 11y ago
      [deleted]
    - michaelcampbell 11y ago
      https://github.com/NLPchina/elasticsearch-sql/ https://github.com/NLPchina/elasticsearch-sql/ does exactly that with a reasonably good subset of SQL and ES query language mixed in. It operates in 2 modes; in one it runs the query, in another it spits back out what the equivalent ES JSON query is. We use this as a quick prototyping tool and modify the ES query as needed, as most of us here still "think" in SQL for a lot of things.
- krakensden 11y ago
  I know, right? I recently did a pilot project with a Splunk competitor called Jut, that was totally query language oriented. It was great, hugely refreshing after Kibana. They're planning on going open source, too, so maybe the language will even survive.
frik 11y ago
Kibana is in incredible heavy weight in comparision to other dashboards like e.g. Grafana. Kibana loads like 4MB+ of JS and data. It is designed for intranet usage where your Elastic Search cluster is near you.
realharo 11y ago
There are also some serious unresolved issues in the current versions of the ELK stack bits, such as https://github.com/elastic/kibana/issues/5170 https://github.com/elastic/kibana/issues/5170, https://github.com/elastic/kibana/issues/1961 https://github.com/elastic/kibana/issues/1961, https://github.com/elastic/logstash/issues/3440 https://github.com/elastic/logstash/issues/3440. Even with all that though, it's still an incredibly useful tool for log analysis. It's a much different scenario than what one would use statsd, graphite, etc. for.
fapjacks 11y ago
Kibana 4.x is atrocious. In addition to generally being as heavy as a ton of bricks.
- 23david 11y ago
  I've been using the latest Grafana release as an alternative to Kibana for visualizations. It now supports Elasticsearch as a db backend. Seems much faster and more stable than Kibana 4.
  - Karunamon 11y ago
    Belated thanks for pointing this out! Grafana is now getting some serious play here, and it would get more if I could figure out how to make the term searches work :)

sagichmal 11y ago

Prometheus is really the answer here. http://prometheus.io http://prometheus.io

5h 11y ago
> What does a modern metrics stack look like? There are so many words... Graphfana, Kibana, StatsD, Graphite, Prometheus etc. Which bits should I choose and attach together?
- bbrazil 11y ago
  Kibana is more logging, but Promethues covers the StatsD and Graphite requirements out of the box. Graphfana is one visualisation option to go with that (http://www.robustperception.io/setting-up-grafana-for-prometheus/ http://www.robustperception.io/setting-up-grafana-for-promet...).
XorNot 11y ago
Prometheus with grafana for visualization. I've got some ideas about trying to run automated correlation queries for peaks and the like but they're probably above my skill level. I have had some decent luck with sending syslog into mtail and counting generic word events like 'error' and 'warning' as a way to do "something might be wrong" alerting.

Mahn 11y ago

Grafana + InfluxDB are the simplest to get up and running. Literally two binaries that you download and run, that's it. Can't beat the simplicity of this set up and you can get a lot done with it, if you are just starting your metrics infrastructure I can recommend it.