5 ms·

Slightly off-topic, but how are users of Grafana and other monitoring tools justifying the investment? And I'm not necessarily talking about monetary amount, it

Slightly off-topic, but how are users of Grafana and other monitoring tools justifying the investment? And I'm not necessarily talking about monetary amount, its also people cost. What features are you looking for and how do you rank them?

syoc 6y ago
I use a telegraf, influxdb, grafana stack. I find the cost in maintainance and initial setup quite low. Telegraf is super easy, just uncomment the things you want it to collect from the config file. Influxdb is just adding the correct users and a database, never touched it since. Grafana can be a time sink if you want to bikeshed your dashboards, but there are a lot of pre-made ones that handle common usecases. It's really nice to have some metrics when for instance a service goes down. It's super easy to spot a OOM situation or other vertical scaling issues.
- tikkabhuna 6y ago
  Thanks. I completely agree with the bikeshedding. We're playing with Prometheus/Grafana + ELK, and being able to visualise all this data, its hard to work out what is useful and what is just fun to play with. I'm now wondering where the line is between necessary monitoring and bikeshedding? I could look at introducing distributed tracing, but will it actually add any meaningful value?
lawrjone 6y ago
We run Prometheus with 60d retention (2TB), and an Elasticsearch cluster for logs (also 60d, about 30TB cluster) with Kibana on top. Prometheus is by far the cheapest, both for infrastructure spend and for human cost. The only time we spend really working on Prometheus is configuring our service discovery, which is the shipping component you’d have with any monitoring tool. I estimate we spend about 1 developer day a month un Prometheus upkeep. Logging is much more fiddly, and required a large up front investment. Now it’s running though, we get a lot of value from our setup. It costs much more though, think about $10k/month in infrastructure spend and about 3 developer days a month to maintain. Grafana is effortless to run, really dead simple. You can consider this to be ‘free’, both for infra and maintenance. Hope that gives you a sense of this? I think we come out equivalent to managed services for cost when you account for infra and human time, but have far more flexibility in how we use the tools and develop the skills to properly leverage each product along the way.
- atombender 6y ago
  Same setup here. My company uses ELK, and I personally really dislike it. It's okay when it works — filtering logs with queries is the primary use case, and it's decent at it when it's not returning 503s. But it's also based on Elasticsearch, inheriting all its warts. I really wish someone would build a better competitor. Loki/Grafana isn't anywhere close yet. One of my pet peeves with ELK is how the indexer assumes all log entries have the same schema. So if one app has the "error" field as a string, and other logs it as an object, then ELK will reject the second one. It boggles my mind that someone thought this was a good technical design. It could easily suffix internal field names with its schema (type, analyzer, etc.) and then unify the fields at the UI level. But no, instead it discards log data. Sluggish performance (ELK requires enormous machine researches for no particular reason, though the JVM is a major driver) and lack of support for tailing are two other pet peeves. (I have many more.)
  - lawrjone 6y ago
    Yep, as you point out, the idea of Elasticsearch having a uniform type for each field can get problematic. That can be fixed by: - logging to different indexes - preprocessing your logs so the keys have their own schema prefix, as you mention The way we've tackled this is to have an official company-wide logging schema. It's just a GitHub repo at gocardless/logging that has an exhaustive list of logging keys, with an explanation of what they should contain. This has the benefit of encouraging consistent logging practices over many teams, as well as improving the chance your logs will get indexed correctly. If the field type doesn't match, we won't index that field but it will appear in the _log field, where you can do a full text search as a fallback if you really needed to find your log. It's not perfect though, and I still hate the dynamic type assignment.
    - atombender 6y ago
      We log probably from 30+ services, plus third party apps. Standardizing on a schema is to some extent possible (but hard to actually enforce and test; many of our services are written in JS/TS, not Go), but not for third-party apps. Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes. But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.
gjulianm 6y ago
I run Prometheus + Grafana on a handful of machines we manage, with different software and setups. The cost of setting up a Prometheus instance and monitoring something is not too high, but things around it depend on your actual setup - Service discovery can get tricky. In our case we manage the physical machines by ourselves so at the beginning we just had to write down static configs, although by now we have automatic discovery (took a few days of development). - Depending on what you want to monitor, you might need to write your own exporters to Prometheus. However, mtail [1] has been really useful to create metrics from logs without too much work. In any case, you'll have to put time in deploying and configuring those exporters. - Dashboards and alerts. There are dashboards for a lot of exporters, and there are collections of alerts too [2], but you will need to put time and effort in modifying/creating dashboards and writing down alerts. However, it's a productive effort because it helps in having a better understanding of which metrics are important and how do they relate to the workings of the software you use. Also, PromQL is a pretty nice query language for the purposes of Prometheus. - Notification integrations. In my case we had to put some time to properly configure a Microsoft Teams integration and a deadman switch channel, but in most cases it will be pretty straightforward. All in all, you'll need to invest some time in the integrations, but those are things that you need to do in any case. Prometheus itself is pretty easy to set up and maintain, and doesn't stand in your way. No tweaks, no undocumented settings, no bugs. I'm pretty happy in that regard, once you get it running you don't have to worry about it. Storage usage is pretty low even with a high amount of exporters and metrics per node, maybe around 10GB for 60 days of data of a single node? (I'm not sure because Prometheus does some compression and it's not exactly linear with the time or the number of nodes). And that relatively low investment pays off quickly. The machines we manage use various tools and programs to deal with quite a lot of data at high bandwidths, so performance problems and bugs can be difficult to debug. The Prometheus + Grafana setup has made several times easier the debugging of issues and performance problems, the alerting system helps us prevent outages and we have even discovered issues that were unknown to us. For me, the moment you manage machines with even just a little bit of complexity in the software or setup, it's already worth it to look into monitoring. 1: https://github.com/google/mtail https://github.com/google/mtail 2: https://awesome-prometheus-alerts.grep.to/ https://awesome-prometheus-alerts.grep.to/
deleted 6y ago
[deleted]

syoc 6y ago

I use a telegraf, influxdb, grafana stack. I find the cost in maintainance and initial setup quite low. Telegraf is super easy, just uncomment the things you want it to collect from the config file. Influxdb is just adding the correct users and a database, never touched it since. Grafana can be a time sink if you want to bikeshed your dashboards, but there are a lot of pre-made ones that handle common usecases. It's really nice to have some metrics when for instance a service goes down. It's super easy to spot a OOM situation or other vertical scaling issues.

tikkabhuna 6y ago
Thanks. I completely agree with the bikeshedding. We're playing with Prometheus/Grafana + ELK, and being able to visualise all this data, its hard to work out what is useful and what is just fun to play with. I'm now wondering where the line is between necessary monitoring and bikeshedding? I could look at introducing distributed tracing, but will it actually add any meaningful value?

lawrjone 6y ago

We run Prometheus with 60d retention (2TB), and an Elasticsearch cluster for logs (also 60d, about 30TB cluster) with Kibana on top. Prometheus is by far the cheapest, both for infrastructure spend and for human cost. The only time we spend really working on Prometheus is configuring our service discovery, which is the shipping component you’d have with any monitoring tool. I estimate we spend about 1 developer day a month un Prometheus upkeep. Logging is much more fiddly, and required a large up front investment. Now it’s running though, we get a lot of value from our setup. It costs much more though, think about $10k/month in infrastructure spend and about 3 developer days a month to maintain. Grafana is effortless to run, really dead simple. You can consider this to be ‘free’, both for infra and maintenance. Hope that gives you a sense of this? I think we come out equivalent to managed services for cost when you account for infra and human time, but have far more flexibility in how we use the tools and develop the skills to properly leverage each product along the way.

atombender 6y ago
Same setup here. My company uses ELK, and I personally really dislike it. It's okay when it works — filtering logs with queries is the primary use case, and it's decent at it when it's not returning 503s. But it's also based on Elasticsearch, inheriting all its warts. I really wish someone would build a better competitor. Loki/Grafana isn't anywhere close yet. One of my pet peeves with ELK is how the indexer assumes all log entries have the same schema. So if one app has the "error" field as a string, and other logs it as an object, then ELK will reject the second one. It boggles my mind that someone thought this was a good technical design. It could easily suffix internal field names with its schema (type, analyzer, etc.) and then unify the fields at the UI level. But no, instead it discards log data. Sluggish performance (ELK requires enormous machine researches for no particular reason, though the JVM is a major driver) and lack of support for tailing are two other pet peeves. (I have many more.)
- lawrjone 6y ago
  Yep, as you point out, the idea of Elasticsearch having a uniform type for each field can get problematic. That can be fixed by: - logging to different indexes - preprocessing your logs so the keys have their own schema prefix, as you mention The way we've tackled this is to have an official company-wide logging schema. It's just a GitHub repo at gocardless/logging that has an exhaustive list of logging keys, with an explanation of what they should contain. This has the benefit of encouraging consistent logging practices over many teams, as well as improving the chance your logs will get indexed correctly. If the field type doesn't match, we won't index that field but it will appear in the _log field, where you can do a full text search as a fallback if you really needed to find your log. It's not perfect though, and I still hate the dynamic type assignment.
  - atombender 6y ago
    We log probably from 30+ services, plus third party apps. Standardizing on a schema is to some extent possible (but hard to actually enforce and test; many of our services are written in JS/TS, not Go), but not for third-party apps. Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes. But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.

gjulianm 6y ago

I run Prometheus + Grafana on a handful of machines we manage, with different software and setups. The cost of setting up a Prometheus instance and monitoring something is not too high, but things around it depend on your actual setup - Service discovery can get tricky. In our case we manage the physical machines by ourselves so at the beginning we just had to write down static configs, although by now we have automatic discovery (took a few days of development). - Depending on what you want to monitor, you might need to write your own exporters to Prometheus. However, mtail [1] has been really useful to create metrics from logs without too much work. In any case, you'll have to put time in deploying and configuring those exporters. - Dashboards and alerts. There are dashboards for a lot of exporters, and there are collections of alerts too [2], but you will need to put time and effort in modifying/creating dashboards and writing down alerts. However, it's a productive effort because it helps in having a better understanding of which metrics are important and how do they relate to the workings of the software you use. Also, PromQL is a pretty nice query language for the purposes of Prometheus. - Notification integrations. In my case we had to put some time to properly configure a Microsoft Teams integration and a deadman switch channel, but in most cases it will be pretty straightforward. All in all, you'll need to invest some time in the integrations, but those are things that you need to do in any case. Prometheus itself is pretty easy to set up and maintain, and doesn't stand in your way. No tweaks, no undocumented settings, no bugs. I'm pretty happy in that regard, once you get it running you don't have to worry about it. Storage usage is pretty low even with a high amount of exporters and metrics per node, maybe around 10GB for 60 days of data of a single node? (I'm not sure because Prometheus does some compression and it's not exactly linear with the time or the number of nodes). And that relatively low investment pays off quickly. The machines we manage use various tools and programs to deal with quite a lot of data at high bandwidths, so performance problems and bugs can be difficult to debug. The Prometheus + Grafana setup has made several times easier the debugging of issues and performance problems, the alerting system helps us prevent outages and we have even discovered issues that were unknown to us. For me, the moment you manage machines with even just a little bit of complexity in the software or setup, it's already worth it to look into monitoring. 1: https://github.com/google/mtail https://github.com/google/mtail 2: https://awesome-prometheus-alerts.grep.to/ https://awesome-prometheus-alerts.grep.to/

deleted 6y ago

[deleted]