5 ms·

We run Prometheus with 60d retention (2TB), and an Elasticsearch cluster for logs (also 60d, about 30TB cluster) with Kibana on top. Prometheus is by far the c

We run Prometheus with 60d retention (2TB), and an Elasticsearch cluster for logs (also 60d, about 30TB cluster) with Kibana on top. Prometheus is by far the cheapest, both for infrastructure spend and for human cost. The only time we spend really working on Prometheus is configuring our service discovery, which is the shipping component you’d have with any monitoring tool. I estimate we spend about 1 developer day a month un Prometheus upkeep. Logging is much more fiddly, and required a large up front investment. Now it’s running though, we get a lot of value from our setup. It costs much more though, think about $10k/month in infrastructure spend and about 3 developer days a month to maintain. Grafana is effortless to run, really dead simple. You can consider this to be ‘free’, both for infra and maintenance. Hope that gives you a sense of this? I think we come out equivalent to managed services for cost when you account for infra and human time, but have far more flexibility in how we use the tools and develop the skills to properly leverage each product along the way.

atombender 6y ago
Same setup here. My company uses ELK, and I personally really dislike it. It's okay when it works — filtering logs with queries is the primary use case, and it's decent at it when it's not returning 503s. But it's also based on Elasticsearch, inheriting all its warts. I really wish someone would build a better competitor. Loki/Grafana isn't anywhere close yet. One of my pet peeves with ELK is how the indexer assumes all log entries have the same schema. So if one app has the "error" field as a string, and other logs it as an object, then ELK will reject the second one. It boggles my mind that someone thought this was a good technical design. It could easily suffix internal field names with its schema (type, analyzer, etc.) and then unify the fields at the UI level. But no, instead it discards log data. Sluggish performance (ELK requires enormous machine researches for no particular reason, though the JVM is a major driver) and lack of support for tailing are two other pet peeves. (I have many more.)
- lawrjone 6y ago
  Yep, as you point out, the idea of Elasticsearch having a uniform type for each field can get problematic. That can be fixed by: - logging to different indexes - preprocessing your logs so the keys have their own schema prefix, as you mention The way we've tackled this is to have an official company-wide logging schema. It's just a GitHub repo at gocardless/logging that has an exhaustive list of logging keys, with an explanation of what they should contain. This has the benefit of encouraging consistent logging practices over many teams, as well as improving the chance your logs will get indexed correctly. If the field type doesn't match, we won't index that field but it will appear in the _log field, where you can do a full text search as a fallback if you really needed to find your log. It's not perfect though, and I still hate the dynamic type assignment.
  - atombender 6y ago
    We log probably from 30+ services, plus third party apps. Standardizing on a schema is to some extent possible (but hard to actually enforce and test; many of our services are written in JS/TS, not Go), but not for third-party apps. Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes. But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.
    - axw 6y ago
      Yes, you can do this using Filebeat or in Elasticsearch itself using an Ingest node script processor. Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#indices-option-es https://www.elastic.co/guide/en/beats/filebeat/current/elast... (You can incorporate variables in the destination index; variables may refer to labels and other fields.) Ingest node: https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... (search for "_index")
tikkabhuna 6y ago
Thanks, great to see some numbers as well. My initial draft of this stated that we are using Prometheus/Grafana + ELK across the company, although its still being standardised and turned from individuals creating their own deployments to a proper managed, strategic system with SLAs/documentation. We're looking at the feature lists of SaaS offerings like Datadog, Logz.io, but I think we're too big to get a sensible price. That said, I don't think we're big enough to justify something like Thanos that turns Prometheus from a really simple application, into a big time investment. Out of interest, are you storing 2TB of Prometheus data in a single instance? Or spread across multiple?
- lawrjone 6y ago
  We run about 10 Kubernetes clusters, and each cluster comes with our company's 'kernel'. That kernel includes an HA Prometheus deployment of two replicas and all the monitoring components that to with it. The 2TB of data is what we have stored across all those Prometheus instances. We use thanos as the entry point that Grafana speaks to, so you get aggregated results. Thanos as a querier is very simple to setup, and is very low maintenance. We have intended to configure long term storage using the GCS backend for years now, but sadly this project always ended up losing to other (genuinely!) more impactful work. We hope to do this within the next 6 months though, and reckon the project will take about 2 weeks of our teams time. For the monitoring angle then, I can recommend Prometheus and thanos as very easy systems to configure. Even for a small team with no prior experience, you'll probably have a good time. The one to watch out for is Elasticsearch, as that is a fundamentally more complex system in which you plan to store much more data. Loki looks much easier to setup and benefits from the Grafana ecosystem integration, if you're looking for a shorter/cheaper/less featureful option.
  - tikkabhuna 6y ago
    Interesting that you find Thanos simple to setup and low maintenance. One vendor we spoke to suggested that their other customers found Thanos complex. I'll make a note to keep it in consideration. You comment on standardising the logging schema is a great idea as well. I'll circulate that around. Thanks for your help!
aprdm 6y ago
This also mirrors my experience. Both Prometheus & Grafana require very little effort. We also have a similar scale to you. I think ELK is the most challenging bit as you outlined yourself but is still very doable.

atombender 6y ago

Same setup here. My company uses ELK, and I personally really dislike it. It's okay when it works — filtering logs with queries is the primary use case, and it's decent at it when it's not returning 503s. But it's also based on Elasticsearch, inheriting all its warts. I really wish someone would build a better competitor. Loki/Grafana isn't anywhere close yet. One of my pet peeves with ELK is how the indexer assumes all log entries have the same schema. So if one app has the "error" field as a string, and other logs it as an object, then ELK will reject the second one. It boggles my mind that someone thought this was a good technical design. It could easily suffix internal field names with its schema (type, analyzer, etc.) and then unify the fields at the UI level. But no, instead it discards log data. Sluggish performance (ELK requires enormous machine researches for no particular reason, though the JVM is a major driver) and lack of support for tailing are two other pet peeves. (I have many more.)

lawrjone 6y ago
Yep, as you point out, the idea of Elasticsearch having a uniform type for each field can get problematic. That can be fixed by: - logging to different indexes - preprocessing your logs so the keys have their own schema prefix, as you mention The way we've tackled this is to have an official company-wide logging schema. It's just a GitHub repo at gocardless/logging that has an exhaustive list of logging keys, with an explanation of what they should contain. This has the benefit of encouraging consistent logging practices over many teams, as well as improving the chance your logs will get indexed correctly. If the field type doesn't match, we won't index that field but it will appear in the _log field, where you can do a full text search as a fallback if you really needed to find your log. It's not perfect though, and I still hate the dynamic type assignment.
- atombender 6y ago
  We log probably from 30+ services, plus third party apps. Standardizing on a schema is to some extent possible (but hard to actually enforce and test; many of our services are written in JS/TS, not Go), but not for third-party apps. Could one set up FileBeat or whatever to ingest into separate indexes based on some source label? Most of our logs are streamed from Kubernetes, so there is an application label we could use. Elasticsearch is pretty good at unioning a single query over multiple indexes. But it'd have to be an automatic mapping — I wouldn't want the ops team to have to maintain a mapping between apps and indexes that could (would!) get out of sync.
  - axw 6y ago
    Yes, you can do this using Filebeat or in Elasticsearch itself using an Ingest node script processor. Filebeat: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#indices-option-es https://www.elastic.co/guide/en/beats/filebeat/current/elast... (You can incorporate variables in the destination index; variables may refer to labels and other fields.) Ingest node: https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... (search for "_index")

tikkabhuna 6y ago

Thanks, great to see some numbers as well. My initial draft of this stated that we are using Prometheus/Grafana + ELK across the company, although its still being standardised and turned from individuals creating their own deployments to a proper managed, strategic system with SLAs/documentation. We're looking at the feature lists of SaaS offerings like Datadog, Logz.io, but I think we're too big to get a sensible price. That said, I don't think we're big enough to justify something like Thanos that turns Prometheus from a really simple application, into a big time investment. Out of interest, are you storing 2TB of Prometheus data in a single instance? Or spread across multiple?

lawrjone 6y ago
We run about 10 Kubernetes clusters, and each cluster comes with our company's 'kernel'. That kernel includes an HA Prometheus deployment of two replicas and all the monitoring components that to with it. The 2TB of data is what we have stored across all those Prometheus instances. We use thanos as the entry point that Grafana speaks to, so you get aggregated results. Thanos as a querier is very simple to setup, and is very low maintenance. We have intended to configure long term storage using the GCS backend for years now, but sadly this project always ended up losing to other (genuinely!) more impactful work. We hope to do this within the next 6 months though, and reckon the project will take about 2 weeks of our teams time. For the monitoring angle then, I can recommend Prometheus and thanos as very easy systems to configure. Even for a small team with no prior experience, you'll probably have a good time. The one to watch out for is Elasticsearch, as that is a fundamentally more complex system in which you plan to store much more data. Loki looks much easier to setup and benefits from the Grafana ecosystem integration, if you're looking for a shorter/cheaper/less featureful option.
- tikkabhuna 6y ago
  Interesting that you find Thanos simple to setup and low maintenance. One vendor we spoke to suggested that their other customers found Thanos complex. I'll make a note to keep it in consideration. You comment on standardising the logging schema is a great idea as well. I'll circulate that around. Thanks for your help!

aprdm 6y ago

This also mirrors my experience. Both Prometheus & Grafana require very little effort. We also have a similar scale to you. I think ELK is the most challenging bit as you outlined yourself but is still very doable.