3 ms·

Thanks, great to see some numbers as well. My initial draft of this stated that we are using Prometheus/Grafana + ELK across the company, although its still be

Thanks, great to see some numbers as well. My initial draft of this stated that we are using Prometheus/Grafana + ELK across the company, although its still being standardised and turned from individuals creating their own deployments to a proper managed, strategic system with SLAs/documentation. We're looking at the feature lists of SaaS offerings like Datadog, Logz.io, but I think we're too big to get a sensible price. That said, I don't think we're big enough to justify something like Thanos that turns Prometheus from a really simple application, into a big time investment. Out of interest, are you storing 2TB of Prometheus data in a single instance? Or spread across multiple?

lawrjone 6y ago
We run about 10 Kubernetes clusters, and each cluster comes with our company's 'kernel'. That kernel includes an HA Prometheus deployment of two replicas and all the monitoring components that to with it. The 2TB of data is what we have stored across all those Prometheus instances. We use thanos as the entry point that Grafana speaks to, so you get aggregated results. Thanos as a querier is very simple to setup, and is very low maintenance. We have intended to configure long term storage using the GCS backend for years now, but sadly this project always ended up losing to other (genuinely!) more impactful work. We hope to do this within the next 6 months though, and reckon the project will take about 2 weeks of our teams time. For the monitoring angle then, I can recommend Prometheus and thanos as very easy systems to configure. Even for a small team with no prior experience, you'll probably have a good time. The one to watch out for is Elasticsearch, as that is a fundamentally more complex system in which you plan to store much more data. Loki looks much easier to setup and benefits from the Grafana ecosystem integration, if you're looking for a shorter/cheaper/less featureful option.
- tikkabhuna 6y ago
  Interesting that you find Thanos simple to setup and low maintenance. One vendor we spoke to suggested that their other customers found Thanos complex. I'll make a note to keep it in consideration. You comment on standardising the logging schema is a great idea as well. I'll circulate that around. Thanks for your help!

lawrjone 6y ago

We run about 10 Kubernetes clusters, and each cluster comes with our company's 'kernel'. That kernel includes an HA Prometheus deployment of two replicas and all the monitoring components that to with it. The 2TB of data is what we have stored across all those Prometheus instances. We use thanos as the entry point that Grafana speaks to, so you get aggregated results. Thanos as a querier is very simple to setup, and is very low maintenance. We have intended to configure long term storage using the GCS backend for years now, but sadly this project always ended up losing to other (genuinely!) more impactful work. We hope to do this within the next 6 months though, and reckon the project will take about 2 weeks of our teams time. For the monitoring angle then, I can recommend Prometheus and thanos as very easy systems to configure. Even for a small team with no prior experience, you'll probably have a good time. The one to watch out for is Elasticsearch, as that is a fundamentally more complex system in which you plan to store much more data. Loki looks much easier to setup and benefits from the Grafana ecosystem integration, if you're looking for a shorter/cheaper/less featureful option.

tikkabhuna 6y ago
Interesting that you find Thanos simple to setup and low maintenance. One vendor we spoke to suggested that their other customers found Thanos complex. I'll make a note to keep it in consideration. You comment on standardising the logging schema is a great idea as well. I'll circulate that around. Thanks for your help!