We run about 10 Kubernetes clusters, and each cluster comes with our company's 'kernel'. That kernel includes an HA Prometheus deployment of two replicas and al

We run about 10 Kubernetes clusters, and each cluster comes with our company's 'kernel'. That kernel includes an HA Prometheus deployment of two replicas and all the monitoring components that to with it. The 2TB of data is what we have stored across all those Prometheus instances. We use thanos as the entry point that Grafana speaks to, so you get aggregated results. Thanos as a querier is very simple to setup, and is very low maintenance. We have intended to configure long term storage using the GCS backend for years now, but sadly this project always ended up losing to other (genuinely!) more impactful work. We hope to do this within the next 6 months though, and reckon the project will take about 2 weeks of our teams time. For the monitoring angle then, I can recommend Prometheus and thanos as very easy systems to configure. Even for a small team with no prior experience, you'll probably have a good time. The one to watch out for is Elasticsearch, as that is a fundamentally more complex system in which you plan to store much more data. Loki looks much easier to setup and benefits from the Grafana ecosystem integration, if you're looking for a shorter/cheaper/less featureful option.