Honestly we've had great luck with HA prometheus instances shipping data to Thanos. All in kubernetes so we get to use pre-built operators. We don't really have

Honestly we've had great luck with HA prometheus instances shipping data to Thanos. All in kubernetes so we get to use pre-built operators. We don't really have to touch it other than updates. We are a 3 man team and we are holding around ~4m active series with around a year of downsampled data in a storage bucket. Doesn't really cost us anything other than some servers on a cluster. We also use Loki for logs and Tempo for tracing in the same pattern. One of the reasons I like Kubernetes, it's rigid enough compared to the wild west of VM land, and utilizing operators and tooling can get you solutions that just work provided you have enough core knowledge to fill in blanks.