We don't care so much about resiliency, we do backup the prometheus folder using the snapshot api. There are a couple of articles about sharding and federation

We don't care so much about resiliency, we do backup the prometheus folder using the snapshot api. There are a couple of articles about sharding and federation with prometheus, dunno if they existed when you tried it. For us our problems are usually local to a datacenter. Having a dropdown where you can pick the datacenter has proven good enough. It is unlikely that we have a global issue in a service. Sorry if unclear but we have our own datacenters, our prometheus VMs are essentially free in the grand scheme of things considering the number of compute we have.