At one company, we used Grafana Cloud for the full monitoring stack. They charge by unique Prometheus series for metrics. I wrote a rather small Go API to allow

At one company, we used Grafana Cloud for the full monitoring stack. They charge by unique Prometheus series for metrics. I wrote a rather small Go API to allow users to access some otherwise hidden data and added rate-limiting because the data itself was large. To figure out costs, I wanted to add monitoring to the handlers, so I added a middleware that caught all requests and logged things like the request path, response time, etc. Sounds perfectly fine until you realize the internet is a vast space for people constantly scraping. I too left it over the weekend and came back to 70k unique series in our cloud account, pushing the bill well over $1k. What's worse is that Grafana is kind enough to not charge for these spikes, if you catch them before 48hrs. I caught it approx 50 hours later. Like the OP, though, Grafana was nice enough to make it fall off after I explained the situation. Lesson learned!