5 ms·

Does it put any limits on cardinality of metrics? Grafana cloud's offering was absolutely awful for my use cases. They charge per-series so if you have metrics

Does it put any limits on cardinality of metrics? Grafana cloud's offering was absolutely awful for my use cases. They charge per-series so if you have metrics with a "pod=..." label your prices go through the roof.

heliodor 6y ago
Plenty has been written about not using the server/container/pod id as a label because it leads to high cardinality which leads to poor performance (cost aside). Time series databases have been purpose-built for certain workloads and you can consider this their weakness.
- gravypod 6y ago
  Plenty has also been written about the bugs/issues that have cropped up that are only visible when inspecting what regions/nodes/cgroups an issue is coming from [0]. My use case wasn't exactly `pod=...` but it was very similar. It was more like `device=...`. Also, for a huge application, it's not uncommon to have 100s or even 1000s of metrics that are important to application health/performance. Constantly saying "do you really need X? It will cost us Y" will lead to an extremely under-monitored application. [0] - https://cloud.google.com/blog/products/management-tools/sre-keeps-digging-to-prevent-problems https://cloud.google.com/blog/products/management-tools/sre-...
  - heliodor 6y ago
    Plenty of companies run their own servers because cloud is too expensive at their scale. Same goes for metrics. It's a direct result of one-price-fits-all pricing models for software as well as pricing that is not correctly tied to value.
webo 6y ago
I like Weave Cloud’s Prometheus hosting model — it’s per host, which is predictable and forecastable.
kasey_junk 6y ago
Every managed metrics system will put a limit on cardinality because all mainstream available metrics systems cost more per cardinality to query and store. If they don’t limit that you can assume you or some other customer is going to use up the clusters resources and cause an outage. Like most metrics systems, under the covers in Prometheus each unique combination of dimensions is the same as a new metric line.
- deleted 6y ago
  [deleted]
valyala 6y ago
Grafana cloud sets high prices for high-cardinality metrics because the underlying system - Cortex - isn't optimized well for storing high number of unique time series. For example, it requires at least 15GB of RAM for processing a million of active time series [1]. This means high infrastructure costs, which increase pricing for end users. Other systems such as VictoriaMetrics require up to 15x lower RAM for the same metrics' cardinality [2]. [1] https://github.com/cortexproject/cortex/blob/67648aabae70f1948b93b90ee56a96c1d7558422/docs/guides/capacity-planning.md https://github.com/cortexproject/cortex/blob/67648aabae70f19... [2] https://victoriametrics.github.io/#capacity-planning https://victoriametrics.github.io/#capacity-planning

heliodor 6y ago

Plenty has been written about not using the server/container/pod id as a label because it leads to high cardinality which leads to poor performance (cost aside). Time series databases have been purpose-built for certain workloads and you can consider this their weakness.

gravypod 6y ago
Plenty has also been written about the bugs/issues that have cropped up that are only visible when inspecting what regions/nodes/cgroups an issue is coming from [0]. My use case wasn't exactly `pod=...` but it was very similar. It was more like `device=...`. Also, for a huge application, it's not uncommon to have 100s or even 1000s of metrics that are important to application health/performance. Constantly saying "do you really need X? It will cost us Y" will lead to an extremely under-monitored application. [0] - https://cloud.google.com/blog/products/management-tools/sre-keeps-digging-to-prevent-problems https://cloud.google.com/blog/products/management-tools/sre-...
- heliodor 6y ago
  Plenty of companies run their own servers because cloud is too expensive at their scale. Same goes for metrics. It's a direct result of one-price-fits-all pricing models for software as well as pricing that is not correctly tied to value.

webo 6y ago

I like Weave Cloud’s Prometheus hosting model — it’s per host, which is predictable and forecastable.

kasey_junk 6y ago

Every managed metrics system will put a limit on cardinality because all mainstream available metrics systems cost more per cardinality to query and store. If they don’t limit that you can assume you or some other customer is going to use up the clusters resources and cause an outage. Like most metrics systems, under the covers in Prometheus each unique combination of dimensions is the same as a new metric line.

deleted 6y ago
[deleted]

valyala 6y ago

Grafana cloud sets high prices for high-cardinality metrics because the underlying system - Cortex - isn't optimized well for storing high number of unique time series. For example, it requires at least 15GB of RAM for processing a million of active time series [1]. This means high infrastructure costs, which increase pricing for end users. Other systems such as VictoriaMetrics require up to 15x lower RAM for the same metrics' cardinality [2]. [1] https://github.com/cortexproject/cortex/blob/67648aabae70f1948b93b90ee56a96c1d7558422/docs/guides/capacity-planning.md https://github.com/cortexproject/cortex/blob/67648aabae70f19... [2] https://victoriametrics.github.io/#capacity-planning https://victoriametrics.github.io/#capacity-planning