3 ms·
Article doesn't mention cardinality. And it's a key concept to understand how these systems differ. When I joined a company that used statsD for metrics and EL
by SideburnsOfDoom 7y ago
Article doesn't mention cardinality. And it's a key concept to understand how these systems differ.
When I joined a company that used statsD for metrics and ELK for logging, I was told the first rule of stats: you will not put a order id, customer id or the like in the stat name. It is OK to have a value with a small set of values (e.g. beta: true|false), but the order Id varies too much and will swamp the statsd server.
Formally: order id is high cardinality. A boolean is low cardinality. cardinality is the "number of elements of the set" - foor a boolean it's 2, for an "order state" enumeration it's a few, and for an order id it's unbounded, but realistically, it's "how many orders might we have in any 2 week window / other stat retention period"
A stats system like StatsD that keeps continuous averages, max, min, p90 etc cannot do this if there are lots of high-cardinality factors. There is only so much combined cardinality that they can handle.
No, Prometheus will not save you here (1)
Whereas a logging system such as an ELK stack stores each individual log entry in detail, and the more relevant unique ids you attach to it, the more informative it is. Cardinality is not problem. But pre-calculated stats across these fields are limited, and ad-hoc queries might be expensive to run.
You could, I suppose, start with a rich log entry and transform it into a metric by discarding some parts. But not the reverse.
1) https://prometheus.io/docs/practices/naming/ https://prometheus.io/docs/practices/naming/
> CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.
- mbar84 7y agoIndeed a very important caveat! Working in the ad-tech industry, we're tracking creative id, placement id, domain and more for each impression. We're lucky if each timeseries has 10 impressions at the hourly granularity.
- valyala 7y agoI'd recommend taking a look at ClickHouse for storing and analyzing billions of ad-tech raw events in real time. There is no need in grouping them beforehand, since ClickHouse is able to scan billions of rows per second per server. See https://clickhouse.yandex https://clickhouse.yandex