3 ms·
Let's determine what is the "high cardinality" logs. Grafana Loki puts log messages with the same set of labels into a single "stream". For example, all the log
by valyala 3y ago
Let's determine what is the "high cardinality" logs. Grafana Loki puts log messages with the same set of labels into a single "stream". For example, all the log messages with the labels {job="nginx", instance="host123"} go into a single stream. Loki stores every stream in a separate set of files and maintains in-memory index per each stream. So big number of streams in Loki lead to "high cardinality" issues such as performance degradation, high memory usage and increased storage load. That's why Loki developers recommend avoiding labels with big number of unique values such as ip, user_id, trace_id, etc.
I'm unsure what does "high cardinality" mean for ElasticSearch case. As I know it happily accepts log messages with high-cardinality labels such as ip, user_id or trace_id.
I don't know whether OpenObserve has any issues related to high cardinality.
P.S. I'm, as VictoriaMetrics core developer, confident that the upcoming log storage and analytics solution from VictoriaMetrics - VictoriaLogs - will be free from Loki-like high cardinality issues.