35 ms·
While logs and metrics have common parts, they are quite different beasts. It is quite expensive producing and storing logs for frequently changing metrics such
by valyala 7y ago
While logs and metrics have common parts, they are quite different beasts. It is quite expensive producing and storing logs for frequently changing metrics such as requests rate or response latency on highly loaded system with millions of rps.
I'd recommend:
- Storing logs in high-performance systems such as cLoki [1], which give enough flexibility to generate arbitrary metrics from high log volumes in real time.
- Storing metrics in high-performance time series database optimized for high volumes of data points (trillions) and high number of time series (aka high cardinality). An example of such a TSDB is VictoriaMetrics [2].
Implementing dashboards with time-based correlation between metrics and logs based on shared attributes (labels) such as datacenter, cluster, instance, job, subsystem, etc [3].
[1] https://github.com/lmangani/cLoki https://github.com/lmangani/cLoki
[2] https://github.com/VictoriaMetrics/VictoriaMetrics/ https://github.com/VictoriaMetrics/VictoriaMetrics/
[3] https://grafana.com/blog/2019/05/06/how-loki-correlates-metrics-and-logs----and-saves-you-money/ https://grafana.com/blog/2019/05/06/how-loki-correlates-metr...