4 ms·

I'm seeing this "3 pillars of observability" framing as the main description these days, and I think it is nice breakdown of options. However, there is a common

I'm seeing this "3 pillars of observability" framing as the main description these days, and I think it is nice breakdown of options. However, there is a common problem I am curious how others are dealing with. All metrics systems I have seen don't handle high cardinality data like Ids well. Most SASS products base their pricing on unique metrics because each unique aggregation has a cost. In the 3 pillars model, companies like Datadog are pushing those high cardinality values into the log pillar, and this article seems to imply the same. However, logs are often unstructured. There are often tools to search through logs, find values within those logs, and I can even do aggregations on them. However, when you know ahead of time what you want to aggregate on, text logs are more brittle than simply defining an event in json or another structured format. This log tier quickly becomes an ad-hoc version of a data warehouse and I feel like there is a missing tier here where you would send structured data to a datastore to do aggregations on for observability purposes only. I know datadog supports parsing structured data in this way, but I'm curious what is a common solution to this. Is sending structured data through the "log" tier common, or are there structured event and reporting systems that are part of another system that is seldom discussed in this topic.

stingraycharles 6y ago
In our case, what we do is use Grafana Loki for logging, which emphasizes on a small amount of indexes, but great support for “sequential scans” of log files. We send the logs in semi-structured format (logfmt), and we have no real performance problems with this. The trick is that you want to optimize for “easy to use regexes”, which we did not have with json. Logfmt gives this to us, while still being easy to read with the eyes. One of the key decisions we made, however, was to not use any cloud provider for this and run it ourselves. It gave us much more freedom with various ways to interact with our metrics / data, without the risk of getting huge bills (I’m looking at you Datadog, your billing practices are terrible).
throwaway2380 6y ago
Traces can be a good place to handle high cardinality observability data. Traces essentially are a particular form of structured logs. Traces support arbitrary tags, and trace analysis systems can be optimized for quick aggregations over high cardinality fields.

stingraycharles 6y ago

In our case, what we do is use Grafana Loki for logging, which emphasizes on a small amount of indexes, but great support for “sequential scans” of log files. We send the logs in semi-structured format (logfmt), and we have no real performance problems with this. The trick is that you want to optimize for “easy to use regexes”, which we did not have with json. Logfmt gives this to us, while still being easy to read with the eyes. One of the key decisions we made, however, was to not use any cloud provider for this and run it ourselves. It gave us much more freedom with various ways to interact with our metrics / data, without the risk of getting huge bills (I’m looking at you Datadog, your billing practices are terrible).

throwaway2380 6y ago

Traces can be a good place to handle high cardinality observability data. Traces essentially are a particular form of structured logs. Traces support arbitrary tags, and trace analysis systems can be optimized for quick aggregations over high cardinality fields.