4 ms·

Looks awesome! Three questions: 1. It sounded like it was easy to set key-value metadata in the promtail tool. Things like hostname, availability zone, etc.

Looks awesome! Three questions: 1. It sounded like it was easy to set key-value metadata in the promtail tool. Things like hostname, availability zone, etc. But can you also append metadata via the log-files themselves? Basically, our log files are JSONL (http://jsonlines.org/ http://jsonlines.org/) and look like this: { "user-id" : "abc", "client-type" : "mobile", "etc" : "…", "messages" : ["Parsing incoming http request", "Saving user data, valid model", "Finishing up http request, sending response to client"] } { "user-id: " "def", "etc" : "you get the idea…" } { "third log line here" : "and so on" } Can we ingest these into Loki and have the "user-id" metadata appended as key-value labels to each message? 2. Sometimes it makes sense to group related log lines. In the example above, we'll have ~20-30 log lines from a single http request. It would be convenient if we somehow could group them together, for example based on a unique X-Request-Id value. And then use that in the UI to see all related log lines together easily. 3. We currently store metadata about log-lines that is numeric. For example, things like request time. Will it be possible to query on that type of numerical value, to e.g. find all the log lines where requests took more than 500 ms?

wikibob 8y ago
I think the problem with setting key-value metadata via the log output, is that permits unbounded cardinality. The current design inherently limits cardinality to the number of pods you have running and the various labels applied to them. For 3 I'd say that doesn't make sense with this design. I'd suggest taking a look at something like scalyr.com. You can configure a log parser and then your logs become both queryable, and you can create time-series on numeric fields such as request time and look at the 99th percentile, min, max, etc.

wikibob 8y ago

I think the problem with setting key-value metadata via the log output, is that permits unbounded cardinality. The current design inherently limits cardinality to the number of pods you have running and the various labels applied to them. For 3 I'd say that doesn't make sense with this design. I'd suggest taking a look at something like scalyr.com. You can configure a log parser and then your logs become both queryable, and you can create time-series on numeric fields such as request time and look at the 99th percentile, min, max, etc.