I think the problem with setting key-value metadata via the log output, is that permits unbounded cardinality. The current design inherently limits cardinality

I think the problem with setting key-value metadata via the log output, is that permits unbounded cardinality. The current design inherently limits cardinality to the number of pods you have running and the various labels applied to them. For 3 I'd say that doesn't make sense with this design. I'd suggest taking a look at something like scalyr.com. You can configure a log parser and then your logs become both queryable, and you can create time-series on numeric fields such as request time and look at the 99th percentile, min, max, etc.