What you're describing is why tracing (it's not just for distributed systems!) and tail sampling are employed in practice over just tons of log lines. Structuri

What you're describing is why tracing (it's not just for distributed systems!) and tail sampling are employed in practice over just tons of log lines. Structuring the data your generate and sampling it is how to approach this. If you sample 100% of errors (or some other meaningful signal), 1% or so of the rest - just so you have a good baseline to compare - and attach metadata for a tool to reweight counts you can more-or-less have your cake and eat it too.