3 ms·
Too much telemetry is more a problem that not enough in my recent experience. I am 100% sure the line I need to find out what happened is there in kibana but ev
by mianos 4y ago
Too much telemetry is more a problem that not enough in my recent experience. I am 100% sure the line I need to find out what happened is there in kibana but every extra filter term to make a trillion lines of log output down to a specific time and sequence, adds a risk of filtering out exactly what I want.
- phillipcarter 4y agoWhat you're describing is why tracing (it's not just for distributed systems!) and tail sampling are employed in practice over just tons of log lines. Structuring the data your generate and sampling it is how to approach this. If you sample 100% of errors (or some other meaningful signal), 1% or so of the rest - just so you have a good baseline to compare - and attach metadata for a tool to reweight counts you can more-or-less have your cake and eat it too.