3 ms·

Monitoring your logs is mostly a tarpit

24 points by ingve 3y ago

mikhailfranco 3y ago
Obviously the first thing to say is that logs should be structured, by design, at least for individual modules and cohesive subsystems. So maybe that's not possible, because other modules are under 3rd party control (even within your org). At my last job, I developed a test library implementing Regular Expressions for Events (regev). It allowed a test case to specify a regular expression grammar for the events it expected to be emitted by a test execution. For example, it could wildcard over the unknown junk events from other nested libraries (the whitespace of the event world), and quantify over repeated emissions from loops (0,1,n,?,+,*). My previous employer is bankrupt, so maybe it could be open sourced now, depending on who ends up with the IP.
- bafe 3y ago
  It sounds like the Match_recognize clause in SQL:2020, where you can define regex patterns for sequences of rows. It's a pity it's not available in most databases yet
- mikhailfranco 3y ago
  P.S. regev had its own unit tests that used ycombinator to generate event streams dynamically specified in the test - just thought I should mention that on this forum :) I have only used ycombinator one other time: for parsing text using grammars defined at runtime. I think that is the main use case for yc. It's in a personal side project, but will be open sourced real soon now.
- PhilipRoman 3y ago
  That is a really cool use of regexes. I feel like regexes (or more generally, grammars) still have lots of unexplored potential. I was toying around with making a strongly typed queue system based on grammars. Every time an element is written or read from a queue, the type gets updated (linear types help here).
m3047 3y ago
Seems defeatist to me. 1) There has to be a notion that some things are worth acknowledging as "events"; this leads to the idea that what logs contain is indicators of events. It's a fundamentally philosophical notion. It means you need to take the time to decide what constitutes an event. Hearkening to machine learning and pirates, global warming may inversely correlate with pirates but that doesn't imply causation (either way): you can't just throw statistical techniques at data looking for "hits" and think that's significant. Even if you find some indicator as the article notes it could change; so you should identify some canary indicators and event those as well. 2) Which leads to the point about "bug parts": don't rely on a specific rare indicator, or the failure to identify such an indicator. If you find high-reliability indicators great, but look for other indicators which occur more often, that can be counted, and track those. For instance an indicator that e.g. systemd is restarting /something/, and that's happening more or less frequently, and correlates with a performance observable. If it stops reporting at all, you can start with the presumption that something about logging itself changed. At this point my philosophical disagreement with centralized logging comes to the fore: it's expensive to load stuff into Splunk. I agree, and that's why I disagree with the approach and prefer federation. You can use the Totalizer Agent (https://github.com/m3047/rkvdns_examples/tree/main/totalizers https://github.com/m3047/rkvdns_examples/tree/main/totalizer...) to increment counters in Redis for regex-identified keys. I don't care whether you use RKVDNS to retrieve the data or something else.
somat 3y ago
I keep looking into statistical methods to help me with this. on the theory that for the most part log traffic is normal so I could have a dashboard item on how statistically anomalous my logs are. so I start by reading up on how email spam engines work, get hung up on locality aware hashing and then give up for a couple months. Any hints on good papers in this domain?
Terretta 3y ago
I don't understand the point of this post. OK, and then? Same for the post it links to: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AlertsNeverComprehensive https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AlertsNeve... Again, OK, and then? More succinctly than these two posts, the generalized problem is how to have a computer do something useful when encountering a state it hasn't seen before. After reformulating the objection this way, a number of paths open up.
- xg15 3y ago
  Yeah. He describes log monitoring as harder than it actually is, but even if he were completely right, what are you supposed to do instead? The problem you're trying to solve is still there, so what would be the better way to approach it than scanning the logs?
  - deleted 3y ago
    [deleted]

24 points by ingve 3y ago

Obviously the first thing to say is that logs should be structured, by design, at least for individual modules and cohesive subsystems. So maybe that's not possible, because other modules are under 3rd party control (even within your org). At my last job, I developed a test library implementing Regular Expressions for Events (regev). It allowed a test case to specify a regular expression grammar for the events it expected to be emitted by a test execution. For example, it could wildcard over the unknown junk events from other nested libraries (the whitespace of the event world), and quantify over repeated emissions from loops (0,1,n,?,+,*). My previous employer is bankrupt, so maybe it could be open sourced now, depending on who ends up with the IP.

bafe 3y ago
It sounds like the Match_recognize clause in SQL:2020, where you can define regex patterns for sequences of rows. It's a pity it's not available in most databases yet
mikhailfranco 3y ago
P.S. regev had its own unit tests that used ycombinator to generate event streams dynamically specified in the test - just thought I should mention that on this forum :) I have only used ycombinator one other time: for parsing text using grammars defined at runtime. I think that is the main use case for yc. It's in a personal side project, but will be open sourced real soon now.
PhilipRoman 3y ago
That is a really cool use of regexes. I feel like regexes (or more generally, grammars) still have lots of unexplored potential. I was toying around with making a strongly typed queue system based on grammars. Every time an element is written or read from a queue, the type gets updated (linear types help here).

m3047 3y ago

Seems defeatist to me. 1) There has to be a notion that some things are worth acknowledging as "events"; this leads to the idea that what logs contain is indicators of events. It's a fundamentally philosophical notion. It means you need to take the time to decide what constitutes an event. Hearkening to machine learning and pirates, global warming may inversely correlate with pirates but that doesn't imply causation (either way): you can't just throw statistical techniques at data looking for "hits" and think that's significant. Even if you find some indicator as the article notes it could change; so you should identify some canary indicators and event those as well. 2) Which leads to the point about "bug parts": don't rely on a specific rare indicator, or the failure to identify such an indicator. If you find high-reliability indicators great, but look for other indicators which occur more often, that can be counted, and track those. For instance an indicator that e.g. systemd is restarting /something/, and that's happening more or less frequently, and correlates with a performance observable. If it stops reporting at all, you can start with the presumption that something about logging itself changed. At this point my philosophical disagreement with centralized logging comes to the fore: it's expensive to load stuff into Splunk. I agree, and that's why I disagree with the approach and prefer federation. You can use the Totalizer Agent (https://github.com/m3047/rkvdns_examples/tree/main/totalizers https://github.com/m3047/rkvdns_examples/tree/main/totalizer...) to increment counters in Redis for regex-identified keys. I don't care whether you use RKVDNS to retrieve the data or something else.

somat 3y ago

I keep looking into statistical methods to help me with this. on the theory that for the most part log traffic is normal so I could have a dashboard item on how statistically anomalous my logs are. so I start by reading up on how email spam engines work, get hung up on locality aware hashing and then give up for a couple months. Any hints on good papers in this domain?

Terretta 3y ago

I don't understand the point of this post. OK, and then? Same for the post it links to: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AlertsNeverComprehensive https://utcc.utoronto.ca/~cks/space/blog/sysadmin/AlertsNeve... Again, OK, and then? More succinctly than these two posts, the generalized problem is how to have a computer do something useful when encountering a state it hasn't seen before. After reformulating the objection this way, a number of paths open up.

xg15 3y ago
Yeah. He describes log monitoring as harder than it actually is, but even if he were completely right, what are you supposed to do instead? The problem you're trying to solve is still there, so what would be the better way to approach it than scanning the logs?
- deleted 3y ago
  [deleted]