3 ms·

Fascinating write-up. I have some experience with log monitoring and alerting systems on cloud infrastructure, and there seem to be some fascinating parallels.

Fascinating write-up. I have some experience with log monitoring and alerting systems on cloud infrastructure, and there seem to be some fascinating parallels. Some of the issues described, including especially "Good at catching known-pattern issues, but not great for discovering novel issues." Large distributed system logs, in the raw, are straight noise. None of the data has meaning until you add a layer of interpretation on top of it, and that interpretation often involves actual software creativity. Is a load spike indicative of a bug in your handler services or your key demographic all logging on at once (because you're, say, a football-focused service and Halftime at the Superbowl just started)? Are spikes in 400 queries on a Monday morning indicative of some kind of cyclical error in your infrastructure or did a client build a broken automated job that fires once a week and tries to feed your system copies of the old query schema? Oh no, a surge in 404s... Did a whole chunk of your site stop serving, or did someone kick off the world's most poorly-tuned web crawler and point an army of instances of it at your domain? You can query the data for these things, but in general that is the flow of information: the data doesn't tell you anything, you ask it. You come up with queries against the aggregated data, identify and discard obvious noise, and decide whether the data supports a hypothesis that something is actually wrong in the world.

toast0 3y ago
> Large distributed system logs, in the raw, are straight noise. None of the data has meaning until you add a layer of interpretation on top of it, and that interpretation often involves actual software creativity. Additionally, I find that when people add logs/metrics 'just in case', they're often not sufficient to satisfy the queries that are developed later. You often need to know what you want to query about so that you can log the right things. The same thing happens with medical statistics: you can't prove or disprove a hypothesis from data unless the data collection was designed with that hypothesis in mind.
Gibbon1 3y ago
You're on the money. People as a group are constantly having transient health related stuff. Reporting all of those when they occur a few days or weeks after someones been vaccinated is no different than snapping a picture with a camera. One thing that bothers me is composing health events after vaccination with the base rate of those events. Those are no the same because in the former there is an known event. In the latter we don't capture the triggering events. Like perhaps both vaccination and the flu in young males causes mild heart inflammation. The base rate for flu related heart inflammation will be proportional to the percentage of males that get the flu per unit time. A small fractional number. Where when measuring after vax, 100% were vaxed.

toast0 3y ago

> Large distributed system logs, in the raw, are straight noise. None of the data has meaning until you add a layer of interpretation on top of it, and that interpretation often involves actual software creativity. Additionally, I find that when people add logs/metrics 'just in case', they're often not sufficient to satisfy the queries that are developed later. You often need to know what you want to query about so that you can log the right things. The same thing happens with medical statistics: you can't prove or disprove a hypothesis from data unless the data collection was designed with that hypothesis in mind.

Gibbon1 3y ago

You're on the money. People as a group are constantly having transient health related stuff. Reporting all of those when they occur a few days or weeks after someones been vaccinated is no different than snapping a picture with a camera. One thing that bothers me is composing health events after vaccination with the base rate of those events. Those are no the same because in the former there is an known event. In the latter we don't capture the triggering events. Like perhaps both vaccination and the flu in young males causes mild heart inflammation. The base rate for flu related heart inflammation will be proportional to the percentage of males that get the flu per unit time. A small fractional number. Where when measuring after vax, 100% were vaxed.