4 ms·

I generally feel that way regarding the log systems I've worked with, specifically since they were some of the first I helped implement and/or design, and thus

I generally feel that way regarding the log systems I've worked with, specifically since they were some of the first I helped implement and/or design, and thus I'd not consider them high quality. However, what would be better than monitoring logs? Any sort of monitoring I know of looks at logs produced by one system or another... I'm not sure if there is an alternative.

Xylakant 3y ago
Tracking specific metrics and monitoring those is an option - for example you could track service restarts and alert on unusually high numbers. The tricky part - just as with logging - is to figure out what metric make sense and give you signal and what metrics are mostly noise. Sometimes you can even extract metrics from logs by looking for specific log lines if you know what to expect.
- fluential 3y ago
  You often create metrics based on logs indicating certain events, not much progress here in that approach
- kubanczyk 3y ago
  > Sometimes you can even extract metrics from logs by looking for specific log lines This looks backwards from the app dev's perspective. - Metrics are strictly structured, documented data. They are a first-class output. - "Strictly structured logs" is contradiction in terms. These are just events to be sent to a pub/sub, also a first-class output. - Logs are unstructured, unknown, bleeding-edge, grey-area data, which can potentially become either metrics, or events, or bug tickets. If I output data as metrics, I don't output it again as logs - that'd be just noise. But of course other apps/systems can have a wholly different approach and force people to invest a lot into sorting out that dumpster fire. The leverage is on the producer side.
gjvc 3y ago
if we are talking about the "log files" aspect then one certainly needs to get away from that and have all log events in a single searchable interface.
thom 3y ago
Start treating reporting and monitoring as first class parts of your domain instead of some text. Do the same modelling/event storming exercises you’d do for everything else with these operational requirements.

Xylakant 3y ago

Tracking specific metrics and monitoring those is an option - for example you could track service restarts and alert on unusually high numbers. The tricky part - just as with logging - is to figure out what metric make sense and give you signal and what metrics are mostly noise. Sometimes you can even extract metrics from logs by looking for specific log lines if you know what to expect.

fluential 3y ago
You often create metrics based on logs indicating certain events, not much progress here in that approach
kubanczyk 3y ago
> Sometimes you can even extract metrics from logs by looking for specific log lines This looks backwards from the app dev's perspective. - Metrics are strictly structured, documented data. They are a first-class output. - "Strictly structured logs" is contradiction in terms. These are just events to be sent to a pub/sub, also a first-class output. - Logs are unstructured, unknown, bleeding-edge, grey-area data, which can potentially become either metrics, or events, or bug tickets. If I output data as metrics, I don't output it again as logs - that'd be just noise. But of course other apps/systems can have a wholly different approach and force people to invest a lot into sorting out that dumpster fire. The leverage is on the producer side.

gjvc 3y ago

if we are talking about the "log files" aspect then one certainly needs to get away from that and have all log events in a single searchable interface.

thom 3y ago

Start treating reporting and monitoring as first class parts of your domain instead of some text. Do the same modelling/event storming exercises you’d do for everything else with these operational requirements.