19 ms·

So, this is just my own view, but here's how I see it: Structured logs are ... well, logs that are structured (probably json) and machines read 'em to offer ni

So, this is just my own view, but here's how I see it: Structured logs are ... well, logs that are structured (probably json) and machines read 'em to offer nice analysis capabilities. We've largely as an industry decided that these are great, and we should use them instead of unstructured logs if we can. But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that". And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging. Hence, a desire to call them "wide events". Anyways, I'm not particularly sold on that term either. But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease.

deathanatos 3y ago
Structured logs are sensible, because you're not setting yourself up to need to write a billion little parsers down the line to attempt to parse unstructured logs, if they're even still parsable. JSON, or usually ndjson, is just simple, and widely supported. It's not the only format, nor even the best format. But it is easily produced, and better than no format at all. > But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that". Yes, you have to instrument the app. There's no getting around that. Experience will tell you what to log. > And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging. Again, experience. Log correlation is easier with what the article calls a "SpanId" or a "TraceId", some places call it a "RequestId" or a "CorrelationId", just some way to say "all longs forming a particular request" so that you can just read a request, from start to end. A simple thing … but you have to log it, or you won't have it. Agreement across services about naming it all the same thing in your log store of choice also helps, so that if you need to trace it across services, you can. This really isn't hard, but it depends IME on how well your organization is otherwise functioning. E.g., do engineers talk & plan? Can they say "X would help with Y" and then have people just go "yeah, it would. Done." or is it a long drawn out fight because technical leadership can't fathom basic stuff like what a RequestId does. > But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease. I feel like you're just saying "log the data we actually need", which isn't a terribly actionable thing if you don't already know the answer. If I had to make a recommendation: outcomes & latencies of I/O. Otherwise, put something into production, & be on-call for it.
- phillipcarter 3y ago
  Yeah, a lot of what you're saying is what OpenTelemetry tries to solve for, and does so fairly well. If I were at an organization and had the ability and time, I'd move towards tracing and correlating existing logs with those traces, then adding new stuff to spans rather than creating new logs. Tracing as the baseline + common naming conventions can go a long way.

deathanatos 3y ago

Structured logs are sensible, because you're not setting yourself up to need to write a billion little parsers down the line to attempt to parse unstructured logs, if they're even still parsable. JSON, or usually ndjson, is just simple, and widely supported. It's not the only format, nor even the best format. But it is easily produced, and better than no format at all. > But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that". Yes, you have to instrument the app. There's no getting around that. Experience will tell you what to log. > And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging. Again, experience. Log correlation is easier with what the article calls a "SpanId" or a "TraceId", some places call it a "RequestId" or a "CorrelationId", just some way to say "all longs forming a particular request" so that you can just read a request, from start to end. A simple thing … but you have to log it, or you won't have it. Agreement across services about naming it all the same thing in your log store of choice also helps, so that if you need to trace it across services, you can. This really isn't hard, but it depends IME on how well your organization is otherwise functioning. E.g., do engineers talk & plan? Can they say "X would help with Y" and then have people just go "yeah, it would. Done." or is it a long drawn out fight because technical leadership can't fathom basic stuff like what a RequestId does. > But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease. I feel like you're just saying "log the data we actually need", which isn't a terribly actionable thing if you don't already know the answer. If I had to make a recommendation: outcomes & latencies of I/O. Otherwise, put something into production, & be on-call for it.

phillipcarter 3y ago
Yeah, a lot of what you're saying is what OpenTelemetry tries to solve for, and does so fairly well. If I were at an organization and had the ability and time, I'd move towards tracing and correlating existing logs with those traces, then adding new stuff to spans rather than creating new logs. Tracing as the baseline + common naming conventions can go a long way.