3 ms·

What's the difference between a Wide Event and a structured log?

deathanatos 3y ago
Nothing; what the author is calling a wide event is a structured log. > Structured Log which is pretty much the Wide Event The OP's point is that you can just capture anything that one might call a metric/span/trace in a structured log. So, it's a bit more specific, in some senses: a "metric" is a structured log bearing some value. E.g., { "@timestamp": "…", "query_kind": "list_people_in_org", "query_time_s": .051 } Is a structured log. It's also a single datapoint along a timeseries. The metric might be something like "query time", with a dimension "query_kind". ELK stacks let you visualize such structured logs into graphs with only moderate pain. A "span" is a structured log with start/end timestamps. A trace is just a tree of spans, built out of many structured logs. In a prior life, we'd have, { "span_id": "root_span_id/subspan_a/subsub_span_b", … } You can pull out whole traces by doing a "span_id startswith 'root_span_id'" query in whatever your query language is. You can pull out sub-traces similarly. It works even across microservice boundaries, if you coordinate the span IDs appropriately. I think where I depart from the OP: it takes too much space. The OP advocates sampling; I'd rather a specialized data structure for metrics that can store a data point efficiently. I.e., at least in sizeof(timestamp || f32) bytes, ideally better, and then I can just not worry about sampling. (In a sense; if you're watching a continuous value, you're always sampling. But for discrete events that you're recording, I can just record all the events. But I'll only get a number. Sometimes, that's okay. I feel like this is a bit spiritually different than the sampling the OP is discussing.)
- karmakaze 3y ago
  Thanks. I don't know why new unpopular names are used for well-known things. Googling for "Wide Events" give millions of results for "City-wide events" etc. Probably just clickbaiting--what's a Wide Event?
  - phillipcarter 3y ago
    So, this is just my own view, but here's how I see it: Structured logs are ... well, logs that are structured (probably json) and machines read 'em to offer nice analysis capabilities. We've largely as an industry decided that these are great, and we should use them instead of unstructured logs if we can. But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that". And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging. Hence, a desire to call them "wide events". Anyways, I'm not particularly sold on that term either. But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease.
    - deathanatos 3y ago
      Structured logs are sensible, because you're not setting yourself up to need to write a billion little parsers down the line to attempt to parse unstructured logs, if they're even still parsable. JSON, or usually ndjson, is just simple, and widely supported. It's not the only format, nor even the best format. But it is easily produced, and better than no format at all. > But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that". Yes, you have to instrument the app. There's no getting around that. Experience will tell you what to log. > And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging. Again, experience. Log correlation is easier with what the article calls a "SpanId" or a "TraceId", some places call it a "RequestId" or a "CorrelationId", just some way to say "all longs forming a particular request" so that you can just read a request, from start to end. A simple thing … but you have to log it, or you won't have it. Agreement across services about naming it all the same thing in your log store of choice also helps, so that if you need to trace it across services, you can. This really isn't hard, but it depends IME on how well your organization is otherwise functioning. E.g., do engineers talk & plan? Can they say "X would help with Y" and then have people just go "yeah, it would. Done." or is it a long drawn out fight because technical leadership can't fathom basic stuff like what a RequestId does. > But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease. I feel like you're just saying "log the data we actually need", which isn't a terribly actionable thing if you don't already know the answer. If I had to make a recommendation: outcomes & latencies of I/O. Otherwise, put something into production, & be on-call for it.
MuffinFlavored 3y ago
How do you keep metadata around well enough to log in a structured fashion between apps? What if one app is a queue listener, which calls an HTTP service, etc. etc. I personally think the challenge is in passing around the metadata.
- karmakaze 3y ago
  This is solved using span IDs commonly used in distributed tracing.

Nothing; what the author is calling a wide event is a structured log. > Structured Log which is pretty much the Wide Event The OP's point is that you can just capture anything that one might call a metric/span/trace in a structured log. So, it's a bit more specific, in some senses: a "metric" is a structured log bearing some value. E.g., { "@timestamp": "…", "query_kind": "list_people_in_org", "query_time_s": .051 } Is a structured log. It's also a single datapoint along a timeseries. The metric might be something like "query time", with a dimension "query_kind". ELK stacks let you visualize such structured logs into graphs with only moderate pain. A "span" is a structured log with start/end timestamps. A trace is just a tree of spans, built out of many structured logs. In a prior life, we'd have, { "span_id": "root_span_id/subspan_a/subsub_span_b", … } You can pull out whole traces by doing a "span_id startswith 'root_span_id'" query in whatever your query language is. You can pull out sub-traces similarly. It works even across microservice boundaries, if you coordinate the span IDs appropriately. I think where I depart from the OP: it takes too much space. The OP advocates sampling; I'd rather a specialized data structure for metrics that can store a data point efficiently. I.e., at least in sizeof(timestamp || f32) bytes, ideally better, and then I can just not worry about sampling. (In a sense; if you're watching a continuous value, you're always sampling. But for discrete events that you're recording, I can just record all the events. But I'll only get a number. Sometimes, that's okay. I feel like this is a bit spiritually different than the sampling the OP is discussing.)

karmakaze 3y ago
Thanks. I don't know why new unpopular names are used for well-known things. Googling for "Wide Events" give millions of results for "City-wide events" etc. Probably just clickbaiting--what's a Wide Event?
- phillipcarter 3y ago
  So, this is just my own view, but here's how I see it: Structured logs are ... well, logs that are structured (probably json) and machines read 'em to offer nice analysis capabilities. We've largely as an industry decided that these are great, and we should use them instead of unstructured logs if we can. But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that". And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging. Hence, a desire to call them "wide events". Anyways, I'm not particularly sold on that term either. But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease.
  - deathanatos 3y ago
    Structured logs are sensible, because you're not setting yourself up to need to write a billion little parsers down the line to attempt to parse unstructured logs, if they're even still parsable. JSON, or usually ndjson, is just simple, and widely supported. It's not the only format, nor even the best format. But it is easily produced, and better than no format at all. > But just because a log is structured it doesn't mean it has what you need to effectively debug an issue! You usually need app-specific data added as key-value pairs to that log so that you can correlate stuff like "stuff is slow on this endpoint" with "and these are the device versions and user_agent strings that correlate the most with that". Yes, you have to instrument the app. There's no getting around that. Experience will tell you what to log. > And a lot of developers are used to dumping some of that data into random logs, but without making that data a part of a "wider" structured log, it can be really hard to correlate the behavior you don't like with other data that aids with debugging. Again, experience. Log correlation is easier with what the article calls a "SpanId" or a "TraceId", some places call it a "RequestId" or a "CorrelationId", just some way to say "all longs forming a particular request" so that you can just read a request, from start to end. A simple thing … but you have to log it, or you won't have it. Agreement across services about naming it all the same thing in your log store of choice also helps, so that if you need to trace it across services, you can. This really isn't hard, but it depends IME on how well your organization is otherwise functioning. E.g., do engineers talk & plan? Can they say "X would help with Y" and then have people just go "yeah, it would. Done." or is it a long drawn out fight because technical leadership can't fathom basic stuff like what a RequestId does. > But I do think there's some need to describe not only structured logs, but structured logs that contain all the rich info you need to debug most stuff in production with general ease. I feel like you're just saying "log the data we actually need", which isn't a terribly actionable thing if you don't already know the answer. If I had to make a recommendation: outcomes & latencies of I/O. Otherwise, put something into production, & be on-call for it.

MuffinFlavored 3y ago

How do you keep metadata around well enough to log in a structured fashion between apps? What if one app is a queue listener, which calls an HTTP service, etc. etc. I personally think the challenge is in passing around the metadata.

karmakaze 3y ago
This is solved using span IDs commonly used in distributed tracing.