3 ms·

Logs for machines should probably be expressed as metrics instead, because one can actually take the output of metrics for easy higher order analysis. Logs can

by adra 4y ago

Logs for machines should probably be expressed as metrics instead, because one can actually take the output of metrics for easy higher order analysis. Logs can do the same, but it takes more human effort to get there.

jasonpeacock 4y ago
Logs should be the input to generate metrics (in addition to any metrics published directly). Otherwise, it's hard to convert a stack trace into a metric ;) But you can process the logs to generate a metric about the frequency of stack traces. Ditto for logs telling you "user Foo doesn't have permissions for that operation". Again, you can make a metric about "user permissions failure" but you still need the log to know that it was user Foo.
- rroot 4y ago
  If you need a metric on the frequency of stack traces, you have a way more serious problem than structuring your logs.
  - lanstin 4y ago
    Or you are working for people with such a problem.
deathanatos 4y ago
Until you want to know which query is slow, which metrics won't tell you. Being able to do "where query_time_ms > 5000" and then seeing the query has made some of my systems incredibly debuggable¹. Or just finding a failing request with "where response_code = 500". Yeah, I can "grep 500", but you would be shocked at how often those characters occur by happenstance in logs "grep ' 500 '" is a bit closer but … you're just dancing around a solution engineered to do the job: structured logs. ¹and then finding out someone drew a rather large bounding box that managed to be centered precisely between and encapsulate both Philly & NYC. This kills the DB.
- adra 4y ago
  Most metric systems include tags, which allows you to have infinite dimensions of data correlation at your finger tips. Want the full query, or just a normalized version of the query? It should take a couple minutes in most metric engines to add such a simple requirement.
  - spondyl 4y ago
    > Most metric systems include tags, which allows you to have infinite dimensions of data correlation at your finger tips. Unfortunately with most SaaS metric services, your credit card does not scale to support infinite dimensions of data correlation. Tagging a high cardinality value like a customer ID for example would quickly blow out your bill.
    - lanstin 4y ago
      A mergable data type “sample values” would be awesome for these things. Host is dc1b001, api called is /whatever, p99 latency is 117 ms, status is 4xx, sample_corr_id is 0xfafedd12346
jkharrat 4y ago
This is an attempt to create a logically organized log format optimized for log parsing systems like newRelic or ELK etc. It will help you generate useful dashboards, metrics and event notifications (example: trigger an alert if the percentage of errors exceeds 5%). You can still turn logs into metrics and with that hit 2 birds with one stone.

by adra 4y ago

jasonpeacock 4y ago

Logs should be the input to generate metrics (in addition to any metrics published directly). Otherwise, it's hard to convert a stack trace into a metric ;) But you can process the logs to generate a metric about the frequency of stack traces. Ditto for logs telling you "user Foo doesn't have permissions for that operation". Again, you can make a metric about "user permissions failure" but you still need the log to know that it was user Foo.

rroot 4y ago
If you need a metric on the frequency of stack traces, you have a way more serious problem than structuring your logs.
- lanstin 4y ago
  Or you are working for people with such a problem.

deathanatos 4y ago

Until you want to know which query is slow, which metrics won't tell you. Being able to do "where query_time_ms > 5000" and then seeing the query has made some of my systems incredibly debuggable¹. Or just finding a failing request with "where response_code = 500". Yeah, I can "grep 500", but you would be shocked at how often those characters occur by happenstance in logs "grep ' 500 '" is a bit closer but … you're just dancing around a solution engineered to do the job: structured logs. ¹and then finding out someone drew a rather large bounding box that managed to be centered precisely between and encapsulate both Philly & NYC. This kills the DB.

adra 4y ago
Most metric systems include tags, which allows you to have infinite dimensions of data correlation at your finger tips. Want the full query, or just a normalized version of the query? It should take a couple minutes in most metric engines to add such a simple requirement.
- spondyl 4y ago
  > Most metric systems include tags, which allows you to have infinite dimensions of data correlation at your finger tips. Unfortunately with most SaaS metric services, your credit card does not scale to support infinite dimensions of data correlation. Tagging a high cardinality value like a customer ID for example would quickly blow out your bill.
  - lanstin 4y ago
    A mergable data type “sample values” would be awesome for these things. Host is dc1b001, api called is /whatever, p99 latency is 117 ms, status is 4xx, sample_corr_id is 0xfafedd12346

jkharrat 4y ago

This is an attempt to create a logically organized log format optimized for log parsing systems like newRelic or ELK etc. It will help you generate useful dashboards, metrics and event notifications (example: trigger an alert if the percentage of errors exceeds 5%). You can still turn logs into metrics and with that hit 2 birds with one stone.