5 ms·

> The fundamental problem with general log monitoring is that logs are functionally unstructured. JSON format logs? One pattern I’ve used before - in a Java a

> The fundamental problem with general log monitoring is that logs are functionally unstructured. JSON format logs? One pattern I’ve used before - in a Java app with an RDBMS, create an error table, and every unexpected Java exception gets logged to that table. Then have a system to match problems in that table to known errors, or flag new ones for analysis. (I just clone the table from the production DB into a non-production DB, then have a bunch of views I wrote to analyse them - the fact it is cloned to the non-production DB means I don’t need release management approval to update the definitions of my views.) At a certain scale this may lead to problems with performance or data volumes, but generally not if the app is small to medium scale (as in practice many apps are), and especially if exceptions are sufficiently infrequent (one hopes so!)

jstanley 3y ago
In my experience JSON format logs are not super useful if they boil down to: {"date":"2023-08-08 10:14:35.129532","log":"... normal unstructured log text ..."}
- kubanczyk 3y ago
  Agreed, JSON is just a wire protocol. What TFA complains about is lack of contract on the part of logs' producers. (Schema would be a crucial part of such contract.)
- taneliv 3y ago
  There are a few more that are fairly common to be logged somewhat separately: severity, host and component/service, but even if so, the structure is lacking in universality (i.e. lot of logging systems support this, but in their own ways). So, yeah, it's not much. In my limited experience the amount of lines logged correlated with expected or actual activity of the system(s) is perhaps useful for monitoring. Looking into and analyzing the actual log events and their text is then the next step if there seems to be too much or too little information logged.
- magicalhippo 3y ago
  In .Net we're using Serilog which supports CLEF[1], and I'm in the process of changing our non-.Net code to do structured logging to CLEF as well (I made an internal library to support structured logging). Since Serilog supports consuming CLEF as well, this makes it trivial to upload the non-.Net logs to Azure Application Insights for example. Might be other options as well, I didn't look much further as this fit our needs well. edit: This doesn't completely solve the main point in the article of course, as the variables, error codes etc in the structured log message can change willy-nilly in a general system setting. [1]: https://clef-json.org/ https://clef-json.org/
jillesvangurp 3y ago
Logging is one of those things that remains mostly an afterthought for a lot of languages, frameworks, and engineers. It's not that hard to get organized. Here's what I've been doing since the last ten years on most of my projects (Java & Kotlin mostly but you should be bale to do this for anything). 1) log levels matter. Debug/tracing is disabled in production. Info is informational only and should not have a signal to noise ratio that gets annoying (people doing debug logging at info level). Warning frequency should be low. If you are not going to fix it, it's not worthy of a warning. Errors should cause alerts and people to be woken up. Simple rule but only if you enforce it. Don't log at error level unless it really is worth waking somebody up for (e.g. me). An error is not "something totally expected happened but I could not be arsed to think about handling that in a sane way". I've seen projects that routinely log thousands of errors per hour and never fix anything until after customers start yelling on the phone. Log levels are totally irrelevant in such projects. Nobody bothers to look at those errors. They are no longer actionable. If errors are normal and expected behavior, how do you tell when something abnormal and unexpected happens? You can't; unless you make people do something about those errors and create a culture where having errors is simply not an acceptable state for the product to be in. Life is great when you do that. We have zero errors on most days. When they do happen, it's usually because something changed. And then we fix that and it goes quiet again. Simple rule to enforce. Generates very little work. But you have to enforce it. 2) Java logging frameworks have something called a mapped diagnostic context (MDC). This is great. Basically it means every log entry can have a context where you can keep track of things in e.g. your request like headers, user agents, ip addresses, session ids, etc. Why don't other languages have this? I don't know. Seriously, how is this not a thing for any web development framework worthy of the name. Why would you not want to know this information when something happens? 3) Logging messages are structured data. Whether you like it or not. Plain text is a shitty way to represent structured data. If you can, log in json format. You have timestamps, logger names, log levels, attributes in your MDC, attributes coming from your server environment like the host name, service name, etc. All of it is relevant. 4) Tailing and grepping plain text logs simply does not scale. It's what you have to do when your ops team is too incompetent to setup proper logging. Usually goes hand in hand with having snow flake servers that people ssh into. It's actually the #1 excuse to have boxes you can ssh into to begin with. Solution: logs go into a data store that allows you to filter on this data. Not having this is the equivalent of running blind. Completely and utterly unacceptable. Most cloud environments come with a reasonably OK logging console but you might want to upgrade to something with a bit more querying capability. But done right even those default logging consoles can be capable enough. 5) Your logs should have alerts on them. If errors happen, alerts should happen and people should do things about those errors. If logs go silent when they shouldn't be, alerts should happen because something is probably broken. If there's a weird spike in logging volume, alerts should happen. Alerts should be actionable and be exceptional. If something is alerting all the time, nobody will check when something important actually does happen. Tricky to get right but once you do, you can react quickly to any incidents. Deliberately keeping product names out of this. There are plenty of libraries, tools and products that allow you to do this properly. That most likely includes your preferred software stack. And if it doesn't, use something more production ready that does. Or fix it (not that hard usually).
lelanthran 3y ago
In context, JSON is unstructured. FWIW, I have a small diagram for a subset of the taxonomy of logging: https://www.lelanthran.com/chap10/Logging-Guide-Light.png https://www.lelanthran.com/chap10/Logging-Guide-Light.png There's a whole blog post on that that I have been meaning to clean up and post to HN: https://www.lelanthran.com/chap10/content.html https://www.lelanthran.com/chap10/content.html
- KronisLV 3y ago
  That's a nice and succinct diagram! I liked the inclusion of examples in particular, good job. I'm sure that the folks here would enjoy you sharing the post as a submission, too.
  - lelanthran 3y ago
    Thank you for your kind words :-) I did share it at https://news.ycombinator.com/item?id=37048276 https://news.ycombinator.com/item?id=37048276, but it didn't get many upvotes. Will try only one more time in a month or so as I don't really want to spam HN.
    - justsomehnguy 3y ago
      FYI not everyone find 'handwritten' diagramms cute and more importantly sometimes this cuteness actively interfere with reading comperhension, ie this font is quite shitty for me, esp. on the mobile when I need to actively pan and zoom this big diagramm.
      - lelanthran 3y ago
        [EDIT: I think you are correct, and that it is not as legible on the phone as I thought it was. Using a different font might not fix that, although it's something for me to keep in mind for future blog post diagrams] I thank you for reading the content. > FYI not everyone find 'handwritten' diagramms cute and more importantly sometimes this cuteness actively interfere with reading comperhension, ie this font is quite shitty for me, esp. on the mobile when I need to actively pan and zoom this big diagramm. I understand that not everyone finds this sort of diagram as usable (by which I mean legibility, approachability, memorability and understandability). By the same token, not everyone finds the usual sort of diagram as usable as this. In my experience, I've found that cartoony/sketched images results in the audience being more receptive to, and remembering more of, the message. I think this is why this sort of thing is more and more popular these days - it mostly works (not for everyone, obviously). > on the mobile when I need to actively pan and zoom this big diagramm I'm sorry about that; I chose an image size that would fit most mobile phones in landscape orientation, and set the fontsize to what I personally could read without my glasses. For reference, on my mobile, HN text is too tiny for me to read without my glasses, so I was fairly sure that, if I could read it with no glasses on, it should be large enough on most screens for most people. Once again, thank you for reading the content.
eyko 3y ago
If you're going to treat logs as a db entry and afford them a schema, then by all means knock yourself out. The main point he's trying to make is not just about the logs you wrote but about logs coming from other systems or services, e.g. monitoring kernel logs in your OS. As he rightly points out, one of the reasons is that logs are not an API. > One reason this happens is that almost no one considers log messages to be an API, and so they feel free to change log messages at whim.
- gabereiser 3y ago
  create table log (dt timestamp not null primary key, msg text not null) is usually all you need to start sifting through the haystack. Problem with this approach is if you aren't diligent, you'll end up killing you app AND you're db.
  - deleted 3y ago
    [deleted]
  - sgarland 3y ago
    You can’t use log timestamps as a PK; you’ll run into non-unique entries nearly immediately.
    - gabereiser 3y ago
      We already established that log sifting at scale isn’t productive nor does it work for reasons outlined in the article. This was just an answer to how to do it in a smaller scale. Yeah, primary key is going to give you problems, a simple index shouldn’t but the issue is this, logs are not event sources - or shouldn’t be. The only time you should be looking is when your looking for stack trace and even then there’s better options like sentry.
Felototo 3y ago
I really hate logging to the db. I have seen this being a big issue in medium apps just because it's used blindly. And if you don't do it asyn out of your request path you add unnecessary latency to your hotpath. Sending logs to a logging system is really straight forward. At least use a second database with an independent connection
- skissane 3y ago
  > And if you don't do it asyn out of your request path you add unnecessary latency to your hotpath. Well, in my case - the table only logs unexpectedly failed requests (so not “user entered wrong password”, more like “HTTP 500 NullPointerException”.) That’s not on hot path by definition. I agree logging every single HTTP request to an RDBMS, even a 200 GET, is dumb.
  - sethammons 3y ago
    metric success, log errors (well, metric error too, but don't log success)
- calvinmorrison 3y ago
  Yes but what about logging over the mail daemon??
  - Felototo 3y ago
    In one company they logged everything in a public S3 bucket. 3 years later 10tb and a s3 bill of 200$/month...
    - calvinmorrison 3y ago
      > 200$/month... I mean literally peanuts to a company
phyrex 3y ago
That’s pretty much how meta does it, so I don’t think scaling is an issue. What you do at a certain scale is sampling.
throw0101a 3y ago
> JSON format logs? RFC 5424 syslog has a structured key=value component: * https://datatracker.ietf.org/doc/html/rfc5424#section-6.3 https://datatracker.ietf.org/doc/html/rfc5424#section-6.3
- nesarkvechnep 3y ago
  I was wondering if I’m the only one who knows about this feature. In general, do people use syslog? I haven’t worked on a project that does.
  - deathanatos 3y ago
    I've used it in the past. I forget what the implementation of syslog was that we were using (I want to say it was rsyslog) but it didn't support the RFC above. There are two syslog formats, the one in that RFC, and an older one, IIRC "BSD syslog", which is worse. So, if your implementation supports it. But JSON is loads easier to emit and to build random parsers around. In our case, syslog was then just upstreaming the logs as JSON to a more central collector, so we ended up just cutting it out of the loop.
layer8 3y ago
Speaking of Java, java.util.logging does support a structured XML-based format (XMLFormatter). And of course you can plug in your own.
renegade-otter 3y ago
This is why I wrote SiftLog (for Python): https://github.com/papito/siftlog-py https://github.com/papito/siftlog-py It logs locally in plain text and in color, for easy visual parsing, but in production - it's in JSON. It also lets you add arbitrary one-time properties or "sticky" constants (environment, process ID), or dynamic properties based on current context of the logger, such as information specific to a specific user request.

jstanley 3y ago

In my experience JSON format logs are not super useful if they boil down to: {"date":"2023-08-08 10:14:35.129532","log":"... normal unstructured log text ..."}

kubanczyk 3y ago
Agreed, JSON is just a wire protocol. What TFA complains about is lack of contract on the part of logs' producers. (Schema would be a crucial part of such contract.)
taneliv 3y ago
There are a few more that are fairly common to be logged somewhat separately: severity, host and component/service, but even if so, the structure is lacking in universality (i.e. lot of logging systems support this, but in their own ways). So, yeah, it's not much. In my limited experience the amount of lines logged correlated with expected or actual activity of the system(s) is perhaps useful for monitoring. Looking into and analyzing the actual log events and their text is then the next step if there seems to be too much or too little information logged.
magicalhippo 3y ago
In .Net we're using Serilog which supports CLEF[1], and I'm in the process of changing our non-.Net code to do structured logging to CLEF as well (I made an internal library to support structured logging). Since Serilog supports consuming CLEF as well, this makes it trivial to upload the non-.Net logs to Azure Application Insights for example. Might be other options as well, I didn't look much further as this fit our needs well. edit: This doesn't completely solve the main point in the article of course, as the variables, error codes etc in the structured log message can change willy-nilly in a general system setting. [1]: https://clef-json.org/ https://clef-json.org/

jillesvangurp 3y ago

Logging is one of those things that remains mostly an afterthought for a lot of languages, frameworks, and engineers. It's not that hard to get organized. Here's what I've been doing since the last ten years on most of my projects (Java & Kotlin mostly but you should be bale to do this for anything). 1) log levels matter. Debug/tracing is disabled in production. Info is informational only and should not have a signal to noise ratio that gets annoying (people doing debug logging at info level). Warning frequency should be low. If you are not going to fix it, it's not worthy of a warning. Errors should cause alerts and people to be woken up. Simple rule but only if you enforce it. Don't log at error level unless it really is worth waking somebody up for (e.g. me). An error is not "something totally expected happened but I could not be arsed to think about handling that in a sane way". I've seen projects that routinely log thousands of errors per hour and never fix anything until after customers start yelling on the phone. Log levels are totally irrelevant in such projects. Nobody bothers to look at those errors. They are no longer actionable. If errors are normal and expected behavior, how do you tell when something abnormal and unexpected happens? You can't; unless you make people do something about those errors and create a culture where having errors is simply not an acceptable state for the product to be in. Life is great when you do that. We have zero errors on most days. When they do happen, it's usually because something changed. And then we fix that and it goes quiet again. Simple rule to enforce. Generates very little work. But you have to enforce it. 2) Java logging frameworks have something called a mapped diagnostic context (MDC). This is great. Basically it means every log entry can have a context where you can keep track of things in e.g. your request like headers, user agents, ip addresses, session ids, etc. Why don't other languages have this? I don't know. Seriously, how is this not a thing for any web development framework worthy of the name. Why would you not want to know this information when something happens? 3) Logging messages are structured data. Whether you like it or not. Plain text is a shitty way to represent structured data. If you can, log in json format. You have timestamps, logger names, log levels, attributes in your MDC, attributes coming from your server environment like the host name, service name, etc. All of it is relevant. 4) Tailing and grepping plain text logs simply does not scale. It's what you have to do when your ops team is too incompetent to setup proper logging. Usually goes hand in hand with having snow flake servers that people ssh into. It's actually the #1 excuse to have boxes you can ssh into to begin with. Solution: logs go into a data store that allows you to filter on this data. Not having this is the equivalent of running blind. Completely and utterly unacceptable. Most cloud environments come with a reasonably OK logging console but you might want to upgrade to something with a bit more querying capability. But done right even those default logging consoles can be capable enough. 5) Your logs should have alerts on them. If errors happen, alerts should happen and people should do things about those errors. If logs go silent when they shouldn't be, alerts should happen because something is probably broken. If there's a weird spike in logging volume, alerts should happen. Alerts should be actionable and be exceptional. If something is alerting all the time, nobody will check when something important actually does happen. Tricky to get right but once you do, you can react quickly to any incidents. Deliberately keeping product names out of this. There are plenty of libraries, tools and products that allow you to do this properly. That most likely includes your preferred software stack. And if it doesn't, use something more production ready that does. Or fix it (not that hard usually).

lelanthran 3y ago

In context, JSON is unstructured. FWIW, I have a small diagram for a subset of the taxonomy of logging: https://www.lelanthran.com/chap10/Logging-Guide-Light.png https://www.lelanthran.com/chap10/Logging-Guide-Light.png There's a whole blog post on that that I have been meaning to clean up and post to HN: https://www.lelanthran.com/chap10/content.html https://www.lelanthran.com/chap10/content.html

KronisLV 3y ago
That's a nice and succinct diagram! I liked the inclusion of examples in particular, good job. I'm sure that the folks here would enjoy you sharing the post as a submission, too.
- lelanthran 3y ago
  Thank you for your kind words :-) I did share it at https://news.ycombinator.com/item?id=37048276 https://news.ycombinator.com/item?id=37048276, but it didn't get many upvotes. Will try only one more time in a month or so as I don't really want to spam HN.
  - justsomehnguy 3y ago
    FYI not everyone find 'handwritten' diagramms cute and more importantly sometimes this cuteness actively interfere with reading comperhension, ie this font is quite shitty for me, esp. on the mobile when I need to actively pan and zoom this big diagramm.
    - lelanthran 3y ago
      [EDIT: I think you are correct, and that it is not as legible on the phone as I thought it was. Using a different font might not fix that, although it's something for me to keep in mind for future blog post diagrams] I thank you for reading the content. > FYI not everyone find 'handwritten' diagramms cute and more importantly sometimes this cuteness actively interfere with reading comperhension, ie this font is quite shitty for me, esp. on the mobile when I need to actively pan and zoom this big diagramm. I understand that not everyone finds this sort of diagram as usable (by which I mean legibility, approachability, memorability and understandability). By the same token, not everyone finds the usual sort of diagram as usable as this. In my experience, I've found that cartoony/sketched images results in the audience being more receptive to, and remembering more of, the message. I think this is why this sort of thing is more and more popular these days - it mostly works (not for everyone, obviously). > on the mobile when I need to actively pan and zoom this big diagramm I'm sorry about that; I chose an image size that would fit most mobile phones in landscape orientation, and set the fontsize to what I personally could read without my glasses. For reference, on my mobile, HN text is too tiny for me to read without my glasses, so I was fairly sure that, if I could read it with no glasses on, it should be large enough on most screens for most people. Once again, thank you for reading the content.

eyko 3y ago

If you're going to treat logs as a db entry and afford them a schema, then by all means knock yourself out. The main point he's trying to make is not just about the logs you wrote but about logs coming from other systems or services, e.g. monitoring kernel logs in your OS. As he rightly points out, one of the reasons is that logs are not an API. > One reason this happens is that almost no one considers log messages to be an API, and so they feel free to change log messages at whim.

gabereiser 3y ago
create table log (dt timestamp not null primary key, msg text not null) is usually all you need to start sifting through the haystack. Problem with this approach is if you aren't diligent, you'll end up killing you app AND you're db.
- deleted 3y ago
  [deleted]
- sgarland 3y ago
  You can’t use log timestamps as a PK; you’ll run into non-unique entries nearly immediately.
  - gabereiser 3y ago
    We already established that log sifting at scale isn’t productive nor does it work for reasons outlined in the article. This was just an answer to how to do it in a smaller scale. Yeah, primary key is going to give you problems, a simple index shouldn’t but the issue is this, logs are not event sources - or shouldn’t be. The only time you should be looking is when your looking for stack trace and even then there’s better options like sentry.

Felototo 3y ago

I really hate logging to the db. I have seen this being a big issue in medium apps just because it's used blindly. And if you don't do it asyn out of your request path you add unnecessary latency to your hotpath. Sending logs to a logging system is really straight forward. At least use a second database with an independent connection

skissane 3y ago
> And if you don't do it asyn out of your request path you add unnecessary latency to your hotpath. Well, in my case - the table only logs unexpectedly failed requests (so not “user entered wrong password”, more like “HTTP 500 NullPointerException”.) That’s not on hot path by definition. I agree logging every single HTTP request to an RDBMS, even a 200 GET, is dumb.
- sethammons 3y ago
  metric success, log errors (well, metric error too, but don't log success)
calvinmorrison 3y ago
Yes but what about logging over the mail daemon??
- Felototo 3y ago
  In one company they logged everything in a public S3 bucket. 3 years later 10tb and a s3 bill of 200$/month...
  - calvinmorrison 3y ago
    > 200$/month... I mean literally peanuts to a company

phyrex 3y ago

That’s pretty much how meta does it, so I don’t think scaling is an issue. What you do at a certain scale is sampling.

throw0101a 3y ago

> JSON format logs? RFC 5424 syslog has a structured key=value component: * https://datatracker.ietf.org/doc/html/rfc5424#section-6.3 https://datatracker.ietf.org/doc/html/rfc5424#section-6.3

nesarkvechnep 3y ago
I was wondering if I’m the only one who knows about this feature. In general, do people use syslog? I haven’t worked on a project that does.
- deathanatos 3y ago
  I've used it in the past. I forget what the implementation of syslog was that we were using (I want to say it was rsyslog) but it didn't support the RFC above. There are two syslog formats, the one in that RFC, and an older one, IIRC "BSD syslog", which is worse. So, if your implementation supports it. But JSON is loads easier to emit and to build random parsers around. In our case, syslog was then just upstreaming the logs as JSON to a more central collector, so we ended up just cutting it out of the loop.

layer8 3y ago

Speaking of Java, java.util.logging does support a structured XML-based format (XMLFormatter). And of course you can plug in your own.

renegade-otter 3y ago

This is why I wrote SiftLog (for Python): https://github.com/papito/siftlog-py https://github.com/papito/siftlog-py It logs locally in plain text and in color, for easy visual parsing, but in production - it's in JSON. It also lets you add arbitrary one-time properties or "sticky" constants (environment, process ID), or dynamic properties based on current context of the logger, such as information specific to a specific user request.