3 ms·

as a big data guy, every time i see structured data written in an unstructured, non standard, and possibly ambiguous and unparseable format i get the same urges

as a big data guy, every time i see structured data written in an unstructured, non standard, and possibly ambiguous and unparseable format i get the same urges. i think the best solution is separate logs for human consumption and machine consumption. if someone logs data that they want analyzed in some goofy format then they can also write and maintain their own damn parsers for it, i don't have time for that shit and the solution is so simple.

thaumaturgy 15y ago
Yeah, I can grok that. I think that the needs of big data are a bit different from the needs of troubleshooters. For example, I usually just need to sift through the data to quickly find the few bits that are interesting to me; you probably need to do aggregation and derive patterns and all kinds of other magic. I would only counterpoint that what I wrote was not at all a defense of badly-formatted logs. I'm also not at all opposed to people generate separate logs for humans and separate logs for machines. In fact, I think that's a great idea. I just really really do not want to see machine logs taking the place of human-readable logs.
- sneak 15y ago
  You could easily make a small unix filter that takes the json logs in, and spits out strings with "%date : %short_message" from them - enabling you to use them just like all of your other one-line-per-entry unixy plain logs. tail -f logfile | grep blah becomes tail -f logfile | json2line | grep blah There is no reason to have two sets of logs. JSON is human readable (with effort), and of course you can make a very small filter that will make it even moreso if you don't want to adapt at all. Machine-readable logs are a win. JSON is very close to providing the holy grail, machine _and_ human readable.
  - CrLf 15y ago
    In what format should that filter output? Which fields should it show? Should it be specific for the application or should it be generic? What parameters should it accept in this case? Do you see where I'm going with this? We have been using text-based logs and text-based pipes on unix for decades, even though it has always been inefficient to parse (especially on ancient machines). The thing is: it's much more efficient for humans. I understand the kind of cognitive dissonance this causes in programmers, I feel it myself. However, I've also been a sysadmin for a few years now and understand just how having plain text makes my life (much) easier.
    - sneak 15y ago
      Fine, then: tail -f logfile | json2line '%date %name %level %shortmsg' | grep expr Machine-readable logs are a huge win. It's trivial to make them backward-compatible with the entire universe of existing-format log processing tools (including your mind). To put up make-believe barriers in the way is simply fear of beneficial change. (To answer your question directly: The filter should be generic, and should allow you to specify simple format strings to be yanked out of top-level json fields.)
      - CrLf 15y ago
        I often find myself troubleshooting applications for which I know little about, or that I don't have to troubleshoot often enough (fortunately). I don't want to have to look what fields are available in the log to prepare the proper format string with the fields that I want because... I do not know what I want. I want to go to "/var/log/application" or whatever place the application stores its logs, and cat/tail whatever it is there and seems remotely related to the problem at hand. This is one of the reasons it is painful to troubleshoot problems in Windows using PowerShell, where what you suggested is more or less what needs to be done to parse the event log in a more intelligent way that filtering stuff using the Event Viewer. This shows in the speed and efficiency in solving issues with unix vs. windows systems. But you can say that's just my anecdotal evidence.

thaumaturgy 15y ago

Yeah, I can grok that. I think that the needs of big data are a bit different from the needs of troubleshooters. For example, I usually just need to sift through the data to quickly find the few bits that are interesting to me; you probably need to do aggregation and derive patterns and all kinds of other magic. I would only counterpoint that what I wrote was not at all a defense of badly-formatted logs. I'm also not at all opposed to people generate separate logs for humans and separate logs for machines. In fact, I think that's a great idea. I just really really do not want to see machine logs taking the place of human-readable logs.

sneak 15y ago
You could easily make a small unix filter that takes the json logs in, and spits out strings with "%date : %short_message" from them - enabling you to use them just like all of your other one-line-per-entry unixy plain logs. tail -f logfile | grep blah becomes tail -f logfile | json2line | grep blah There is no reason to have two sets of logs. JSON is human readable (with effort), and of course you can make a very small filter that will make it even moreso if you don't want to adapt at all. Machine-readable logs are a win. JSON is very close to providing the holy grail, machine _and_ human readable.
- CrLf 15y ago
  In what format should that filter output? Which fields should it show? Should it be specific for the application or should it be generic? What parameters should it accept in this case? Do you see where I'm going with this? We have been using text-based logs and text-based pipes on unix for decades, even though it has always been inefficient to parse (especially on ancient machines). The thing is: it's much more efficient for humans. I understand the kind of cognitive dissonance this causes in programmers, I feel it myself. However, I've also been a sysadmin for a few years now and understand just how having plain text makes my life (much) easier.
  - sneak 15y ago
    Fine, then: tail -f logfile | json2line '%date %name %level %shortmsg' | grep expr Machine-readable logs are a huge win. It's trivial to make them backward-compatible with the entire universe of existing-format log processing tools (including your mind). To put up make-believe barriers in the way is simply fear of beneficial change. (To answer your question directly: The filter should be generic, and should allow you to specify simple format strings to be yanked out of top-level json fields.)
    - CrLf 15y ago
      I often find myself troubleshooting applications for which I know little about, or that I don't have to troubleshoot often enough (fortunately). I don't want to have to look what fields are available in the log to prepare the proper format string with the fields that I want because... I do not know what I want. I want to go to "/var/log/application" or whatever place the application stores its logs, and cat/tail whatever it is there and seems remotely related to the problem at hand. This is one of the reasons it is painful to troubleshoot problems in Windows using PowerShell, where what you suggested is more or less what needs to be done to parse the event log in a more intelligent way that filtering stuff using the Event Viewer. This shows in the speed and efficiency in solving issues with unix vs. windows systems. But you can say that's just my anecdotal evidence.