5 ms·

Log for Machines

55 points by pquerna 15y ago

zdw 15y ago
This falls under "good intention, but not well planned out" IMO. Single line logs are MUCH easier to process with text-oriented tools, which is why they've stuck around for so long. This solves the encoding problem, but not the "how do I read/search it" problem, which is kicked down the line to other tools. And having a different format for every daemon is just a new problem. I'd look at how others have dealt with the problem - for example, Logstash, which is a log parsing and movement engine: http://logstash.net/docs/1.0.17/learn http://logstash.net/docs/1.0.17/learn Also, timestamps aren't recommended for storing time data - they're not human readable, and convey less information than something like ISO8601 which covers more edge cases.
btipling 15y ago
I wonder if we need text logs at all anymore. It seems silly to expect people to be able to read millions of log entries from tens, hundreds or even thousands of servers. Maybe binary logs using protocol buffers or thrift are the way of the future.
- pquerna 15y ago
  Maybe. Protobuffers or Thrift can definitely be smaller in byte size, which does matter somewhat, but JSON right now is just more easily accessible. I think in a few years, assuming JSON logging really takes off, you'll see people using these other RPC formats in order to optimize and reduce the size of their messages. I view 'better' RPC formats as a performance optimization of the same basic idea though: Easily add new fields, and use a format designed for computers to consume.
  - jarito 15y ago
    JSON lends itself pretty well to compression as well so a binary format might not buy you as much as you think.
- enjoy-your-stay 15y ago
  What would be binary data? The only possible thing I could think of would be the timestamp. Everything else would most likely be text, ASCII for English speakers, UTF-8 otherwise.
latchkey 15y ago
Apache Flume also xfers things in json.
loeg 15y ago
s,JSON,BSON,g These logs are for machines, and should be compact -- why use JSON at all?
- CoffeeDregs 15y ago
  Two reasons: Easily read by human *and* by machine; Highly compressible since it's full of repeating strings. My Apache logs compress by about 94%. Chances are good that you wouldn't find much of a size difference between gzipped JSON and gzipped BSON (though parsing BSON would be faster).
  - loeg 15y ago
    One could imagine a "bzoncat" utility which simply reads bson objects on stdin and emits formatted json objects on stdout. I don't know, the difference probably doesn't matter.
thaumaturgy 15y ago
NO! No, no, no, a thousand times, no. I spend all day every day troubleshooting broken things. Human readable logs are the very last thing that I have left that leaves me with any energy to keep troubleshooting things. If you take that away from me too, I swear I'm going to shoot somebody. (If I have any sense left when I go 'round the bend, I'll target a software engineer that thought it would be a good idea to fix what wasn't broke.) Here's the thing: I can do anything with a human readable text log. I can scan it visually and look for anomalies; I can use my favorite text editor with it; I can grep it; I can cobble together some magic incantation on a command line to munge it to death until it cries out in mercy and gives me exactly what I'm looking for. If I have to use somebody's special web-based woo-widget to view and search the contents of a log, if I'm restricted to the limits of their imagination when it comes to trying to get useful information out of the log, if I have to debug their dumbass invention while somebody's time-critical server is flopping around on the floor before I can even begin to figure out what in the heck is actually wrong with the server, I'm going to be stupid not to give up completely on support. If you want to log information which is useful to you, you can do that without hardly any trouble at all without replacing perfectly good, reliable logging facilities which are the way they are for damned good reasons. Just write a Ruby whatchamacallit or a Python thingamajig or a PHP flibbedyfloo and log it to your MySQL database or your Postgres database or your no-sql nuh-uh-it's-not-a-database. I don't care. But if I ever open up a server log and get blasted by a jillion lines of JSON that make it a thousand times harder for me to figure out how something broke I'm going to go on a shooting spree.
- jeffffff 15y ago
  as a big data guy, every time i see structured data written in an unstructured, non standard, and possibly ambiguous and unparseable format i get the same urges. i think the best solution is separate logs for human consumption and machine consumption. if someone logs data that they want analyzed in some goofy format then they can also write and maintain their own damn parsers for it, i don't have time for that shit and the solution is so simple.
  - thaumaturgy 15y ago
    Yeah, I can grok that. I think that the needs of big data are a bit different from the needs of troubleshooters. For example, I usually just need to sift through the data to quickly find the few bits that are interesting to me; you probably need to do aggregation and derive patterns and all kinds of other magic. I would only counterpoint that what I wrote was not at all a defense of badly-formatted logs. I'm also not at all opposed to people generate separate logs for humans and separate logs for machines. In fact, I think that's a great idea. I just really really do not want to see machine logs taking the place of human-readable logs.
mdwrigh2 15y ago
There's actually a plan to replace syslog with something that's easier to create tools for. https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs&pli=1 https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBA...
donut 15y ago
Regardless of the syntax for encoding a server event, having a request id (txnId in the post) is an often overlooked detail. See how Google do this in their systems: http://highscalability.com/blog/2010/4/27/paper-dapper-googles-large-scale-distributed-systems-tracing.html http://highscalability.com/blog/2010/4/27/paper-dapper-googl...
angersock 15y ago
Oh christ. I'm pretty sure my sysadmin friends would murder anyone that tried this on their boxen. As an engineer, while I appreciate the amount of shiny going on here, I doubt that adding yet another layer of tooling requirements to dealing with logs is the right answer--this could lead to serious ax-sharpening of picking the best JSON-to-readable scripts, and you can bet your ass that'll be different on every machine, install, site, etc. etc. All that being said, I could see this being hilariously useful in game development. Having little JSON crumbs dribble out during playtesting and QA would be really handy, and that is an area where data mining is actually quite useful. Having a trivial way of sorting through crumbs by issue or tester, and then being able to, say, project the bugs into an HTML5/WebGL-based tool for devs to use would be really nifty.
- HalibetLector 15y ago
  Social games has been doing this for 2+ years already, with JSON log files scraped into various databases and viewed using web tools for everything from A/B test impressions to virality to monetization to error logging (especially useful when you can't get the error to reproduce locally).
gosub 15y ago
I'd prefer to have two logs, one for human reading and one for machine consumption.
- sneak 15y ago
  DRY. Just make a text filter that takes the machine ones on stdin and prints out the human ones on stdout. Unix, baby!
CrLf 15y ago
Not this again... * The first level consumer of a log message is a human. * The programer knows what information is needed to debug an issue. I believe these presumptions are no longer correct in server side software. The author's assumptions are wrong on both cases: There may be any number of automated agents consuming the log, so a human isn't the largest consumer of log messages (in volume), but it is the first consumer of log messages as it is a human that ultimately will need them to troubleshoot issues. Programmers are not the ones who "debug" issues in production environments. Both of these are just symptoms of the same base misconception about how systems administrators go about their work. The same misconception that brought us XML configuration files. When troubleshooting production issues I want to be able to grep for specific strings or regular expressions. I want to be able to tail the log. I do not want to have to parse structure (programatically or mentally), I do not want multi-line log entries. I do not want to be forced to use a tool just to transform the log from whatever format into something I can read. Going even further: often production issues arise when only people less familiar with the system are available locally. In these cases I don't want to explain log structure over the phone.
johnjhayes 15y ago
Before tea, standard sysadmin response: NO After tea and some thought: No thanks

55 points by pquerna 15y ago

zdw 15y ago

This falls under "good intention, but not well planned out" IMO. Single line logs are MUCH easier to process with text-oriented tools, which is why they've stuck around for so long. This solves the encoding problem, but not the "how do I read/search it" problem, which is kicked down the line to other tools. And having a different format for every daemon is just a new problem. I'd look at how others have dealt with the problem - for example, Logstash, which is a log parsing and movement engine: http://logstash.net/docs/1.0.17/learn http://logstash.net/docs/1.0.17/learn Also, timestamps aren't recommended for storing time data - they're not human readable, and convey less information than something like ISO8601 which covers more edge cases.

btipling 15y ago

I wonder if we need text logs at all anymore. It seems silly to expect people to be able to read millions of log entries from tens, hundreds or even thousands of servers. Maybe binary logs using protocol buffers or thrift are the way of the future.

pquerna 15y ago
Maybe. Protobuffers or Thrift can definitely be smaller in byte size, which does matter somewhat, but JSON right now is just more easily accessible. I think in a few years, assuming JSON logging really takes off, you'll see people using these other RPC formats in order to optimize and reduce the size of their messages. I view 'better' RPC formats as a performance optimization of the same basic idea though: Easily add new fields, and use a format designed for computers to consume.
- jarito 15y ago
  JSON lends itself pretty well to compression as well so a binary format might not buy you as much as you think.
enjoy-your-stay 15y ago
What would be binary data? The only possible thing I could think of would be the timestamp. Everything else would most likely be text, ASCII for English speakers, UTF-8 otherwise.

latchkey 15y ago

Apache Flume also xfers things in json.

loeg 15y ago

s,JSON,BSON,g These logs are for machines, and should be compact -- why use JSON at all?

CoffeeDregs 15y ago
Two reasons: Easily read by human *and* by machine; Highly compressible since it's full of repeating strings. My Apache logs compress by about 94%. Chances are good that you wouldn't find much of a size difference between gzipped JSON and gzipped BSON (though parsing BSON would be faster).
- loeg 15y ago
  One could imagine a "bzoncat" utility which simply reads bson objects on stdin and emits formatted json objects on stdout. I don't know, the difference probably doesn't matter.

thaumaturgy 15y ago

NO! No, no, no, a thousand times, no. I spend all day every day troubleshooting broken things. Human readable logs are the very last thing that I have left that leaves me with any energy to keep troubleshooting things. If you take that away from me too, I swear I'm going to shoot somebody. (If I have any sense left when I go 'round the bend, I'll target a software engineer that thought it would be a good idea to fix what wasn't broke.) Here's the thing: I can do anything with a human readable text log. I can scan it visually and look for anomalies; I can use my favorite text editor with it; I can grep it; I can cobble together some magic incantation on a command line to munge it to death until it cries out in mercy and gives me exactly what I'm looking for. If I have to use somebody's special web-based woo-widget to view and search the contents of a log, if I'm restricted to the limits of their imagination when it comes to trying to get useful information out of the log, if I have to debug their dumbass invention while somebody's time-critical server is flopping around on the floor before I can even begin to figure out what in the heck is actually wrong with the server, I'm going to be stupid not to give up completely on support. If you want to log information which is useful to you, you can do that without hardly any trouble at all without replacing perfectly good, reliable logging facilities which are the way they are for damned good reasons. Just write a Ruby whatchamacallit or a Python thingamajig or a PHP flibbedyfloo and log it to your MySQL database or your Postgres database or your no-sql nuh-uh-it's-not-a-database. I don't care. But if I ever open up a server log and get blasted by a jillion lines of JSON that make it a thousand times harder for me to figure out how something broke I'm going to go on a shooting spree.

jeffffff 15y ago
as a big data guy, every time i see structured data written in an unstructured, non standard, and possibly ambiguous and unparseable format i get the same urges. i think the best solution is separate logs for human consumption and machine consumption. if someone logs data that they want analyzed in some goofy format then they can also write and maintain their own damn parsers for it, i don't have time for that shit and the solution is so simple.
- thaumaturgy 15y ago
  Yeah, I can grok that. I think that the needs of big data are a bit different from the needs of troubleshooters. For example, I usually just need to sift through the data to quickly find the few bits that are interesting to me; you probably need to do aggregation and derive patterns and all kinds of other magic. I would only counterpoint that what I wrote was not at all a defense of badly-formatted logs. I'm also not at all opposed to people generate separate logs for humans and separate logs for machines. In fact, I think that's a great idea. I just really really do not want to see machine logs taking the place of human-readable logs.

mdwrigh2 15y ago

There's actually a plan to replace syslog with something that's easier to create tools for. https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs&pli=1 https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBA...

donut 15y ago

Regardless of the syntax for encoding a server event, having a request id (txnId in the post) is an often overlooked detail. See how Google do this in their systems: http://highscalability.com/blog/2010/4/27/paper-dapper-googles-large-scale-distributed-systems-tracing.html http://highscalability.com/blog/2010/4/27/paper-dapper-googl...

angersock 15y ago

Oh christ. I'm pretty sure my sysadmin friends would murder anyone that tried this on their boxen. As an engineer, while I appreciate the amount of shiny going on here, I doubt that adding yet another layer of tooling requirements to dealing with logs is the right answer--this could lead to serious ax-sharpening of picking the best JSON-to-readable scripts, and you can bet your ass that'll be different on every machine, install, site, etc. etc. All that being said, I could see this being hilariously useful in game development. Having little JSON crumbs dribble out during playtesting and QA would be really handy, and that is an area where data mining is actually quite useful. Having a trivial way of sorting through crumbs by issue or tester, and then being able to, say, project the bugs into an HTML5/WebGL-based tool for devs to use would be really nifty.

HalibetLector 15y ago
Social games has been doing this for 2+ years already, with JSON log files scraped into various databases and viewed using web tools for everything from A/B test impressions to virality to monetization to error logging (especially useful when you can't get the error to reproduce locally).

gosub 15y ago

I'd prefer to have two logs, one for human reading and one for machine consumption.

sneak 15y ago
DRY. Just make a text filter that takes the machine ones on stdin and prints out the human ones on stdout. Unix, baby!

CrLf 15y ago

Not this again... * The first level consumer of a log message is a human. * The programer knows what information is needed to debug an issue. I believe these presumptions are no longer correct in server side software. The author's assumptions are wrong on both cases: There may be any number of automated agents consuming the log, so a human isn't the largest consumer of log messages (in volume), but it is the first consumer of log messages as it is a human that ultimately will need them to troubleshoot issues. Programmers are not the ones who "debug" issues in production environments. Both of these are just symptoms of the same base misconception about how systems administrators go about their work. The same misconception that brought us XML configuration files. When troubleshooting production issues I want to be able to grep for specific strings or regular expressions. I want to be able to tail the log. I do not want to have to parse structure (programatically or mentally), I do not want multi-line log entries. I do not want to be forced to use a tool just to transform the log from whatever format into something I can read. Going even further: often production issues arise when only people less familiar with the system are available locally. In these cases I don't want to explain log structure over the phone.

johnjhayes 15y ago

Before tea, standard sysadmin response: NO After tea and some thought: No thanks