10 ms·

Log Everything as JSON. Make Your Life Easier.

155 points by kiyoto 15y ago

rhizome 15y ago
Except that fixed-field loglines are much faster to process than parsing JSON, which makes a difference when working with large logs.
- kelnos 15y ago
  Depends on what you value more: CPU time, or your developers' time keeping the log format and parsers in sync after any change.
  - mkross 15y ago
    You could test that the logging and parsing are in step, which then reduces the developer time to "Doh, red test" and a simple fix.
    - LeafStorm 15y ago
      So...integrate quick scripts to get a count of IP addresses into your app's test suite?
    - kelnos 14y ago
      Why increase your burden even to that? Computers are there to do stupid repetitive stuff for us so we don't have to. Why design a logging format that requires even minimal manual intervention to keep automated parsing working?
rachelbythebay 15y ago
This article feels like it would work just as well with "Protocol Buffers", "Thrift", "XML", or even maybe "ASN.1". If that's truly the case, maybe the better thing to say is "please don't (only) log in ASCII", followed by "please use a format which is hard to get wrong". JSON scares me a little. Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? I saw a page not too long ago which listed about a dozen characters which should be substituted in some manner when used in JSON. Full disclosure: I got tired of ASCII logging from my web server and wrote something to stream binary protocol buffers (!) to a file instead. http://rachelbythebay.com/w/2012/02/12/progress/ http://rachelbythebay.com/w/2012/02/12/progress/
- djcjr 15y ago
  what, no mention of sexp's?
  - kiyoto 15y ago
    djcjr: Are you an OCaml user ;p?
- kelnos 15y ago
  Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? Yes, but you don't do it yourself. As the article mentions, pretty much every language you'd want to use has a JSON library. Use that, and (bugs aside) it'll take care of escaping for you.
  - rachelbythebay 15y ago
    I sure hope everyone uses a library. What worries me is that it looks so simple, so someone might just glue together their own encoding and call it "done". A higher barrier to entry isn't always a bad thing.
- kiyoto 15y ago
  rachelbythebay: Yea, you've got some good points. I don't think the author means JSON is the be-all-end-all solution to structured logging. It has its issues (escaping for one), but I think it is a reasonable choice because of its wide adoption. A nice use of protocol buffer there. Speaking of serialization, my personal favorite is MessagePack (httpx://github.com/msgpack).
  - deleted 15y ago
    [deleted]
- frsyuki 15y ago
  There are binary-based serialization formats such as MessagePack [1] or BSON [2] that doesn't need escapeing overhead. You can simply replace JSON with MessagePack since MessagePack is compatible with JSON. [1] Why Not MessagePack? http://blog.andrewvc.com/why-arent-you-using-messagepack http://blog.andrewvc.com/why-arent-you-using-messagepack [2] Performant Entity Serialization: BSON vs MessagePack (vs JSON) http://stackoverflow.com/questions/6355497/performant-entity-serialization-bson-vs-messagepack-vs-json http://stackoverflow.com/questions/6355497/performant-entity...
daenz 15y ago
Logging to mongo (as JSON) has proven useful to us. Makes it easy to slice and dice the data.
- asuth 15y ago
  Ya, logs are soooo much more valuable when they're in a database that you can query. We log to MySQL and then correlate logs with users, events, IPs, pages, etc. Logs are just as relational as anything else, it took me awhile to realize it though.
jacques_chester 15y ago
One of the non-functional requirements of logs is that they should be fast to write. Marshalling data into a structured format takes longer than spitting out sprintfs. If you really need structure for ease of querying, you might as well go all the way and throw it into a proper data store.
- frsyuki 15y ago
  Spitting out sprintfs means parsing texts. JSON might be slow since it's text, but binary based formats should be faster especially the log includes many integers.
  - jacques_chester 15y ago
    sprintf tends to be a faster path than "build tree, render down to text".
    - deno 15y ago
      Both XML and JSON can be streamed. No tree building necessary.
- gorset 15y ago
  Remember we're talking about writing to disk. The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. We use JSON on disk heavily in several places where we generate or receive huge amounts of data. JSON-in-a-file is a pretty good datastore for sequential data processing because it's so convent and you can work with the data using cat, tail or any tool supporting JSON.
  - stevewilhelm 15y ago
    You're still logging to local disk?
  - jacques_chester 15y ago
    > The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. And JSON is more verbose. Basically this is another read/write dichotomy. When do we pay the price of structuring data? At write time? Or at read time? I'd rather pay it at read time, as writing may be shaving performance from my actual primary production system.
NyxWulf 15y ago
I've seen several articles like this, and there are a number of things to consider. Logging to ascii means that the standard unix tools work out of the box with your log files. Especially if you use something like a tab delimiter, then you typically don't need to specify the delimiter. As an upside you are aren't storing the column definition in every single line, which if you are doing large volume traffic definitely matters. For instance we store gigabytes of log files per hour, grossing up that space by a significant margin impacts storage, transit and process times during write (marshallers and custom log formatting). Writes are the hardest to scale, so if I'm going to add scale or extra parsing time, I'd rather handle that in Hadoop where I can throw massive parallel resources at it. Next you can achieve much of the advantages of json or protocol buffers by having a defined format and a structured release process before someone can change the format. Add fields to the end and don't remove defunct fields. This is the same process you have to use with protocol buffers or conceptually with JSON to have it work. Overall there are advantages to these other formats, but the articles like this that I've seen gloss over the havoc this creates with a standard linux tool chain. You can process a LOT of data with simple tools like Gawk and bash pipelines. It turns out you can even scale those same processes all the way up to Hadoop streaming.
- leef 15y ago
  The use of JSON logging plus a tool like Record Stream [1] is very powerful and solves the tool chain issue. Recs is complimentary to standard unix tools as well. 1- http://search.cpan.org/~bernard/App-RecordStream-3.7.3/README.pod http://search.cpan.org/~bernard/App-RecordStream-3.7.3/READM...
- ntkachov 15y ago
  You make a very good point about unix tools. However, as a Javascript/JSON guy myself, I really like the way JSON works. And for a small site like mine, JSON would work much better out of the box than some sort of tab structure. I work with node.js which means I can console.log and then pipe into a log file. Any object sent into console.log are automatically converted to JSON. I can also do stack traces with JSON if I ever end up with some sort of nasty bugs. When you log Gb a day, absolutely, tabs are the way to go. But when you have a tiny little thing like mine, saving the effort for something more value adding is probably the better choice.
frsyuki 15y ago
We're also using Fluentd as well as original JSON-based logging libraries. Fluentd deals with JSON-based logs. JSON is good for human facing interface because it is human readable and GREPable. On the other side, Fluentd handles logs in MessagePack format internally. Msgpack is a serialization format compatible with JSON and can be an efficient replacement of JSON. I wrote plugin for Fluentd that send those structured logs to Librato Metrics (https://metrics.librato.com/ https://metrics.librato.com/) which provides charting and dashboard features. With Fluentd, our logs became program-friendly as well as human-firnedly.
Ixiaus 15y ago
I dunno, this feels like the "web developers" approach to logging. I can't say that it wouldn't be cool to be able to parse logs into a structured format, but honestly, tools are already there to parse logs that are very powerful (gawk + shell pipes + {whatever_unix_tool_you_can_think_of}). If you don't have programmers that can knock out a real quick awk liner to process a logfile for you in any custom way you want, then I could see where this approach (using JSON) is useful because then they can use something they are familiar with instead of something they are not. But really, you should know how to use the Unix tool chain if you're a programmer.
thezilch 15y ago
Or provide unit test for said log parser and require (or don't) all tests to pass pre-commit. A JSON struct isn't going to stop your colleague from removing nor renaming a field. Removing the logging all together. Or changing the format himself, if your company is really setup for allowing colleagues to so easily break your code -- not that anyone's perfect.
deleted 15y ago
[deleted]
sciurus 15y ago
There's been a lot of noise about logging in the linux ecosystem lately. There's Project Lumberjack (http://bazsi.blogs.balabit.com/2012/02/project-lumberjack-to-improve-linux-logging/ http://bazsi.blogs.balabit.com/2012/02/project-lumberjack-to...) to encourage applications to generate structured logs and better document/integrate tools for working with those logs. The proposed structure is Common Event Expression (http://cee.mitre.org/ http://cee.mitre.org/). At the last kernel summit, ideas (http://lwn.net/Articles/492125/ http://lwn.net/Articles/492125/) were presented on how to make kernel messages more structured. More radically, there's The Journal (http://lwn.net/Articles/468049/ http://lwn.net/Articles/468049/), a proposed replacement for syslog.
- oasisbob 15y ago
  Also, the newest syslog RFC (http://tools.ietf.org/html/rfc5424 http://tools.ietf.org/html/rfc5424) allows for structured data to be included with the message. The nice thing about this approach is that you can serialize the received messages however you'd like: JSON, XML, TSV, whatever. Support for RFC 5424 in syslog daemons and logging libraries is thin, but will hopefully improve soon.
skrebbel 15y ago
The real takeaway is that log files invariably tend to become interfaces for something. They often end up being used for monitoring tools, business intelligence, system diagnostic tools, system tests, and so on. And they're great for this. But not when they contain sentences like "Opening conection...", which break half those tools the moment someone fixes the typo. The log strings became an interface. Avoid this. If it's an interface, it has to be specced, and it has to allow for backward compatibility, just like any other interface that crosses component / tool boundaries. Whether you do the actual data storage with JSON or something else doesn't matter. It's an implementation detail (though I agree that keeping it not only machine-readable, but also human-readable, is probably a good thing). Design the classes that represent log files, and treat them like they're part of a library API. Don't remove fields. Ideally, use the same classes for writing (from your main software) and parsing the logs (from all that other tooling) and include version information in the parser so that the class interface can be current yet the data can be ancient.
- gbog 15y ago
  That exactly the opposite of my understanding of logs, and the reason why I can't agree with the OP. For me logs are a way to store extensive historical data about what happened, in the cheapest and simplest way possible. Logs are a "write a lot and read almost never" kind of storage. For this kind of storage, the simplest way to do it is tab delimited flat files. Logs are for debug or "legal" purposes: a client complains he has lost all his data, your boss comes in the room with fume steaming outside his ears telling you names because you "can't store f* client's F* valuables cleany". Then, using awk or grep kungfu, you come up ten minutes later with the exact millisec when the client did click on "Yes, I am sure I want to delete my data", his IP, his session_id, his browser fingerprint and so on. In case of security audit, you are required to send 10 years of server logs to a trusted third-party (they will not do anything with it, they just want to make sure you have logs). You zip the thing and send it by email to then (thus crashing security auditor's email server). You have done your duty. These are what logs are for. Having JSON or any other format inside just make it more fragile, less versatile, will mess with line-oriented commands and is unnecessary. If you are using your logs regularly to track some business data about your product, you are using logs for the wrong purpose and should consider using something else.
anonymoushn 15y ago
Is it worth switching to JSON to avoid having to edit your bash 1-liner when you change the format of the log?
mmphosis 15y ago
Log Many things as [my favorite format]. Make My Life Easier by doing the difficult work. I would log in a fast compact, but not limited, and heavily documented binary format at a hardware level with lots of fail-safes. Maybe what I am doing is more appropriately called creating a journal. [My favorite scheduler] would very lazily and at opportunistic idle times convert the older non-human readable binary logs and insert the log data into [my favorite] database as very query-friendly information.
- chronomex 15y ago
  I would log in a fast compact, but not limited, and heavily documented binary format Such as ASN.1, perhaps?
drhowarddrfine 15y ago
Kids these days.
zmj 15y ago
This idea is as old as Lisp. http://sites.google.com/site/steveyegge2/the-emacs-problem http://sites.google.com/site/steveyegge2/the-emacs-problem
Simpletoon 15y ago
I only need three programs to deal with anti-ASCII, pro-complexity JSON, XML, etc. crowd: tr, sed and lex. All the effort these Javascripters expend putting data JSON just gets undone by my custom UNIX-style filters; then I can actually work with the text. Are they making life easier? For who? Seems like it's just more work for everybody, translating text back and forth into myriad formats. But what can you do?
- deno 15y ago
  Your “plain text” probably has some implicit structure. XML, JSON, ProtocolBuffers, just make that structure explicit. Dropping to plain text only to run sed or grep is a classic case of “if you have a hammer….” XML has a myriad of tools that do make your life easier — you just need to learn to embrace them.
  - Simpletoon 15y ago
    It all starts as a stream. That is the "universal format". sed was designed to edit streams. A stream can be transformed via stream editing into any text format, for any downstream consumer. It's line based. That's the only limitation. lex can handle multiline "records". There is nothing you cannot do with lex. But only if you know how to use it. It's usually faster than any scripting language. Worth learning to use? Your choice. But it is what it is. It works. It, or some clone of it, was used to build the compiler that someone used to compile the shared library you're using as part of your special solution for the format of the month. If you produce a stream as JSON, that's great. But now we're limited to consumers that understand JSON. If you know your consumer wants JSON, then sure use some specialised library. But that's not what this guy is suggesting. He wants everything in JSON. Well, not every consumer wants JSON. This is a case of "I learned [X]. Please everyone use [X]." None of us want to have to learn every language and every application. Now consider if [X] is UNIX. For better or worse, it's the foundation on which most stuff talked about here runs. Perhaps it seems crude, it lacks sophistication in the eyes of a younger generation. It's a "hammer". But what can you build without a "hammer"? In his case, [X] is Javascript. What's the foundation for Javascript? A "web browser". Perhaps some people think nothing is possible without a web browser that can run Javascript. It's a very narrow view.
sauravc 15y ago
We've been logging all of our analytics data in JSON for years now.
leif 15y ago
TSV and you're done smaller readable, esp. with `column -t <log` works with awk/cut/join/grep/sort/column/etc./etc./etc. if you have complicated enough logs that you can't maintain the shell scripts that parse them, you probably also have enough log data that json's going to blow up your space and you probably want indexes anyway, so throw it in a real database (oh hi I work for one of these, log analysis is actually one of our strong suits) but others have already commented to this effect
majmun 15y ago
I tried this , it was no good ( because of escaping of special characters, and parsing performance. ) then i switch to newline and n r . and all my problems were solved (for now)
jakejake 15y ago
I've done various different log formats over the years including JSON. One thing I've done for logging errors or warning is to log them in RSS format. I monitor them just like any RSS feed. It's really handy because there's already tons of ways to read these logs so we don't have to create anything. I wouldn't use this for a debug log because it would probably be unusable if there was a large volume of logs, but for watching errors it's great.
a3_nm 15y ago
What if I need, say, to find the 10 IPs that make the most requests? With the Apache log format, I can write the following in about 15 seconds: cut -d ' ' -f1 log | sort | uniq -c | sort -nr | head Say you need to follow accesses to a particular file? The following quick and dirty one-liner probably works well enough: tail -f log | grep --line-buffered file.pdf How do you do that with json? Granted, as soon as your logs stop being a sequence of records (lines) with a fixed sequence of neatly delimited records, you will need something more than text. However, I still don't know of tools to work with json from the command line that are as concise, efficient, flexible and robust as the standard unix utilities for text.
- tim_sw 15y ago
  recordStream provides some of these features. Take a look here https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream
- deno 15y ago
  Not JSON, but one vision as to how structured data could fit into UNIX ecosystem: http://xml-coreutils.sourceforge.net/ http://xml-coreutils.sourceforge.net/ In your current pipeline just replace `cut` with an XPath expression, everything else stays practically the same. Of course current XML ecosystem has different tools that makes your case trivial, though that’s something new to learn.
- keenerd 15y ago
  > How do you do that with json? Depends on the schema used, but it would probably be something like jshon -a -e "ip" -u < log | sort | uniq -c | sort -nr | head
- gravitystorm 15y ago
  The problem with your first command is the "cut" - works great for a while. Then you deal with other_vhosts_combined.log, and there's a leading hostname and port, so you want -f2 instead. After that, you start working with the user-agent field, and after remembering that the date and request fields have internal spaces, settle for something like -f13, which works until someone puts a space in their faked-up referer, and you can't use cut any more. I love plain-text logs, but that apache logformat is a bad example of "neatly delimited records", given the unstable mixture of spaces between and within fields (some of which are quoted, some not, and one even quoted with square brackets!).
Hopka 15y ago
How do you even log as JSON? Is your entire log file a giant JSON array? That would be challenging for most parsers I know because they would have to read the entire array into memory first. Or do you log one JSON object per line? Then you would get problems as soon as you have line breaks inside strings and still have to parse until the object ends in some other line. Also, JSON objects do not have to be single-line to be valid, so you would in fact be working with some self-defined subset of JSON.
- drostie 15y ago
  Is your entire log file a giant JSON array? That's not necessarily a bad choice, but it's problematic for the reasons you describe. Still, since JSON is concatenative, you could indeed store all of your objects with a comma at the end and then use: import json; json.loads("[%s null]" % file_contents)[:-1] Then you would get problems as soon as you have line breaks inside strings Then you are not logging JSON. JSON does not permit that. Also, JSON objects do not have to be single-line to be valid, so you would in fact be working with some self-defined subset of JSON. Yes, and? Working in a subset of JSON which forbids newlines as whitespace -- that's still JSON, and it solves your problem elegantly. Do... do you have multiple logging programs, logging to the same file, and one of them wants to insert newlines? Is this a real problem in your dev stack?
  - Hopka 15y ago
    It is a real problem in someone's dev stack. I have already written code to preprocess such log files before feeding chunks into a real JSON parser. It didn't make my life easier. I'm not against logging as JSON at all, but as pointed out, you have to use a subset that makes parsing the logs easy.
webjunkie 15y ago
Okay, and as soon as I switch to JSON, I have not just 5 million referrers logged per day, I also have 5 million times the word "referrer" in my log. Nice.
- deno 15y ago
  Trivially solved by compression. In those five million referrers you’re probably also repeating “Firefox” and similar strings over and over, so compressing logs is already standard practice.
  - webjunkie 14y ago
    Compressing logs that are being written to? Never heard of that...
dasil003 15y ago
Loggly supports this, and they provided a good interface for querying the data as well. We used it for a while as a way to unify a couple GB of daily log data from our Rails app running on multiple instances. I even wrote a library that allows you to quickly add arbitrary keys to the request log entry anywhere in the app. Unfortunately we had to disable it temporarily as the Ruby client did not cope well when latency increased to the Loggly service. It was fine for a while since we are both on AWS, but one day our site started getting super slow. It took a while to track down the problem because the Loggly client has threaded delivery, so a given request would not be delayed. But the problem was that the next request couldn't be started until the delivery thread terminated. Okay I realize this is not the best architecture. There should be a completely isolated process that's pushing the queued logs to Loggly so that the app never deals with anything but a local logging service. Loggly supports syslogng, but that would be standard logging not JSON, so I think if we want to go this route we need to come up with something on our own...
delinka 15y ago
"Alex ... [realizes] that someone added an extra field in each line" Someone?!? Who's touching the server configuration and why? Unless Alex put a publicly accessible web interface on his .conf files, this shouldn't be happening. Back on topic. The increase in size for logging in JSON could easily be a deal breaker.
- pbiggar 15y ago
  OT, but I really like the way you expressed this: "could easily be a deal breaker". Too many comments on HN instead say "This would never work because of the increase in size" instead. Your comment recognizes the nuance without getting bogged down in what-ifs and disclaimers. Very well put.
- cdmoyer 15y ago
  Maybe Joe or Tina the other developers. Or maybe it was Mark or Cheryl from systems. That said, in my experience, this doesn't end up being a common issue because people do the sensible thing and add the new field to the end of the file, so your tools just continue to work.
wooptoo 15y ago
Why not go even further and store them in MongoDB?
joelthelion 15y ago
I would love to see a JSON based shell, instead of the traditional shells based on raw strings. Heck, we could have a whole ecosystem of tools built around JSON or similar semi-structured representations.
kablamo 15y ago
I've been thinking about this recently as well. I wrote a simple JSON logger for Perl recently. It will probably be on CPAN this weekend. Until then you can see it on prepan and github. prepan http://prepan.org/module/3Yz7PYrBSd http://prepan.org/module/3Yz7PYrBSd github https://github.com/kablamo/Log-JSON https://github.com/kablamo/Log-JSON
wolframarnold 15y ago
I like this idea a lot. Frameworks like Rails come with excellent log messages, granularity and a pub/sub mechanism. Often this can be a lower hanging fruit than throwing in a ton of custom instrumentation for some third party analytics tool, especially when you're pressed for time. My question is how fluentd can be hooked into Rails so that Rails' native messages use it and how does it work in the Heroku infrastructure?

155 points by kiyoto 15y ago

Except that fixed-field loglines are much faster to process than parsing JSON, which makes a difference when working with large logs.

kelnos 15y ago
Depends on what you value more: CPU time, or your developers' time keeping the log format and parsers in sync after any change.
- mkross 15y ago
  You could test that the logging and parsing are in step, which then reduces the developer time to "Doh, red test" and a simple fix.
  - LeafStorm 15y ago
    So...integrate quick scripts to get a count of IP addresses into your app's test suite?
  - kelnos 14y ago
    Why increase your burden even to that? Computers are there to do stupid repetitive stuff for us so we don't have to. Why design a logging format that requires even minimal manual intervention to keep automated parsing working?

rachelbythebay 15y ago

This article feels like it would work just as well with "Protocol Buffers", "Thrift", "XML", or even maybe "ASN.1". If that's truly the case, maybe the better thing to say is "please don't (only) log in ASCII", followed by "please use a format which is hard to get wrong". JSON scares me a little. Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? I saw a page not too long ago which listed about a dozen characters which should be substituted in some manner when used in JSON. Full disclosure: I got tired of ASCII logging from my web server and wrote something to stream binary protocol buffers (!) to a file instead. http://rachelbythebay.com/w/2012/02/12/progress/ http://rachelbythebay.com/w/2012/02/12/progress/

djcjr 15y ago
what, no mention of sexp's?
- kiyoto 15y ago
  djcjr: Are you an OCaml user ;p?
kelnos 15y ago
Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? Yes, but you don't do it yourself. As the article mentions, pretty much every language you'd want to use has a JSON library. Use that, and (bugs aside) it'll take care of escaping for you.
- rachelbythebay 15y ago
  I sure hope everyone uses a library. What worries me is that it looks so simple, so someone might just glue together their own encoding and call it "done". A higher barrier to entry isn't always a bad thing.
kiyoto 15y ago
rachelbythebay: Yea, you've got some good points. I don't think the author means JSON is the be-all-end-all solution to structured logging. It has its issues (escaping for one), but I think it is a reasonable choice because of its wide adoption. A nice use of protocol buffer there. Speaking of serialization, my personal favorite is MessagePack (httpx://github.com/msgpack).
- deleted 15y ago
  [deleted]
frsyuki 15y ago
There are binary-based serialization formats such as MessagePack [1] or BSON [2] that doesn't need escapeing overhead. You can simply replace JSON with MessagePack since MessagePack is compatible with JSON. [1] Why Not MessagePack? http://blog.andrewvc.com/why-arent-you-using-messagepack http://blog.andrewvc.com/why-arent-you-using-messagepack [2] Performant Entity Serialization: BSON vs MessagePack (vs JSON) http://stackoverflow.com/questions/6355497/performant-entity-serialization-bson-vs-messagepack-vs-json http://stackoverflow.com/questions/6355497/performant-entity...

daenz 15y ago

Logging to mongo (as JSON) has proven useful to us. Makes it easy to slice and dice the data.

asuth 15y ago
Ya, logs are soooo much more valuable when they're in a database that you can query. We log to MySQL and then correlate logs with users, events, IPs, pages, etc. Logs are just as relational as anything else, it took me awhile to realize it though.

jacques_chester 15y ago

One of the non-functional requirements of logs is that they should be fast to write. Marshalling data into a structured format takes longer than spitting out sprintfs. If you really need structure for ease of querying, you might as well go all the way and throw it into a proper data store.

frsyuki 15y ago
Spitting out sprintfs means parsing texts. JSON might be slow since it's text, but binary based formats should be faster especially the log includes many integers.
- jacques_chester 15y ago
  sprintf tends to be a faster path than "build tree, render down to text".
  - deno 15y ago
    Both XML and JSON can be streamed. No tree building necessary.
gorset 15y ago
Remember we're talking about writing to disk. The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. We use JSON on disk heavily in several places where we generate or receive huge amounts of data. JSON-in-a-file is a pretty good datastore for sequential data processing because it's so convent and you can work with the data using cat, tail or any tool supporting JSON.
- stevewilhelm 15y ago
  You're still logging to local disk?
- jacques_chester 15y ago
  > The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. And JSON is more verbose. Basically this is another read/write dichotomy. When do we pay the price of structuring data? At write time? Or at read time? I'd rather pay it at read time, as writing may be shaving performance from my actual primary production system.

NyxWulf 15y ago

I've seen several articles like this, and there are a number of things to consider. Logging to ascii means that the standard unix tools work out of the box with your log files. Especially if you use something like a tab delimiter, then you typically don't need to specify the delimiter. As an upside you are aren't storing the column definition in every single line, which if you are doing large volume traffic definitely matters. For instance we store gigabytes of log files per hour, grossing up that space by a significant margin impacts storage, transit and process times during write (marshallers and custom log formatting). Writes are the hardest to scale, so if I'm going to add scale or extra parsing time, I'd rather handle that in Hadoop where I can throw massive parallel resources at it. Next you can achieve much of the advantages of json or protocol buffers by having a defined format and a structured release process before someone can change the format. Add fields to the end and don't remove defunct fields. This is the same process you have to use with protocol buffers or conceptually with JSON to have it work. Overall there are advantages to these other formats, but the articles like this that I've seen gloss over the havoc this creates with a standard linux tool chain. You can process a LOT of data with simple tools like Gawk and bash pipelines. It turns out you can even scale those same processes all the way up to Hadoop streaming.

leef 15y ago
The use of JSON logging plus a tool like Record Stream [1] is very powerful and solves the tool chain issue. Recs is complimentary to standard unix tools as well. 1- http://search.cpan.org/~bernard/App-RecordStream-3.7.3/README.pod http://search.cpan.org/~bernard/App-RecordStream-3.7.3/READM...
ntkachov 15y ago
You make a very good point about unix tools. However, as a Javascript/JSON guy myself, I really like the way JSON works. And for a small site like mine, JSON would work much better out of the box than some sort of tab structure. I work with node.js which means I can console.log and then pipe into a log file. Any object sent into console.log are automatically converted to JSON. I can also do stack traces with JSON if I ever end up with some sort of nasty bugs. When you log Gb a day, absolutely, tabs are the way to go. But when you have a tiny little thing like mine, saving the effort for something more value adding is probably the better choice.

frsyuki 15y ago

We're also using Fluentd as well as original JSON-based logging libraries. Fluentd deals with JSON-based logs. JSON is good for human facing interface because it is human readable and GREPable. On the other side, Fluentd handles logs in MessagePack format internally. Msgpack is a serialization format compatible with JSON and can be an efficient replacement of JSON. I wrote plugin for Fluentd that send those structured logs to Librato Metrics (https://metrics.librato.com/ https://metrics.librato.com/) which provides charting and dashboard features. With Fluentd, our logs became program-friendly as well as human-firnedly.

Ixiaus 15y ago

I dunno, this feels like the "web developers" approach to logging. I can't say that it wouldn't be cool to be able to parse logs into a structured format, but honestly, tools are already there to parse logs that are very powerful (gawk + shell pipes + {whatever_unix_tool_you_can_think_of}). If you don't have programmers that can knock out a real quick awk liner to process a logfile for you in any custom way you want, then I could see where this approach (using JSON) is useful because then they can use something they are familiar with instead of something they are not. But really, you should know how to use the Unix tool chain if you're a programmer.

thezilch 15y ago

Or provide unit test for said log parser and require (or don't) all tests to pass pre-commit. A JSON struct isn't going to stop your colleague from removing nor renaming a field. Removing the logging all together. Or changing the format himself, if your company is really setup for allowing colleagues to so easily break your code -- not that anyone's perfect.

deleted 15y ago

[deleted]

sciurus 15y ago

There's been a lot of noise about logging in the linux ecosystem lately. There's Project Lumberjack (http://bazsi.blogs.balabit.com/2012/02/project-lumberjack-to-improve-linux-logging/ http://bazsi.blogs.balabit.com/2012/02/project-lumberjack-to...) to encourage applications to generate structured logs and better document/integrate tools for working with those logs. The proposed structure is Common Event Expression (http://cee.mitre.org/ http://cee.mitre.org/). At the last kernel summit, ideas (http://lwn.net/Articles/492125/ http://lwn.net/Articles/492125/) were presented on how to make kernel messages more structured. More radically, there's The Journal (http://lwn.net/Articles/468049/ http://lwn.net/Articles/468049/), a proposed replacement for syslog.

oasisbob 15y ago
Also, the newest syslog RFC (http://tools.ietf.org/html/rfc5424 http://tools.ietf.org/html/rfc5424) allows for structured data to be included with the message. The nice thing about this approach is that you can serialize the received messages however you'd like: JSON, XML, TSV, whatever. Support for RFC 5424 in syslog daemons and logging libraries is thin, but will hopefully improve soon.

skrebbel 15y ago

The real takeaway is that log files invariably tend to become interfaces for something. They often end up being used for monitoring tools, business intelligence, system diagnostic tools, system tests, and so on. And they're great for this. But not when they contain sentences like "Opening conection...", which break half those tools the moment someone fixes the typo. The log strings became an interface. Avoid this. If it's an interface, it has to be specced, and it has to allow for backward compatibility, just like any other interface that crosses component / tool boundaries. Whether you do the actual data storage with JSON or something else doesn't matter. It's an implementation detail (though I agree that keeping it not only machine-readable, but also human-readable, is probably a good thing). Design the classes that represent log files, and treat them like they're part of a library API. Don't remove fields. Ideally, use the same classes for writing (from your main software) and parsing the logs (from all that other tooling) and include version information in the parser so that the class interface can be current yet the data can be ancient.

gbog 15y ago
That exactly the opposite of my understanding of logs, and the reason why I can't agree with the OP. For me logs are a way to store extensive historical data about what happened, in the cheapest and simplest way possible. Logs are a "write a lot and read almost never" kind of storage. For this kind of storage, the simplest way to do it is tab delimited flat files. Logs are for debug or "legal" purposes: a client complains he has lost all his data, your boss comes in the room with fume steaming outside his ears telling you names because you "can't store f* client's F* valuables cleany". Then, using awk or grep kungfu, you come up ten minutes later with the exact millisec when the client did click on "Yes, I am sure I want to delete my data", his IP, his session_id, his browser fingerprint and so on. In case of security audit, you are required to send 10 years of server logs to a trusted third-party (they will not do anything with it, they just want to make sure you have logs). You zip the thing and send it by email to then (thus crashing security auditor's email server). You have done your duty. These are what logs are for. Having JSON or any other format inside just make it more fragile, less versatile, will mess with line-oriented commands and is unnecessary. If you are using your logs regularly to track some business data about your product, you are using logs for the wrong purpose and should consider using something else.

anonymoushn 15y ago

Is it worth switching to JSON to avoid having to edit your bash 1-liner when you change the format of the log?

mmphosis 15y ago

Log Many things as [my favorite format]. Make My Life Easier by doing the difficult work. I would log in a fast compact, but not limited, and heavily documented binary format at a hardware level with lots of fail-safes. Maybe what I am doing is more appropriately called creating a journal. [My favorite scheduler] would very lazily and at opportunistic idle times convert the older non-human readable binary logs and insert the log data into [my favorite] database as very query-friendly information.

chronomex 15y ago
I would log in a fast compact, but not limited, and heavily documented binary format Such as ASN.1, perhaps?

drhowarddrfine 15y ago

Kids these days.

zmj 15y ago

This idea is as old as Lisp. http://sites.google.com/site/steveyegge2/the-emacs-problem http://sites.google.com/site/steveyegge2/the-emacs-problem

Simpletoon 15y ago

I only need three programs to deal with anti-ASCII, pro-complexity JSON, XML, etc. crowd: tr, sed and lex. All the effort these Javascripters expend putting data JSON just gets undone by my custom UNIX-style filters; then I can actually work with the text. Are they making life easier? For who? Seems like it's just more work for everybody, translating text back and forth into myriad formats. But what can you do?

deno 15y ago
Your “plain text” probably has some implicit structure. XML, JSON, ProtocolBuffers, just make that structure explicit. Dropping to plain text only to run sed or grep is a classic case of “if you have a hammer….” XML has a myriad of tools that do make your life easier — you just need to learn to embrace them.
- Simpletoon 15y ago
  It all starts as a stream. That is the "universal format". sed was designed to edit streams. A stream can be transformed via stream editing into any text format, for any downstream consumer. It's line based. That's the only limitation. lex can handle multiline "records". There is nothing you cannot do with lex. But only if you know how to use it. It's usually faster than any scripting language. Worth learning to use? Your choice. But it is what it is. It works. It, or some clone of it, was used to build the compiler that someone used to compile the shared library you're using as part of your special solution for the format of the month. If you produce a stream as JSON, that's great. But now we're limited to consumers that understand JSON. If you know your consumer wants JSON, then sure use some specialised library. But that's not what this guy is suggesting. He wants everything in JSON. Well, not every consumer wants JSON. This is a case of "I learned [X]. Please everyone use [X]." None of us want to have to learn every language and every application. Now consider if [X] is UNIX. For better or worse, it's the foundation on which most stuff talked about here runs. Perhaps it seems crude, it lacks sophistication in the eyes of a younger generation. It's a "hammer". But what can you build without a "hammer"? In his case, [X] is Javascript. What's the foundation for Javascript? A "web browser". Perhaps some people think nothing is possible without a web browser that can run Javascript. It's a very narrow view.

sauravc 15y ago

We've been logging all of our analytics data in JSON for years now.

leif 15y ago

TSV and you're done smaller readable, esp. with `column -t <log` works with awk/cut/join/grep/sort/column/etc./etc./etc. if you have complicated enough logs that you can't maintain the shell scripts that parse them, you probably also have enough log data that json's going to blow up your space and you probably want indexes anyway, so throw it in a real database (oh hi I work for one of these, log analysis is actually one of our strong suits) but others have already commented to this effect

majmun 15y ago

I tried this , it was no good ( because of escaping of special characters, and parsing performance. ) then i switch to newline and n r . and all my problems were solved (for now)

jakejake 15y ago

I've done various different log formats over the years including JSON. One thing I've done for logging errors or warning is to log them in RSS format. I monitor them just like any RSS feed. It's really handy because there's already tons of ways to read these logs so we don't have to create anything. I wouldn't use this for a debug log because it would probably be unusable if there was a large volume of logs, but for watching errors it's great.

a3_nm 15y ago

What if I need, say, to find the 10 IPs that make the most requests? With the Apache log format, I can write the following in about 15 seconds: cut -d ' ' -f1 log | sort | uniq -c | sort -nr | head Say you need to follow accesses to a particular file? The following quick and dirty one-liner probably works well enough: tail -f log | grep --line-buffered file.pdf How do you do that with json? Granted, as soon as your logs stop being a sequence of records (lines) with a fixed sequence of neatly delimited records, you will need something more than text. However, I still don't know of tools to work with json from the command line that are as concise, efficient, flexible and robust as the standard unix utilities for text.

tim_sw 15y ago
recordStream provides some of these features. Take a look here https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream
deno 15y ago
Not JSON, but one vision as to how structured data could fit into UNIX ecosystem: http://xml-coreutils.sourceforge.net/ http://xml-coreutils.sourceforge.net/ In your current pipeline just replace `cut` with an XPath expression, everything else stays practically the same. Of course current XML ecosystem has different tools that makes your case trivial, though that’s something new to learn.
keenerd 15y ago
> How do you do that with json? Depends on the schema used, but it would probably be something like jshon -a -e "ip" -u < log | sort | uniq -c | sort -nr | head
gravitystorm 15y ago
The problem with your first command is the "cut" - works great for a while. Then you deal with other_vhosts_combined.log, and there's a leading hostname and port, so you want -f2 instead. After that, you start working with the user-agent field, and after remembering that the date and request fields have internal spaces, settle for something like -f13, which works until someone puts a space in their faked-up referer, and you can't use cut any more. I love plain-text logs, but that apache logformat is a bad example of "neatly delimited records", given the unstable mixture of spaces between and within fields (some of which are quoted, some not, and one even quoted with square brackets!).

Hopka 15y ago

How do you even log as JSON? Is your entire log file a giant JSON array? That would be challenging for most parsers I know because they would have to read the entire array into memory first. Or do you log one JSON object per line? Then you would get problems as soon as you have line breaks inside strings and still have to parse until the object ends in some other line. Also, JSON objects do not have to be single-line to be valid, so you would in fact be working with some self-defined subset of JSON.

drostie 15y ago
Is your entire log file a giant JSON array? That's not necessarily a bad choice, but it's problematic for the reasons you describe. Still, since JSON is concatenative, you could indeed store all of your objects with a comma at the end and then use: import json; json.loads("[%s null]" % file_contents)[:-1] Then you would get problems as soon as you have line breaks inside strings Then you are not logging JSON. JSON does not permit that. Also, JSON objects do not have to be single-line to be valid, so you would in fact be working with some self-defined subset of JSON. Yes, and? Working in a subset of JSON which forbids newlines as whitespace -- that's still JSON, and it solves your problem elegantly. Do... do you have multiple logging programs, logging to the same file, and one of them wants to insert newlines? Is this a real problem in your dev stack?
- Hopka 15y ago
  It is a real problem in someone's dev stack. I have already written code to preprocess such log files before feeding chunks into a real JSON parser. It didn't make my life easier. I'm not against logging as JSON at all, but as pointed out, you have to use a subset that makes parsing the logs easy.

webjunkie 15y ago

Okay, and as soon as I switch to JSON, I have not just 5 million referrers logged per day, I also have 5 million times the word "referrer" in my log. Nice.

deno 15y ago
Trivially solved by compression. In those five million referrers you’re probably also repeating “Firefox” and similar strings over and over, so compressing logs is already standard practice.
- webjunkie 14y ago
  Compressing logs that are being written to? Never heard of that...

dasil003 15y ago

Loggly supports this, and they provided a good interface for querying the data as well. We used it for a while as a way to unify a couple GB of daily log data from our Rails app running on multiple instances. I even wrote a library that allows you to quickly add arbitrary keys to the request log entry anywhere in the app. Unfortunately we had to disable it temporarily as the Ruby client did not cope well when latency increased to the Loggly service. It was fine for a while since we are both on AWS, but one day our site started getting super slow. It took a while to track down the problem because the Loggly client has threaded delivery, so a given request would not be delayed. But the problem was that the next request couldn't be started until the delivery thread terminated. Okay I realize this is not the best architecture. There should be a completely isolated process that's pushing the queued logs to Loggly so that the app never deals with anything but a local logging service. Loggly supports syslogng, but that would be standard logging not JSON, so I think if we want to go this route we need to come up with something on our own...

delinka 15y ago

"Alex ... [realizes] that someone added an extra field in each line" Someone?!? Who's touching the server configuration and why? Unless Alex put a publicly accessible web interface on his .conf files, this shouldn't be happening. Back on topic. The increase in size for logging in JSON could easily be a deal breaker.

pbiggar 15y ago
OT, but I really like the way you expressed this: "could easily be a deal breaker". Too many comments on HN instead say "This would never work because of the increase in size" instead. Your comment recognizes the nuance without getting bogged down in what-ifs and disclaimers. Very well put.
cdmoyer 15y ago
Maybe Joe or Tina the other developers. Or maybe it was Mark or Cheryl from systems. That said, in my experience, this doesn't end up being a common issue because people do the sensible thing and add the new field to the end of the file, so your tools just continue to work.

wooptoo 15y ago

Why not go even further and store them in MongoDB?

joelthelion 15y ago

I would love to see a JSON based shell, instead of the traditional shells based on raw strings. Heck, we could have a whole ecosystem of tools built around JSON or similar semi-structured representations.

kablamo 15y ago

I've been thinking about this recently as well. I wrote a simple JSON logger for Perl recently. It will probably be on CPAN this weekend. Until then you can see it on prepan and github. prepan http://prepan.org/module/3Yz7PYrBSd http://prepan.org/module/3Yz7PYrBSd github https://github.com/kablamo/Log-JSON https://github.com/kablamo/Log-JSON

wolframarnold 15y ago

I like this idea a lot. Frameworks like Rails come with excellent log messages, granularity and a pub/sub mechanism. Often this can be a lower hanging fruit than throwing in a ton of custom instrumentation for some third party analytics tool, especially when you're pressed for time. My question is how fluentd can be hooked into Rails so that Rails' native messages use it and how does it work in the Heroku infrastructure?