6 ms·

This article feels like it would work just as well with "Protocol Buffers", "Thrift", "XML", or even maybe "ASN.1". If that's truly the case, maybe the better

This article feels like it would work just as well with "Protocol Buffers", "Thrift", "XML", or even maybe "ASN.1". If that's truly the case, maybe the better thing to say is "please don't (only) log in ASCII", followed by "please use a format which is hard to get wrong". JSON scares me a little. Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? I saw a page not too long ago which listed about a dozen characters which should be substituted in some manner when used in JSON. Full disclosure: I got tired of ASCII logging from my web server and wrote something to stream binary protocol buffers (!) to a file instead. http://rachelbythebay.com/w/2012/02/12/progress/ http://rachelbythebay.com/w/2012/02/12/progress/

djcjr 15y ago
what, no mention of sexp's?
- kiyoto 15y ago
  djcjr: Are you an OCaml user ;p?
kelnos 15y ago
Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? Yes, but you don't do it yourself. As the article mentions, pretty much every language you'd want to use has a JSON library. Use that, and (bugs aside) it'll take care of escaping for you.
- rachelbythebay 15y ago
  I sure hope everyone uses a library. What worries me is that it looks so simple, so someone might just glue together their own encoding and call it "done". A higher barrier to entry isn't always a bad thing.
kiyoto 15y ago
rachelbythebay: Yea, you've got some good points. I don't think the author means JSON is the be-all-end-all solution to structured logging. It has its issues (escaping for one), but I think it is a reasonable choice because of its wide adoption. A nice use of protocol buffer there. Speaking of serialization, my personal favorite is MessagePack (httpx://github.com/msgpack).
- deleted 15y ago
  [deleted]
frsyuki 15y ago
There are binary-based serialization formats such as MessagePack [1] or BSON [2] that doesn't need escapeing overhead. You can simply replace JSON with MessagePack since MessagePack is compatible with JSON. [1] Why Not MessagePack? http://blog.andrewvc.com/why-arent-you-using-messagepack http://blog.andrewvc.com/why-arent-you-using-messagepack [2] Performant Entity Serialization: BSON vs MessagePack (vs JSON) http://stackoverflow.com/questions/6355497/performant-entity-serialization-bson-vs-messagepack-vs-json http://stackoverflow.com/questions/6355497/performant-entity...
- simonw 15y ago
  Because using a binary format prevents you from tailing your log file without additional tools.
lpolovets 15y ago
Over the last 10 years, I've gradually moved from ASCII to Thrift/ProtocolBuffers to JSON. Here are some random thoughts: 1) As others said, you should definitely use a library for JSON reading/writing. 2) JSON is simple and surprisingly fast to read/write. In one large benchmark (http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking http://code.google.com/p/thrift-protobuf-compare/wiki/Benchm...), the Jackson JSON library (for Java) comes in somewhere between Thrift and Protocol Buffers. 3) JSON feels friendlier than protocol buffers and Thrift because it's human readable instead of binary. 4) JSON is more convenient than multi-line formats like XML because you can grep for things easily. For example, if you have {time:015128752, event:"authentication error", user:"bob smith", details: "..."}, then I can take do the following: cat log.json | grep 'user:"bob smith"' It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines. 5) JSON is not as compact as binary, but it gets surprisingly close if you gzip it. All of those repeated descriptive attribute names are great for humans, but compression algorithms love them as well. 6) The format is fairly universal, but if you ever come across a language without JSON parsing libraries, the format is not so hard that you couldn't write your own parser (compare that to Thrift, XML, etc)
- LeafStorm 15y ago
  Fun fact: `user:"bob smith"` is invalid JSON because object keys must always be quoted. The JSON spec is very precise, you can find it right on the front page of http://json.org/ http://json.org/.
- tim_sw 15y ago
  You can also use tools like recordStream, which allow you to do grep, etc. on individual json fields. https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream (It has a lot of other features as well) Sidenote: benbernard worked on Amazon dev tools team
- deno 15y ago
  > JSON is more convenient than multi-line formats like XML because you can grep for things easily. Only if the text representation is pretty-printed in a certain way. > It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines. XML tooling makes grepping obsolete. cat log.xml | xpath //log[@user="bob smith"]
deleted 15y ago
[deleted]
jerf 15y ago
"Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field?" Either you're using a length-delimited format, which is pretty hostile for humans to read, or you're using a string that will require something to be escaped. The only third option is bugs, like a newline that gets out into your logs and oops, new log line. "I saw a page not too long ago which listed about a dozen characters which should be substituted in some manner when used in JSON." It is very likely what you saw was the list of Unicode characters that are accepted by the JSON format but rejected by browsers and/or HTML in the context of a Javascript embedded in an HTML page. In this case, it's the browsers and/or JS-in-HTML with the issue, not JSON. And I don't mean that as a who-to-blame sort of issue, I mean, if you're producing JSON and consuming JSON and it doesn't pass through HTML it's a non-issue, and any string that passes through JS-in-HTML will have the same problem even if it isn't "JSON", so it really hasn't got anything to do with JSON if I am correct about what you're referring to.
rogerbinns 15y ago
I'm happy with JSON and it is the native format of MongoDB, CouchDB and similar databases. And grep/sed/awk. The biggest problem with JSON is that it has no standard way of representing dates, times or binary data.

djcjr 15y ago

what, no mention of sexp's?

kiyoto 15y ago
djcjr: Are you an OCaml user ;p?

kelnos 15y ago

Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field? Yes, but you don't do it yourself. As the article mentions, pretty much every language you'd want to use has a JSON library. Use that, and (bugs aside) it'll take care of escaping for you.

rachelbythebay 15y ago
I sure hope everyone uses a library. What worries me is that it looks so simple, so someone might just glue together their own encoding and call it "done". A higher barrier to entry isn't always a bad thing.

kiyoto 15y ago

rachelbythebay: Yea, you've got some good points. I don't think the author means JSON is the be-all-end-all solution to structured logging. It has its issues (escaping for one), but I think it is a reasonable choice because of its wide adoption. A nice use of protocol buffer there. Speaking of serialization, my personal favorite is MessagePack (httpx://github.com/msgpack).

deleted 15y ago
[deleted]

frsyuki 15y ago

There are binary-based serialization formats such as MessagePack [1] or BSON [2] that doesn't need escapeing overhead. You can simply replace JSON with MessagePack since MessagePack is compatible with JSON. [1] Why Not MessagePack? http://blog.andrewvc.com/why-arent-you-using-messagepack http://blog.andrewvc.com/why-arent-you-using-messagepack [2] Performant Entity Serialization: BSON vs MessagePack (vs JSON) http://stackoverflow.com/questions/6355497/performant-entity-serialization-bson-vs-messagepack-vs-json http://stackoverflow.com/questions/6355497/performant-entity...

simonw 15y ago
Because using a binary format prevents you from tailing your log file without additional tools.

lpolovets 15y ago

Over the last 10 years, I've gradually moved from ASCII to Thrift/ProtocolBuffers to JSON. Here are some random thoughts: 1) As others said, you should definitely use a library for JSON reading/writing. 2) JSON is simple and surprisingly fast to read/write. In one large benchmark (http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking http://code.google.com/p/thrift-protobuf-compare/wiki/Benchm...), the Jackson JSON library (for Java) comes in somewhere between Thrift and Protocol Buffers. 3) JSON feels friendlier than protocol buffers and Thrift because it's human readable instead of binary. 4) JSON is more convenient than multi-line formats like XML because you can grep for things easily. For example, if you have {time:015128752, event:"authentication error", user:"bob smith", details: "..."}, then I can take do the following: cat log.json | grep 'user:"bob smith"' It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines. 5) JSON is not as compact as binary, but it gets surprisingly close if you gzip it. All of those repeated descriptive attribute names are great for humans, but compression algorithms love them as well. 6) The format is fairly universal, but if you ever come across a language without JSON parsing libraries, the format is not so hard that you couldn't write your own parser (compare that to Thrift, XML, etc)

LeafStorm 15y ago
Fun fact: `user:"bob smith"` is invalid JSON because object keys must always be quoted. The JSON spec is very precise, you can find it right on the front page of http://json.org/ http://json.org/.
tim_sw 15y ago
You can also use tools like recordStream, which allow you to do grep, etc. on individual json fields. https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream (It has a lot of other features as well) Sidenote: benbernard worked on Amazon dev tools team
deno 15y ago
> JSON is more convenient than multi-line formats like XML because you can grep for things easily. Only if the text representation is pretty-printed in a certain way. > It's hard to do that with XML because even if you find the right user, the entire log message/object spans multiple lines. XML tooling makes grepping obsolete. cat log.xml | xpath //log[@user="bob smith"]

deleted 15y ago

[deleted]

jerf 15y ago

"Don't you have to worry about escaping a whole bunch of characters just in case something gets the wrong idea about what you have in a field?" Either you're using a length-delimited format, which is pretty hostile for humans to read, or you're using a string that will require something to be escaped. The only third option is bugs, like a newline that gets out into your logs and oops, new log line. "I saw a page not too long ago which listed about a dozen characters which should be substituted in some manner when used in JSON." It is very likely what you saw was the list of Unicode characters that are accepted by the JSON format but rejected by browsers and/or HTML in the context of a Javascript embedded in an HTML page. In this case, it's the browsers and/or JS-in-HTML with the issue, not JSON. And I don't mean that as a who-to-blame sort of issue, I mean, if you're producing JSON and consuming JSON and it doesn't pass through HTML it's a non-issue, and any string that passes through JS-in-HTML will have the same problem even if it isn't "JSON", so it really hasn't got anything to do with JSON if I am correct about what you're referring to.

rogerbinns 15y ago

I'm happy with JSON and it is the native format of MongoDB, CouchDB and similar databases. And grep/sed/awk. The biggest problem with JSON is that it has no standard way of representing dates, times or binary data.