3 ms·

There's also ndjson[0], personally I feel like I don't see jsonlines/ndjson in the wild enough -- people still aren't convinced of json logging (or doing stuff

There's also ndjson[0], personally I feel like I don't see jsonlines/ndjson in the wild enough -- people still aren't convinced of json logging (or doing stuff like using Content-Type to deliver ndjson/jsonlines over certain endpoints) in the codebases I run into. [0]: http://ndjson.org/ http://ndjson.org/

secondcoming 6y ago
Using json for logging or event streams is fine until your cloud provider hands you your bill for cross zone or region traffic.
sixothree 6y ago
FHIR Bulk Data Extract is using ndjson as a requirement. https://build.fhir.org/ig/HL7/bulk-data/ https://build.fhir.org/ig/HL7/bulk-data/
fulafel 6y ago
Oh, so there were two different formats, what's the difference?
- hardwaresofton 6y ago
  Funnily enough, issue #1 on ndjson[0] is this same question. There's a fantastic response with a table too[1]. [0]: https://github.com/ndjson/ndjson.github.io/issues/1 https://github.com/ndjson/ndjson.github.io/issues/1 [1]: https://github.com/ndjson/ndjson.github.io/issues/1#issuecomment-109935996 https://github.com/ndjson/ndjson.github.io/issues/1#issuecom...
  - fulafel 6y ago
    Thanks. One of the comments also points out that there's a third spec, IETF RFC7464.
hedora 6y ago
JSON encoded logs are really only useful if you’re putting them into something like splunk. The problem with splunk is that it’s way less expressive than perl, grep, etc, so there are certain useful analyses that simply can’t be run. To work around that, I see applications duplicate information over and over in their log output. The end result vs text files is a 2x blow up because of JSON, then a 5-10x blow up because of splunk. It’s not hard to implement a distributed log processor on top of ssh, and I think open source solutions exist. I don’t know why more companies don’t do that. As a side benefit, your log files stay human readable. In addition to being more flexible, in practice, the result is something like an order of magnitude cheaper and faster than splunk.

secondcoming 6y ago

Using json for logging or event streams is fine until your cloud provider hands you your bill for cross zone or region traffic.

sixothree 6y ago

FHIR Bulk Data Extract is using ndjson as a requirement. https://build.fhir.org/ig/HL7/bulk-data/ https://build.fhir.org/ig/HL7/bulk-data/

fulafel 6y ago

Oh, so there were two different formats, what's the difference?

hardwaresofton 6y ago
Funnily enough, issue #1 on ndjson[0] is this same question. There's a fantastic response with a table too[1]. [0]: https://github.com/ndjson/ndjson.github.io/issues/1 https://github.com/ndjson/ndjson.github.io/issues/1 [1]: https://github.com/ndjson/ndjson.github.io/issues/1#issuecomment-109935996 https://github.com/ndjson/ndjson.github.io/issues/1#issuecom...
- fulafel 6y ago
  Thanks. One of the comments also points out that there's a third spec, IETF RFC7464.

hedora 6y ago

JSON encoded logs are really only useful if you’re putting them into something like splunk. The problem with splunk is that it’s way less expressive than perl, grep, etc, so there are certain useful analyses that simply can’t be run. To work around that, I see applications duplicate information over and over in their log output. The end result vs text files is a 2x blow up because of JSON, then a 5-10x blow up because of splunk. It’s not hard to implement a distributed log processor on top of ssh, and I think open source solutions exist. I don’t know why more companies don’t do that. As a side benefit, your log files stay human readable. In addition to being more flexible, in practice, the result is something like an order of magnitude cheaper and faster than splunk.