4 ms·

Any love for JSON logging when aiming for central aggregation? All kinds of benefits.

tannhaeuser 6y ago
Actually the syslog protocol has it's own structured format for log messages [1]. [1]: https://datatracker.ietf.org/doc/html/rfc5424 https://datatracker.ietf.org/doc/html/rfc5424
- kevincox 6y ago
  Which as far as I can tell is implemented by next to nothing. Logging single-line JSON events will give you far more support for downstream tools.
0xbadcafebee 6y ago
It wouldn't really change anything, it's just a data format. The problem is how to get people to both send and receive specific fields correctly. Lazy programmers/vendors would also still think they could throw together some crappy code and pretend it's generating JSON, but it would end up generating illegal JSON. Same for parsing. There are a lot of things you need in order to prevent the lazy from fucking up: - A version. A lazy programmer might either ignore or hard-code a version number, but at least you have a hint as to which standard you're trying to conform to, and can retain backwards compatibility. (The new RFC has a version, but the old one doesn't, preventing interoperability) - A format that's easy enough for programmers to understand, but difficult (or "feature-filled") enough that they won't attempt to implement it all themselves and will reach for real libraries. - Extensions. Vendors will always want to do something different than everyone else, so if you don't add the option of extensions, they will either fork the protocol and make breaking changes, or try to sneak changes into other parts of the protocol/format. - A standard reference implementation + tests. Make it easy for vendors to test their versions against another one, so the developers don't have to do busy-work like "read a standard" or "write tests". - Think about the future. Does your standard include a specific width integer? Does it preclude a specific network payload size? Will addressing change in the future? Can your data payload support arbitrary binary data? Can your standard change later and still be backwards compatible?
ashtonbaker 6y ago
I've parsed a lot of logs for our SIEM - adding a JSON parser is usually the nicest, barring overly-clever nesting, which does happen.
xorcist 6y ago
There's something about logging that conjures all sort of strange situations. A very well established and expensive product apparently thought the way to write json logs is something akin to: printf("{date: \"%s\", msg: \"%s\"}\n", date, msg); That turned out real good when msg contained quotes. Json isn't a magic bullet here. Most log pipelines end up with custom logic for all sorts of reasons that mostly shouldn't be an issue.
marviio 5y ago
If only Elastic Common Schema would emerge as a common log format. It's flexible but still structured.

Actually the syslog protocol has it's own structured format for log messages [1]. [1]: https://datatracker.ietf.org/doc/html/rfc5424 https://datatracker.ietf.org/doc/html/rfc5424

kevincox 6y ago
Which as far as I can tell is implemented by next to nothing. Logging single-line JSON events will give you far more support for downstream tools.

0xbadcafebee 6y ago

It wouldn't really change anything, it's just a data format. The problem is how to get people to both send and receive specific fields correctly. Lazy programmers/vendors would also still think they could throw together some crappy code and pretend it's generating JSON, but it would end up generating illegal JSON. Same for parsing. There are a lot of things you need in order to prevent the lazy from fucking up: - A version. A lazy programmer might either ignore or hard-code a version number, but at least you have a hint as to which standard you're trying to conform to, and can retain backwards compatibility. (The new RFC has a version, but the old one doesn't, preventing interoperability) - A format that's easy enough for programmers to understand, but difficult (or "feature-filled") enough that they won't attempt to implement it all themselves and will reach for real libraries. - Extensions. Vendors will always want to do something different than everyone else, so if you don't add the option of extensions, they will either fork the protocol and make breaking changes, or try to sneak changes into other parts of the protocol/format. - A standard reference implementation + tests. Make it easy for vendors to test their versions against another one, so the developers don't have to do busy-work like "read a standard" or "write tests". - Think about the future. Does your standard include a specific width integer? Does it preclude a specific network payload size? Will addressing change in the future? Can your data payload support arbitrary binary data? Can your standard change later and still be backwards compatible?

ashtonbaker 6y ago

I've parsed a lot of logs for our SIEM - adding a JSON parser is usually the nicest, barring overly-clever nesting, which does happen.

xorcist 6y ago

There's something about logging that conjures all sort of strange situations. A very well established and expensive product apparently thought the way to write json logs is something akin to: printf("{date: \"%s\", msg: \"%s\"}\n", date, msg); That turned out real good when msg contained quotes. Json isn't a magic bullet here. Most log pipelines end up with custom logic for all sorts of reasons that mostly shouldn't be an issue.

marviio 5y ago

If only Elastic Common Schema would emerge as a common log format. It's flexible but still structured.