2 ms·

Unstructured logging, while easy for application developers and ops people, is a major source of headache for data scientists/engineers. I am a big proponent o

by kiyoto 11y ago

Unstructured logging, while easy for application developers and ops people, is a major source of headache for data scientists/engineers. I am a big proponent of JSON-based, semi-structured logging. Most of log data today can be parsed with reasonable rigor at source, and doing this before shipping data to the backend saves so much future agony. I recently blogged about this here: http://radar.oreilly.com/2015/04/the-log-the-lifeblood-of-your-data-pipeline.html http://radar.oreilly.com/2015/04/the-log-the-lifeblood-of-yo...

thaumaturgy 11y ago
That was a good read (I wish there was more like it posted to HN). I think there's something to be said though for not adding complexity before it's needed, which isn't really covered in your article. You assume you're writing for an audience that needs to process transactions and events for thousands of servers, but I'd bet that's not the case for most of the people reading your article. That kind of fractal complexity -- complexity added to every single layer of software architecture -- is a real pain in smaller environments where there's just one guy responsible for figuring out what the heck just went pear-shaped in the application. (Although, the real world problem I've had more often, as that guy, is nonexistent logs...) Ideally, I'd like to see both purposes handled -- applications logging semi-structured JSON with a utility that's real-time translating the output to a human readable format, or, preferably, vice-versa, with applications logging unstructured data to a file that's then being munged into JSON (or whatever) for machine consumption. I'm not such a fan of tools that are intended for humans to mine structured logs as though they were text files, because it's too easy to miss important data.

2 ms·

Unstructured logging, while easy for application developers and ops people, is a major source of headache for data scientists/engineers. I am a big proponent o

by kiyoto 11y ago

thaumaturgy 11y ago
That was a good read (I wish there was more like it posted to HN). I think there's something to be said though for not adding complexity before it's needed, which isn't really covered in your article. You assume you're writing for an audience that needs to process transactions and events for thousands of servers, but I'd bet that's not the case for most of the people reading your article. That kind of fractal complexity -- complexity added to every single layer of software architecture -- is a real pain in smaller environments where there's just one guy responsible for figuring out what the heck just went pear-shaped in the application. (Although, the real world problem I've had more often, as that guy, is nonexistent logs...) Ideally, I'd like to see both purposes handled -- applications logging semi-structured JSON with a utility that's real-time translating the output to a human readable format, or, preferably, vice-versa, with applications logging unstructured data to a file that's then being munged into JSON (or whatever) for machine consumption. I'm not such a fan of tools that are intended for humans to mine structured logs as though they were text files, because it's too easy to miss important data.