3 ms·
One of the non-functional requirements of logs is that they should be fast to write. Marshalling data into a structured format takes longer than spitting out sp
by jacques_chester 15y ago
One of the non-functional requirements of logs is that they should be fast to write. Marshalling data into a structured format takes longer than spitting out sprintfs.
If you really need structure for ease of querying, you might as well go all the way and throw it into a proper data store.
- frsyuki 15y agoSpitting out sprintfs means parsing texts. JSON might be slow since it's text, but binary based formats should be faster especially the log includes many integers.
- jacques_chester 15y agosprintf tends to be a faster path than "build tree, render down to text".
- deno 15y agoBoth XML and JSON can be streamed. No tree building necessary.
- gorset 15y agoRemember we're talking about writing to disk. The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. We use JSON on disk heavily in several places where we generate or receive huge amounts of data. JSON-in-a-file is a pretty good datastore for sequential data processing because it's so convent and you can work with the data using cat, tail or any tool supporting JSON.
- stevewilhelm 15y agoYou're still logging to local disk?
- jacques_chester 15y ago> The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. And JSON is more verbose. Basically this is another read/write dichotomy. When do we pay the price of structuring data? At write time? Or at read time? I'd rather pay it at read time, as writing may be shaving performance from my actual primary production system.
- gorset 15y agoThis is premature optimization. Do you think it's expensive to write a few brackets, quotes and escape a few characters? We have harder problems to think about than optimizing for writes in our logging system :-) How can you trade write/read without some structure in the log message? And do you know that sprintf actually costs something too? It's basically a mini-language which must be parsed and interpreted. You can go google sprintf+performance to see stories of people finding this out - but most of the time it doesn't matter, because the cost is insignificant and you can do what's most convenient for you.
- jacques_chester 15y agoMy point is that structure is imposed somewhere if you want to query your data. Either that structure is in the data, or in tools that parse the data. There is a "conservation of structure", if you like. Most of the bunfights between proponents of different technologies is really an argument about where to pay structural tax. You can pay more at write time or read time on a sliding scale. For logs, I think the smart option is to pay as little write-time overhead as possible. Their purpose is to maximally describe an event with minimal interruption to service. Every strictly unnecessary adornment to structure takes you further away from that core non-functional goal.
- thezilch 15y agoa) log writes should be buffered b) once buffered or immediately (ex. UDP [ex. Etsy's StatsD]), the client can typically continue and even complete without the log being flushed all the way to non-volatile media