3 ms·
Remember we're talking about writing to disk. The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. W
by gorset 14y ago
Remember we're talking about writing to disk. The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write.
We use JSON on disk heavily in several places where we generate or receive huge amounts of data. JSON-in-a-file is a pretty good datastore for sequential data processing because it's so convent and you can work with the data using cat, tail or any tool supporting JSON.
- stevewilhelm 14y agoYou're still logging to local disk?
- jacques_chester 14y ago> The time it takes to marshal the data is pretty much insignificant to the time it takes to do the actual write. And JSON is more verbose. Basically this is another read/write dichotomy. When do we pay the price of structuring data? At write time? Or at read time? I'd rather pay it at read time, as writing may be shaving performance from my actual primary production system.
- gorset 14y agoThis is premature optimization. Do you think it's expensive to write a few brackets, quotes and escape a few characters? We have harder problems to think about than optimizing for writes in our logging system :-) How can you trade write/read without some structure in the log message? And do you know that sprintf actually costs something too? It's basically a mini-language which must be parsed and interpreted. You can go google sprintf+performance to see stories of people finding this out - but most of the time it doesn't matter, because the cost is insignificant and you can do what's most convenient for you.
- jacques_chester 14y agoMy point is that structure is imposed somewhere if you want to query your data. Either that structure is in the data, or in tools that parse the data. There is a "conservation of structure", if you like. Most of the bunfights between proponents of different technologies is really an argument about where to pay structural tax. You can pay more at write time or read time on a sliding scale. For logs, I think the smart option is to pay as little write-time overhead as possible. Their purpose is to maximally describe an event with minimal interruption to service. Every strictly unnecessary adornment to structure takes you further away from that core non-functional goal.
- thezilch 14y agoa) log writes should be buffered b) once buffered or immediately (ex. UDP [ex. Etsy's StatsD]), the client can typically continue and even complete without the log being flushed all the way to non-volatile media