3 ms·
Storing a human readable format is a amateurish practice with immense performance and resource costs. It conflates the storage format with the visualization/pro
by Veserv 4y ago
Storing a human readable format is a amateurish practice with immense performance and resource costs. It conflates the storage format with the visualization/processing format.
In basically every case you should store/log a dense binary dump of your varying data and a ID that uniquely identifies the data format. Then, when a human wants to read the log, you do a post-processing step in your debug environment that formats the data in a human readable way.
This moves string formatting off of the target and prevents the frankly outrageous amount of redundant string duplication when storing human readable logs. It also avoids a roundtrip through stringly-typed logs and allows for efficient binary parsing of your logged data in their natural data format instead of doing brittle string parsing.
It is frankly challenging to come up with even a single case where you should not do it this way as long as your tooling is mature enough to manage the separated encoding/decoding pipeline. Even if it is not, you could always just run the post-processing step on all your new raw logs and store that afterwards as your “official logs” which still gives you the performance advantages on your production systems while being no worse than any existing scheme.
- capableweb 4y ago> It is frankly challenging to come up with even a single case where you should not do it this way One use case for human readable format: your service/whatever that output logs doesn't output TBs of output every month and you can easily handle the amount on commodity hardware. Why complicate something that doesn't need it? Both methods have their place.
- nerdponx 4y ago> tooling is mature enough to manage the separated encoding/decoding pipeline Is there a library for it? This sounds like something I can't justify DIYing in the kinds of software that I work on.