4 ms·

Correctly and safely serializing to CSV and JSON is more overhead than spitting out bytes. It's also easier to spit or ingest bytes as a stream—yes, there are t

Correctly and safely serializing to CSV and JSON is more overhead than spitting out bytes. It's also easier to spit or ingest bytes as a stream—yes, there are tools that understand streams of json objects, but you still have to have all the data in memory at once for a given record, to serialize it, and then you've basically got it in memory twice at the same time (original + serialized text). Your minimum record size to consider becomes "this entire object" rather than "this field". You've also now got the code for CSV or JSON serialization in memory.

CodesInChaos 5y ago
You can json or csv encode data in a streaming manner, you just need to escape some special characters (only " for csv) and add a separator. But I never worked on an application where log records were big enough for streaming to make sense. If you want to preserve the "one line = one record" property typically expected from logs, standard csv will not be suitable (since it doesn't escape linebreaks), but json without unnecessary linebreaks will handle that (known as json-lines).
staticassertion 5y ago
Serializing is very fast, and ingesting JSON as a stream is trivial when it's newline delineated. No one emits a giant array of json log objects, they emit a stream of newline delineated json. You almost certainly also have it in memory more than twice since you're copying a buffer out to IO. If you care about record size use protobuf, compression, backend parsers, etc.
no_circuit 5y ago
Unless we're talking about something performance critical up front like an embedded system, odds are by the time an organization is going to care about logging performance overhead, they probably already have a few engineers dedicated to just optimizing logging. Of course some developers may just go ahead and create a JSON/CSV string from scratch, which is about as safe as string interpolation instead of parameter bindings for SQL statements /s. But provided they're using a proper JSON/CSV/proto object builder library that uses String references, no data is going to get copied until a logging component needs to write it to a file or a network buffer. Therefore the only overhead are the pointers that form the structure.

CodesInChaos 5y ago

You can json or csv encode data in a streaming manner, you just need to escape some special characters (only " for csv) and add a separator. But I never worked on an application where log records were big enough for streaming to make sense. If you want to preserve the "one line = one record" property typically expected from logs, standard csv will not be suitable (since it doesn't escape linebreaks), but json without unnecessary linebreaks will handle that (known as json-lines).

staticassertion 5y ago

Serializing is very fast, and ingesting JSON as a stream is trivial when it's newline delineated. No one emits a giant array of json log objects, they emit a stream of newline delineated json. You almost certainly also have it in memory more than twice since you're copying a buffer out to IO. If you care about record size use protobuf, compression, backend parsers, etc.

no_circuit 5y ago

Unless we're talking about something performance critical up front like an embedded system, odds are by the time an organization is going to care about logging performance overhead, they probably already have a few engineers dedicated to just optimizing logging. Of course some developers may just go ahead and create a JSON/CSV string from scratch, which is about as safe as string interpolation instead of parameter bindings for SQL statements /s. But provided they're using a proper JSON/CSV/proto object builder library that uses String references, no data is going to get copied until a logging component needs to write it to a file or a network buffer. Therefore the only overhead are the pointers that form the structure.