5 ms·

except if one line of JSON gets corrupted ... all your log storage does go to the trash. Heard about people login JSON ? (that may be truncated because log lin

by SFJulie 10y ago

except if one line of JSON gets corrupted ... all your log storage does go to the trash. Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?) I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens. Hence a day where a corruption of logs is more likely to happen. True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.

int_19h 10y ago
The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line. And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).
- SFJulie 10y ago
  well, what a nice way to make a greper lose order of magnitudes in speed ; grep is fast because he does not line split. And if \n is prohibited in string it is NOT JSON per ecma xyz anymore. it is something else. Additional complexities that do not seem meaningful but when in congestion they add up to make your life a hell. But well, energy is cheap, VMs are cheap, operational expenses for cloud so less than coders, why bother? With this kind of reasoning we will have so broken internet appliance coded with feet that we will experience a massive DDOS made by connected toasters and poorly coded cameras.
  - falcolas 10y ago
    Line Feed (/n) is a Unicode control character (C0 group), so it must be escaped in a JSON string. This per ECMA-404.
  - int_19h 10y ago
    "\n" (that is, ASCII character "\", followed by ASCII character "n", producing escape sequence "\n" for newline) is not prohibited inside a string literal in that model. Only actual newlines (that is, ASCII characters 10 and 13) are prohibited, and that is already a part of JSON spec. Like I said, the only place where JSON permits actual newlines is as part of insignificant whitespace. But because it's insignificant, newlines can be replaced with any other whitespace character, and the resulting JSON will have the same exact meaning. And why would grep lose "order of magnitudes in speed"? If you were actually using grep, it'd work exactly the same as it does for any text file. The only thing that JSON does is add some structure to the contents, so that it can be easily converted to some tabular format that's more convenient for structured queries, aggregation etc. But you can still treat it as plain text for all purposes.
JdeBP 10y ago
The days that I frequently need logs are the days that I need to prove that a particular application did a particular thing -- handled a particular transaction, sent a particular message, responded to/triggered a particular event, and so forth. Faults in these areas are applications software faults (often not even fatal ones), and are unlikely to affect the logger process at all, especially since it is insulated from the applications softwares by each running under the aegis of its own dedicated unprivileged user account. Crash diagnosis is but one use for logs.

by SFJulie 10y ago

int_19h 10y ago

The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line. And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).

SFJulie 10y ago
well, what a nice way to make a greper lose order of magnitudes in speed ; grep is fast because he does not line split. And if \n is prohibited in string it is NOT JSON per ecma xyz anymore. it is something else. Additional complexities that do not seem meaningful but when in congestion they add up to make your life a hell. But well, energy is cheap, VMs are cheap, operational expenses for cloud so less than coders, why bother? With this kind of reasoning we will have so broken internet appliance coded with feet that we will experience a massive DDOS made by connected toasters and poorly coded cameras.
- falcolas 10y ago
  Line Feed (/n) is a Unicode control character (C0 group), so it must be escaped in a JSON string. This per ECMA-404.
- int_19h 10y ago
  "\n" (that is, ASCII character "\", followed by ASCII character "n", producing escape sequence "\n" for newline) is not prohibited inside a string literal in that model. Only actual newlines (that is, ASCII characters 10 and 13) are prohibited, and that is already a part of JSON spec. Like I said, the only place where JSON permits actual newlines is as part of insignificant whitespace. But because it's insignificant, newlines can be replaced with any other whitespace character, and the resulting JSON will have the same exact meaning. And why would grep lose "order of magnitudes in speed"? If you were actually using grep, it'd work exactly the same as it does for any text file. The only thing that JSON does is add some structure to the contents, so that it can be easily converted to some tabular format that's more convenient for structured queries, aggregation etc. But you can still treat it as plain text for all purposes.

JdeBP 10y ago

The days that I frequently need logs are the days that I need to prove that a particular application did a particular thing -- handled a particular transaction, sent a particular message, responded to/triggered a particular event, and so forth. Faults in these areas are applications software faults (often not even fatal ones), and are unlikely to affect the logger process at all, especially since it is insulated from the applications softwares by each running under the aegis of its own dedicated unprivileged user account. Crash diagnosis is but one use for logs.