3 ms·

Compatibility isn't the problem. The problem is the use of structured binary data itself for logs. Logging binary structures instead of raw text makes it very

by jude- 12y ago

Compatibility isn't the problem. The problem is the use of structured binary data itself for logs. Logging binary structures instead of raw text makes it very difficult to recover from crashes or misbehavior, since the logging facility (journald or otherwise) must ensure that the log's structure on disk is consistent at all times. I've seen more than my fair share of journald corrupting its own log due to unclean shutdowns. If I'm going to be grepping the journald log file anyway to reconstruct it (possible, but not easy, since journalctl is useless here), then why bother using it at all? It fails at the very task it was built for.

bigbugbag 12y ago
Not sure it was built for the task we expect a log facility to fulfill. I mean the corruption issue is a built in feature of journald: https://bugs.freedesktop.org/show_bug.cgi?id=64116 https://bugs.freedesktop.org/show_bug.cgi?id=64116
drdaeman 12y ago
I think I disagree. Classic plain text log files are structured too - they're files of '\n'-separated records, without much else to it. It doesn't really matter (integrity-wise) whenever one's writing, say, JSON or mostly-unstructured plain English records. I'm unaware on particular journald internal implementation quirks and issues. Maybe it's badly coded and has lots of bugs that corrupt data. That would be implementation issue. But the overall idea of using "binary" logs isn't that bad to me.
pdkl95 12y ago
> journald corrupting its own log due to unclean shutdowns Exactly! What the binary log advocates seem to be missing is that those unclean shutdowns (often called "crashes") are probably the very thing you are going to want to search for the in log. In general, few people care how (or if) log stores that the cron daemon yet again ran the hourly maintenance without any errors. What everybody who has had to search a system log cares about is "what happened right before the crash happened". The current data that needs to be committed to the log successfully and immediately almost by definition happens at a time when you do not have the time for the complexity of an atomic addition to a database. Often there is barely have time to any disk write at all. The only way make the system log useful would be to make adding events synchronous. As nobody wants to deal with a syslog that is 10,000x slower (or worse), the only sane option is what we always did - make the writes simple and immediate, and defer any fancier feature. Have they never heard of "log parsers" before? If you want it in a search able DB (which can be very useful), you do that from original log either as an async daemon or defer it with cron or similar.

by jude- 12y ago

bigbugbag 12y ago

Not sure it was built for the task we expect a log facility to fulfill. I mean the corruption issue is a built in feature of journald: https://bugs.freedesktop.org/show_bug.cgi?id=64116 https://bugs.freedesktop.org/show_bug.cgi?id=64116

drdaeman 12y ago

I think I disagree. Classic plain text log files are structured too - they're files of '\n'-separated records, without much else to it. It doesn't really matter (integrity-wise) whenever one's writing, say, JSON or mostly-unstructured plain English records. I'm unaware on particular journald internal implementation quirks and issues. Maybe it's badly coded and has lots of bugs that corrupt data. That would be implementation issue. But the overall idea of using "binary" logs isn't that bad to me.

pdkl95 12y ago

> journald corrupting its own log due to unclean shutdowns Exactly! What the binary log advocates seem to be missing is that those unclean shutdowns (often called "crashes") are probably the very thing you are going to want to search for the in log. In general, few people care how (or if) log stores that the cron daemon yet again ran the hourly maintenance without any errors. What everybody who has had to search a system log cares about is "what happened right before the crash happened". The current data that needs to be committed to the log successfully and immediately almost by definition happens at a time when you do not have the time for the complexity of an atomic addition to a database. Often there is barely have time to any disk write at all. The only way make the system log useful would be to make adding events synchronous. As nobody wants to deal with a syslog that is 10,000x slower (or worse), the only sane option is what we always did - make the writes simple and immediate, and defer any fancier feature. Have they never heard of "log parsers" before? If you want it in a search able DB (which can be very useful), you do that from original log either as an async daemon or defer it with cron or similar.