6 ms·

Great discussion, but one issue I've run into journald, compared to a traditional logging mechanism, is the size of the journal itself. journald's archives are

Great discussion, but one issue I've run into journald, compared to a traditional logging mechanism, is the size of the journal itself. journald's archives are generally a factor of at least 200x of syslog for the same amount of information stored. That is, on two identical servers, I've run rsyslog with the traditional logrotate mechanism and journald side-by-side to see how they compared. I'm pleased with the extra querying mechanism provided by journald, which is absolutely great for debugging, however while I can store the logs one server virtually forever with just bz2 compression, I've unexpectedly run out of space in my var partition with journald after just a couple of months. The extra bandwidth is also a factor in the performance of journald, which is equally ignored. In fact, this has forced me to reconsider the storage logging policy. It's much more effective to use rsyslog to send encrypted logs through the network (faster, less latency and less bandwidth) compared to what it would be with journald. I still keep the journal on individual machines for easier querying, but with a very limited retention. If I have to chose between a log I can query easily, and a log file that can have 200x the density, I'll easily pick the second. I've also found that the resilience of the journald archives are much lower. A corrupted text log file (due to a crash) has nothing to recover and can be easily read to inspect issues in any worst-case scenario. journald archives however are often corrupted in crashes and moved away after the reboot. For remote logging it might not be an issue, but for embedded systems and/or desktop machines it is an issue. Ensuring logs can be read even when corrupted is easily much more important than introducing the signing mechanism in the archives (which, on the other hand, is totally a non-issue as sensitive hosts should do remote logging).

ape4 11y ago
I can't help but think... if they just used traditional log files there would be no delays, corruption.
- pilif 11y ago
  I don't think this is entirely true: Even if you're appending lines to a text-file, if you do it from multiple concurrent threads, you're going to potentially corrupt individual lines. And if you want to make absolutely sure that the log line you've just written is actually safely on disk, then you still have to fsync(). I would argue that journald's mmap-with-periodic-fsync() approach is actually better for latency. If you want to make sure that your system performance isn't affected by stalled logging i/o, then you should ship the logs off-machine using a non-blocking protocol. Of course then you lose guarantees that your log-entries have actually been stored (which might or might not be a problem for you).
  - otterley 11y ago
    > Even if you're appending lines to a text-file, if you do it from multiple concurrent threads, you're going to potentially corrupt individual lines. Not true; the write(2) syscall is atomic. You just have put the the entire line in the buffer you pass to it.
    - lomnakkus 11y ago
      Not true AFAICT. (On Linux at least) Here are a couple of snippets from the man page: > If a write() requests that more bytes be written than there is room for (for example, the file size limit of the process or the physical end of a medium), only as many bytes as there is room for shall be written. (Admittely disk full is pretty much an edge case.) > If write() is interrupted by a signal after it successfully writes some data, it shall return the number of bytes written.
      - Someone 11y ago
        I think a slightly better way to describe this is "write() is atomic, but may not write the number of bytes asked to write." A possible workaround is to use a start of line marker and provide a way to detect broken lines. For example, if one starts every line with a 0x02 and ends it in a 0x03 (Start of Text/End of Text ASCII characters) and removes any such character from the text being logged, partly written log entries can be detected and ignored when reading the log. If one does that, one can just retry writing the entire line until write() indicates it wrote the entire buffer passed in.
        mjevans 11y ago
        I disagree. It is far better to use a String like approach and use a structure that includes the length of the next written object. I believe that a simple sacrifice of a little more storage would also make that easier to split out. Pre-pad the message up to pad size zeros and then a pseudo-random number of additional pad objects (1 to n). Next you would store either the offset of the start or end, as well as the length of the payload (start would probably be easier to verify). With that structure it should be difficult to craft a payload message that looked like a valid message for the log.
kbenson 11y ago
You shouldn't need to choose. MySQL's InnoDB engine supports compression, and not only does it use less space on disk, it's actually faster to read and write. I don't see why journald can't implement something similar.
- msandford 11y ago
  Why not log them directly to a hadoop cluster while you're at it? I mean, while we're cramming everything into the low-level OS services.
  - kbenson 11y ago
    The deflate algorithm is well understood, well defined, available in every language, and not hard to implement. At this point, we might benefit from having it included and provided by the kernel. In fact, the kernel does have limited support for compression[1] (well, decompression only currently, it seems). Expanding upon this to allow compression as well would likely yield quite a few benefits for multiple applications. 1: https://www.kernel.org/doc/Documentation/xz.txt https://www.kernel.org/doc/Documentation/xz.txt

ape4 11y ago

I can't help but think... if they just used traditional log files there would be no delays, corruption.

pilif 11y ago
I don't think this is entirely true: Even if you're appending lines to a text-file, if you do it from multiple concurrent threads, you're going to potentially corrupt individual lines. And if you want to make absolutely sure that the log line you've just written is actually safely on disk, then you still have to fsync(). I would argue that journald's mmap-with-periodic-fsync() approach is actually better for latency. If you want to make sure that your system performance isn't affected by stalled logging i/o, then you should ship the logs off-machine using a non-blocking protocol. Of course then you lose guarantees that your log-entries have actually been stored (which might or might not be a problem for you).
- otterley 11y ago
  > Even if you're appending lines to a text-file, if you do it from multiple concurrent threads, you're going to potentially corrupt individual lines. Not true; the write(2) syscall is atomic. You just have put the the entire line in the buffer you pass to it.
  - lomnakkus 11y ago
    Not true AFAICT. (On Linux at least) Here are a couple of snippets from the man page: > If a write() requests that more bytes be written than there is room for (for example, the file size limit of the process or the physical end of a medium), only as many bytes as there is room for shall be written. (Admittely disk full is pretty much an edge case.) > If write() is interrupted by a signal after it successfully writes some data, it shall return the number of bytes written.
    - Someone 11y ago
      I think a slightly better way to describe this is "write() is atomic, but may not write the number of bytes asked to write." A possible workaround is to use a start of line marker and provide a way to detect broken lines. For example, if one starts every line with a 0x02 and ends it in a 0x03 (Start of Text/End of Text ASCII characters) and removes any such character from the text being logged, partly written log entries can be detected and ignored when reading the log. If one does that, one can just retry writing the entire line until write() indicates it wrote the entire buffer passed in.
      - mjevans 11y ago
        I disagree. It is far better to use a String like approach and use a structure that includes the length of the next written object. I believe that a simple sacrifice of a little more storage would also make that easier to split out. Pre-pad the message up to pad size zeros and then a pseudo-random number of additional pad objects (1 to n). Next you would store either the offset of the start or end, as well as the length of the payload (start would probably be easier to verify). With that structure it should be difficult to craft a payload message that looked like a valid message for the log.

kbenson 11y ago

You shouldn't need to choose. MySQL's InnoDB engine supports compression, and not only does it use less space on disk, it's actually faster to read and write. I don't see why journald can't implement something similar.

msandford 11y ago
Why not log them directly to a hadoop cluster while you're at it? I mean, while we're cramming everything into the low-level OS services.
- kbenson 11y ago
  The deflate algorithm is well understood, well defined, available in every language, and not hard to implement. At this point, we might benefit from having it included and provided by the kernel. In fact, the kernel does have limited support for compression[1] (well, decompression only currently, it seems). Expanding upon this to allow compression as well would likely yield quite a few benefits for multiple applications. 1: https://www.kernel.org/doc/Documentation/xz.txt https://www.kernel.org/doc/Documentation/xz.txt