3 ms·
> In more extreme scenarios, if journald is blocked long enough for its watchdog timer to expire, it will be killed with SIGABRT, dump core, and abruptly close
by falcolas 11y ago
> In more extreme scenarios, if journald is blocked long enough for its watchdog timer to expire, it will be killed with SIGABRT, dump core, and abruptly close all connected sockets losing whatever log data was in-flight. In some cases this can even cause services to crash due to the unexpected hangup of their stdout and/or stderr.
Ouch.
> we were seeing a delay between .1-.5 seconds twice every 5 seconds
Double ouch. Half a second can be an eternity when wringing all the performance you can out of a system, and with so many cloud providers using network attached storage, that half second could balloon so quickly.
200kb is not an unusual size for unix socket buffers, and it wouldn't be too hard to bypass 200k in log writes in half a second on a busy system.
As an org who will be moving onto systemd with the release of the next Ubuntu LTS version, I appreciate what CoreOS is doing here. Great article, great finds.
As a side note, this is something to look out for on AWS instances where the default for /var/log is on an EBS mount (which can see delays due to network latency). It would make sense to mount /var/log to the ephemeral drive to avoid or minimize such unpleasantness. Of course, that limits the ability to use /var/log to troubleshoot kernel panics.
- otterley 11y agoDurability and performance is a classic tradeoff, and systemd-journald is hardly the only application affected by this. If you need more performance while maintaining durability, buy faster storage technology. That said, if journald could use multiple I/O threads, it might be able to take advantage of concurrency in the underlying storage if available (e.g. RAID). MySQL's InnoDB is a good example of such an implementation. At any rate, in a modern production system, you generally want logs to be copied to an indexed archive located elsewhere in the network. On-host logs should be considered ephemeral, and mainly used as a buffer against slow consumers.
- falcolas 11y agoI think the bigger problem is more related to how outside services are being negativiely impacted (up to and including crashing) by delays in writing to disk, and have no choice in the performance/safety trade off.
- otterley 11y agoAny program that uses the syslog(3) library function will also have this problem, as it also blocks (internally it writes to the /dev/log socket file). If the consumer can't keep up, whether it be rsyslogd or syslog-ng or whatever is listening to the socket, the producer can block as well. So this is not a new problem. If you really want your producer to drop logs in the event of backpressure, you can directly use the syslog protocol over UDP, but it has tradeoffs too.
- falcolas 11y agoThat depends on the syslogd implementation, and will not cause programs to crash due to disconnected stderr/stdout buffers. Perhaps more importantly, you can swap out the syslogd if you don't like its implementation. This is harder to do (if even possible) with journald.
- otterley 11y agoHave you seen any syslog server implementations that are significantly more performant than journald?
- ktRolster 11y agoWith delays like that, no wonder people are having troubles with log corruption