6 ms·

I was forced to use systemd and journald for a while and I've seriously grown to like them, especially journald.

lathiat 5y ago
Yep
hughrr 5y ago
They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness. It’s best to do some drills on a purposely broken system to gain some deep insight into recovery scenarios.
- oblio 5y ago
  If everything is broken and you need to do recovery, don't you usually do that from another system anywhere? And mount the target system. I imagine the journald logs are just files at the end of the day and you can just read them with some tool?
- fullstop 5y ago
  This can be tricky because journald compression, but is this really that much different than a corrupted compressed log file, like that which logrotate produces, in /var/log? If that is your concern, you can disable compression in the journald configuration so that the contents can be read with "strings" or similar tools.
- theamk 5y ago
  I am curious what kind of problems have you seen? I am transitioning some of my systems to journald, and I am very interested in things that could go wrong. So far, I tried looking at logs from the dead system using “journalctl -D” - it seemed to work. And the way how the log files from each boot are always separate is pretty handy. Other than that, the only problems I have seen were having more to type and having to learn more commands. Am I in for a nasty surprise?
  - hughrr 5y ago
    Key issues for me were machine-id related and journal corruption. Also journalctl dumping core after a recovery-mode boot. That left us with no tools to deal with a compressed journal and no strace to find out what it was doing. I actually attached a USB disk to the machine and cp’ed the files off it in the end. This was inconvenient as the node was 1550 miles away from me.
    - theamk 5y ago
      Did you use -D / --directory option to journalctl? It switches the tool from "do the common thing and inspect live system" mode to "do exactly what I want and just convert files in that directory to text". There is also "--file" option for one-file-at a time access, but it is harder to use. In particular, I have just verified with "strace" that running journalctl with either "-D /var/log/journal" or "--file" option does not even open machine-id file nor a d-bus socket -- so whatever problems you had with machine-id would be gone with. Also, when you said, that you "cp'ed the files off it", did you mean you copied off the contents of /var/log/journal? Were you able to open the files on the other machine?
- 2ion 5y ago
  If your logs are that important they should get shipped off the node immediately anyway. That's what ELK, Graylog, loki etc are for. If you don't do centralized automated log management on a many-node server farm and that impedes your forensic processes when a node fails then not compressed logs are to blame but your system. On any server I am responsible for, any log that is created on the node and required or useful for forensics gets shipped off the node immediately after it's written, including all system logs, container outputs, etc. I can only highly recommend it. The amounts of data to store centrally are very manageable as well even if you ship 100% of logs.
  - 0xbadcafebee 5y ago
    So, if you care about logs, don't depend on journalctl... Well that's fine if you can ship logs off the box, and if the log shipping is working. But if you have to troubleshoot on the machine itself (hello, desktops?) then having tools that don't come with a lot of pain and big learning curve is critical to addressing the issue quickly. What's the point in even having these tools if we're not supposed to use them? Normally, Unix-like tools are not as painful as journalctl. But Poettering's interfaces are a gauntlet of unusual concepts and hidden inter-relationships, with no common examples or intuitiveness. And things like dbus make it worse, now that there are many parts of a modern Linux system that have no console interface, because nobody's written one for the particular application you need to view or change the right dbus settings. And /sys/ is literally a wilderness of random undocumented settings that are often the only interface to critical system functions. Linux distributions are now a tiresome no-mans-land of overcomplicated mysterious crap. I'm willing to bet the major Linux distributions will be abandoned over the next decade for simpler systems that are cloud-native, mobile-friendly, and have less Kafkaesque interfaces.
    - fullstop 5y ago
      What kind of desktop do you have where your journal is constantly getting corrupted? Most distributions log both to the journal and to /var/log/, so I'm seriously struggling to understand the complaint here. GP's message about centralized logging is saying that if logs are that important to you, you will keep them stored in a centralized location. You should be able to catch the server on fire and have logs recorded centrally up until that point. This would be the case with or without journald. You started with a strawman about caring about logs, and ended complaining about sysfs.

Yep

They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness. It’s best to do some drills on a purposely broken system to gain some deep insight into recovery scenarios.

oblio 5y ago
If everything is broken and you need to do recovery, don't you usually do that from another system anywhere? And mount the target system. I imagine the journald logs are just files at the end of the day and you can just read them with some tool?
fullstop 5y ago
This can be tricky because journald compression, but is this really that much different than a corrupted compressed log file, like that which logrotate produces, in /var/log? If that is your concern, you can disable compression in the journald configuration so that the contents can be read with "strings" or similar tools.
theamk 5y ago
I am curious what kind of problems have you seen? I am transitioning some of my systems to journald, and I am very interested in things that could go wrong. So far, I tried looking at logs from the dead system using “journalctl -D” - it seemed to work. And the way how the log files from each boot are always separate is pretty handy. Other than that, the only problems I have seen were having more to type and having to learn more commands. Am I in for a nasty surprise?
- hughrr 5y ago
  Key issues for me were machine-id related and journal corruption. Also journalctl dumping core after a recovery-mode boot. That left us with no tools to deal with a compressed journal and no strace to find out what it was doing. I actually attached a USB disk to the machine and cp’ed the files off it in the end. This was inconvenient as the node was 1550 miles away from me.
  - theamk 5y ago
    Did you use -D / --directory option to journalctl? It switches the tool from "do the common thing and inspect live system" mode to "do exactly what I want and just convert files in that directory to text". There is also "--file" option for one-file-at a time access, but it is harder to use. In particular, I have just verified with "strace" that running journalctl with either "-D /var/log/journal" or "--file" option does not even open machine-id file nor a d-bus socket -- so whatever problems you had with machine-id would be gone with. Also, when you said, that you "cp'ed the files off it", did you mean you copied off the contents of /var/log/journal? Were you able to open the files on the other machine?
2ion 5y ago
If your logs are that important they should get shipped off the node immediately anyway. That's what ELK, Graylog, loki etc are for. If you don't do centralized automated log management on a many-node server farm and that impedes your forensic processes when a node fails then not compressed logs are to blame but your system. On any server I am responsible for, any log that is created on the node and required or useful for forensics gets shipped off the node immediately after it's written, including all system logs, container outputs, etc. I can only highly recommend it. The amounts of data to store centrally are very manageable as well even if you ship 100% of logs.
- 0xbadcafebee 5y ago
  So, if you care about logs, don't depend on journalctl... Well that's fine if you can ship logs off the box, and if the log shipping is working. But if you have to troubleshoot on the machine itself (hello, desktops?) then having tools that don't come with a lot of pain and big learning curve is critical to addressing the issue quickly. What's the point in even having these tools if we're not supposed to use them? Normally, Unix-like tools are not as painful as journalctl. But Poettering's interfaces are a gauntlet of unusual concepts and hidden inter-relationships, with no common examples or intuitiveness. And things like dbus make it worse, now that there are many parts of a modern Linux system that have no console interface, because nobody's written one for the particular application you need to view or change the right dbus settings. And /sys/ is literally a wilderness of random undocumented settings that are often the only interface to critical system functions. Linux distributions are now a tiresome no-mans-land of overcomplicated mysterious crap. I'm willing to bet the major Linux distributions will be abandoned over the next decade for simpler systems that are cloud-native, mobile-friendly, and have less Kafkaesque interfaces.
  - fullstop 5y ago
    What kind of desktop do you have where your journal is constantly getting corrupted? Most distributions log both to the journal and to /var/log/, so I'm seriously struggling to understand the complaint here. GP's message about centralized logging is saying that if logs are that important to you, you will keep them stored in a centralized location. You should be able to catch the server on fire and have logs recorded centrally up until that point. This would be the case with or without journald. You started with a strawman about caring about logs, and ended complaining about sysfs.