5 ms·

If your logs are that important they should get shipped off the node immediately anyway. That's what ELK, Graylog, loki etc are for. If you don't do centralized

by 2ion 5y ago

If your logs are that important they should get shipped off the node immediately anyway. That's what ELK, Graylog, loki etc are for. If you don't do centralized automated log management on a many-node server farm and that impedes your forensic processes when a node fails then not compressed logs are to blame but your system. On any server I am responsible for, any log that is created on the node and required or useful for forensics gets shipped off the node immediately after it's written, including all system logs, container outputs, etc. I can only highly recommend it. The amounts of data to store centrally are very manageable as well even if you ship 100% of logs.

0xbadcafebee 5y ago
So, if you care about logs, don't depend on journalctl... Well that's fine if you can ship logs off the box, and if the log shipping is working. But if you have to troubleshoot on the machine itself (hello, desktops?) then having tools that don't come with a lot of pain and big learning curve is critical to addressing the issue quickly. What's the point in even having these tools if we're not supposed to use them? Normally, Unix-like tools are not as painful as journalctl. But Poettering's interfaces are a gauntlet of unusual concepts and hidden inter-relationships, with no common examples or intuitiveness. And things like dbus make it worse, now that there are many parts of a modern Linux system that have no console interface, because nobody's written one for the particular application you need to view or change the right dbus settings. And /sys/ is literally a wilderness of random undocumented settings that are often the only interface to critical system functions. Linux distributions are now a tiresome no-mans-land of overcomplicated mysterious crap. I'm willing to bet the major Linux distributions will be abandoned over the next decade for simpler systems that are cloud-native, mobile-friendly, and have less Kafkaesque interfaces.
- fullstop 5y ago
  What kind of desktop do you have where your journal is constantly getting corrupted? Most distributions log both to the journal and to /var/log/, so I'm seriously struggling to understand the complaint here. GP's message about centralized logging is saying that if logs are that important to you, you will keep them stored in a centralized location. You should be able to catch the server on fire and have logs recorded centrally up until that point. This would be the case with or without journald. You started with a strawman about caring about logs, and ended complaining about sysfs.
  - 0xbadcafebee 5y ago
    Yeah I digressed quite a bit, sorry. > I'm seriously struggling to understand the complaint here The complaint was trying to explain a parent commenter's point about journald "They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness." Point: Logs are really annoying to manage on systems with journald. Counter-point: Ship your logs somewhere else / you probably don't have these problems in real life Counter-Counter: If we're not supposed to use these tools on our hosts, exactly why are they installed?
    - fullstop 5y ago
      Logs are written to both the journal and /var/log/ I would argue that logs are less annoying to manage on systems with journald, once you take the time to learn how to leverage the tools. I would also argue that shipping mission-critical logs off-server is a worthy endeavor, regardless of logging system used. I like journald because it lets me isolate logs for a particular unit without grepping and accidentally including output from unrelated services. It's faster to find the data that I need between time ranges rather than manually comparing time stamps. I can count on one finger the number of times the journal has been corrupted on the servers that I manage, and it was because of hardware failure.
      - deleted 5y ago
        [deleted]
      - mousepilot 5y ago
        I have this fantasy that all the pro systemd posts are really just bots run by leonard
        fullstop 5y ago
        I, Leonard, do not know of what you speak. ;-)
hughrr 5y ago
Yes 100% agree with this and we do that but there’s normally a few seconds window between something going snap and being flushed to the upstream log aggregator. At which point the smouldering remains of your node or container are fairly important to have at hand. It’s like having a black box that forgets the last 30 seconds of the flight otherwise. This is incredibly difficult to get right.
- fullstop 5y ago
  Which has better odds, the smoldering remains having actually flushed the write to disk or the flushed to the upstream log aggregator?
  - hughrr 5y ago
    From experience of both, the smouldering remains seems to win out by a fair margin. The aggregator may have some smoke signals in it but that’s about it. Most of the issues I’ve dealt with shaft the network before the filesystem.
    - fullstop 5y ago
      Outside of the "cloud world" most of my failures have been the result of disk failure. In those cases, the logs definitely did not get written to disk, but they were flushed to graylog.
      - 0xbadcafebee 5y ago
        I can't remember the name of it anymore, but we used to use a user-space daemon to collect logs from applications over various methods (sockets pipes files etc) and it would store them in a ring buffer until they could be flushed to disk or network. If neither happened, you could attach to the host somehow and get the logs out of memory, watch them on a terminal screen, etc. Opening an EC2 console in AWS is still a simple and reliable way to find out what's going wrong with your instance. Wouldn't be possible if we didn't already have the convention to have the kernel, syslog, etc print to tty1.

by 2ion 5y ago

0xbadcafebee 5y ago

So, if you care about logs, don't depend on journalctl... Well that's fine if you can ship logs off the box, and if the log shipping is working. But if you have to troubleshoot on the machine itself (hello, desktops?) then having tools that don't come with a lot of pain and big learning curve is critical to addressing the issue quickly. What's the point in even having these tools if we're not supposed to use them? Normally, Unix-like tools are not as painful as journalctl. But Poettering's interfaces are a gauntlet of unusual concepts and hidden inter-relationships, with no common examples or intuitiveness. And things like dbus make it worse, now that there are many parts of a modern Linux system that have no console interface, because nobody's written one for the particular application you need to view or change the right dbus settings. And /sys/ is literally a wilderness of random undocumented settings that are often the only interface to critical system functions. Linux distributions are now a tiresome no-mans-land of overcomplicated mysterious crap. I'm willing to bet the major Linux distributions will be abandoned over the next decade for simpler systems that are cloud-native, mobile-friendly, and have less Kafkaesque interfaces.

fullstop 5y ago
What kind of desktop do you have where your journal is constantly getting corrupted? Most distributions log both to the journal and to /var/log/, so I'm seriously struggling to understand the complaint here. GP's message about centralized logging is saying that if logs are that important to you, you will keep them stored in a centralized location. You should be able to catch the server on fire and have logs recorded centrally up until that point. This would be the case with or without journald. You started with a strawman about caring about logs, and ended complaining about sysfs.
- 0xbadcafebee 5y ago
  Yeah I digressed quite a bit, sorry. > I'm seriously struggling to understand the complaint here The complaint was trying to explain a parent commenter's point about journald "They’re really nice until everything on the node is completely broken. Then they are a massive obstruction to access and understanding what went wrong due to the opaqueness." Point: Logs are really annoying to manage on systems with journald. Counter-point: Ship your logs somewhere else / you probably don't have these problems in real life Counter-Counter: If we're not supposed to use these tools on our hosts, exactly why are they installed?
  - fullstop 5y ago
    Logs are written to both the journal and /var/log/ I would argue that logs are less annoying to manage on systems with journald, once you take the time to learn how to leverage the tools. I would also argue that shipping mission-critical logs off-server is a worthy endeavor, regardless of logging system used. I like journald because it lets me isolate logs for a particular unit without grepping and accidentally including output from unrelated services. It's faster to find the data that I need between time ranges rather than manually comparing time stamps. I can count on one finger the number of times the journal has been corrupted on the servers that I manage, and it was because of hardware failure.
    - deleted 5y ago
      [deleted]
    - mousepilot 5y ago
      I have this fantasy that all the pro systemd posts are really just bots run by leonard
      - fullstop 5y ago
        I, Leonard, do not know of what you speak. ;-)

hughrr 5y ago

Yes 100% agree with this and we do that but there’s normally a few seconds window between something going snap and being flushed to the upstream log aggregator. At which point the smouldering remains of your node or container are fairly important to have at hand. It’s like having a black box that forgets the last 30 seconds of the flight otherwise. This is incredibly difficult to get right.

fullstop 5y ago
Which has better odds, the smoldering remains having actually flushed the write to disk or the flushed to the upstream log aggregator?
- hughrr 5y ago
  From experience of both, the smouldering remains seems to win out by a fair margin. The aggregator may have some smoke signals in it but that’s about it. Most of the issues I’ve dealt with shaft the network before the filesystem.
  - fullstop 5y ago
    Outside of the "cloud world" most of my failures have been the result of disk failure. In those cases, the logs definitely did not get written to disk, but they were flushed to graylog.
    - 0xbadcafebee 5y ago
      I can't remember the name of it anymore, but we used to use a user-space daemon to collect logs from applications over various methods (sockets pipes files etc) and it would store them in a ring buffer until they could be flushed to disk or network. If neither happened, you could attach to the host somehow and get the logs out of memory, watch them on a terminal screen, etc. Opening an EC2 console in AWS is still a simple and reliable way to find out what's going wrong with your instance. Wouldn't be possible if we didn't already have the convention to have the kernel, syslog, etc print to tty1.