7 ms·

> remember when you had to write bash scripts to start, stop, restart services This was a really big pain, yes, but I also remember how I could `tail -f *.logs

by agilob 2y ago

> remember when you had to write bash scripts to start, stop, restart services This was a really big pain, yes, but I also remember how I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`. I knew where the files were, what they were called. I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename. The init system was probably peak of systemd, after that they started reinveting things in a more complicated way. Do we really need journald, systemd-boot, machinectl, systemd-networkd, sd-bus, systemd-resolved, systemd-nspawn? Do people actually use it all? Are there any metrics to show how many systems have it installed and in use?

superb_dev 2y ago
The follow argument is identical to tail `-f`. `sudo journalctl -f -u <service name>`
- rangerelf 2y ago
  And what would be the equivalent to, "Oh, I don´t know the name of the log for this process I can see in 'ps aux', let me cd into /var/log and see what filenames I can find ... or grep everything until I can find a couple of words that make some sense so I can keep digging further"? The lack of explorability in journalctl, the "need" to keep everything locked behind their own flavor of tools and magic file types, is what makes the rest of us abhor them.
  - iforgotpassword 2y ago
    Huh, isn't grepping journalctl output pretty much the same? It even prefixes messages by the binary name.
    - mdaniel 2y ago
      Pedantically, they're the unit name which only sometimes matches the binary name
      - rezonant 2y ago
        Actually no, they are prefixed by the binary name and the PID number. Technically they are prefixed with: `<date> <host> <binary>[<pid>]`. You can then use `systemctl status <pid>` to identify the unit if you need to. I would imagine this is configurable, and this might be the configuration chosen by my distribution (since I have not changed it myself). It would actually be nice to show the unit name instead of the binary/PID combination, though not strictly necessary. EDIT: Ooh, systemd 239 adds `journalctl -o with-unit` to do this exact thing. There are lots of other formats you can choose from as well. EDIT 2: Unfortunately there's no way to set this as default, you must use `-o with-unit` each time or set up a shell alias :-\
  - superb_dev 2y ago
    Well `-u` refers to the unit file, so I would start with `sudo systemctl status` which lists the status of active unit files. I bet I could find the unit name I'm looking for there. If not, then `sudo systemctl list-units` should have it. (and you can grep the output of both) Systemd and Journald are less opaque than I used to think. Even if you don't want to learn the commands, all of its unit files (and the relationships between them) are available through the filesystem. Most of your unit files will be in `/etc/systemd/system`, and the active relationships between units are expressed through soft links.
    - aeonik 2y ago
      How would you exclude certain units from your logs? Let's say that Auditd is spamming your logs, and you just want to exclude those? Can you filter on search? How do you prevent applications from writing to Systemd at all?
      - superb_dev 2y ago
        I would start with `man systemd`, or maybe `journalctl —help`. If you don’t want a service to log to systemd, then don’t run it in systemd?
        pcthrowaway 2y ago
        I think you can also configure the journal per unit right? Don't remember how exactly, at the very least you can stream logs to a different system. There are also services which will just use their own logging convention for some reason. Honestly it's easier just to go with journalctl for everything you can.
    - ramses0 2y ago
      systemctl does not have the Unix nature.
  - ciupicri 2y ago
    # journalctl -f _PID=${your_pid} # option 1 # systemctl status ${your_pid} # option 2 [1]: https://www.freedesktop.org/software/systemd/man/latest/journalctl.html#Examples https://www.freedesktop.org/software/systemd/man/latest/jour... [2]: https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#status%20PATTERN%E2%80%A6%7CPID%E2%80%A6%5D https://www.freedesktop.org/software/systemd/man/latest/syst...
    - Brian_K_White 2y ago
      These absolutely SUCK as answers to that question. They entirely miss the point. They provide a specific answer to a general problem. The problem with systemnd is it assumes that it's possible for all needs to be predicted and accounted for ahead of time. While "look around at directories and files, and grep within them" works after the fact without any special knowledge or tools. The person who wrote the log file did not need to know how someone will maybe try to access it 23 years later on a different OS. It's just a regular file that can be read by anything over any kind of channel on any os. The person finding themselves needing to read the file does not need to have any particular command installed, or installable, or runnable. It does not require the happy path in order to work. I have a joke I always say, often self-deprecating making fun of my own self for the way I do things sometimes, but also when I'm trying to commiserate with a customer so they don't feel intimidated by "the expert" or "the engineer": "37 easy steps!" Every answer that starts with "it's simple, just journalctl ..." is FUCKING 37 easy steps. The very name of the program itself is a trainwreck. journalctl... it takes me 18 seconds just to type it. systemd is great for managing exquisitely washed masses of drone vms. It's utter and complete shit for direct administration, operation, development, debugging, flexibility, or custom integrations.
      - throwaway7356 2y ago
        > journalctl... it takes me 18 seconds just to type it. Sounds like you don't use a keyboard often enough to even remember where the keys are. Not sure I would expect deep (or even shallow) knowledge on UNIX system administration from such a person.
      - robertlagrant 2y ago
  - sunshowers 2y ago
    You do have to learn a few new things, yes. But it's not too difficult. We all have to learn new things sometimes. I spent around 15 minutes a few weeks ago learning how to do a few things with journalctl, and I came away from it with a great appreciation for its power.
  - rcxdude 2y ago
    systemctl status <pid> will show you the unit and helpfully the last few lines of log from it. Journalctl -u <unit name> will then show you the full logs
  - JoshTriplett 2y ago
    > Oh, I don´t know the name of the log for this process I can see in 'ps aux', With services using journald, there's no "name of the log" because everything's in the journal, so that part isn't a problem. Rather than "is this in auth.log or syslog or thisservice.log", it'll always be in the journal. > let me cd into /var/log and see what filenames I can find You can filter journal entries by unit (-u) or by service identifier (-t). Often, though, I find it really useful to be able to see the adjacent entries from other services at the same time, since that can give some indication of what's happening on the system that caused an issue. > or grep everything until I can find a couple of words that make some sense so I can keep digging further journalctl --grep, or more conveniently, journalctl and then / to search. > The lack of explorability in journalctl It's explorable by dozens of different axes, and if all of those aren't sufficient, you always can get the whole thing as text and run any command you like on it, or get it in structured form and do structured queries on it.
- deleted 2y ago
  [deleted]
hughesjj 2y ago
My 2c, I definitely prefer systemd boot+networkd, fine with journald, don't use the others enough to have an opinion.
- iforgotpassword 2y ago
  Same. Init system is awesome, love .ini files, networkd is as simple as it can be, journald is fine. Love systemd-nspawn for simple containers as it feels more lightweight than docker. Especially for quick one-offs where I don't need isolated networking you basically have chroot on steroids with "systemd-nspawn -D /path/to/fstree".
lolinder 2y ago
> I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`. journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there. > I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename. My memory of this time was that every single application had its own unique method of handling logs and its own unique location for storing them. So sure, once you found the logs you didn't have to Google how to find the relevant dates if the application is using logrotate as you describe, but finding the logs in the first place was always a challenge. Systemd is nice in that it provides a single place where all logs go and a single interface for navigating them.
- saghm 2y ago
  > journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there Yeah, this is a bit of an odd complaint. If I were this bothered by having to type those, I'd just make an alias to "journalctl --follow --nopager"` and would have forgotten about it years ago.
  - stoperaticless 2y ago
    Never tried it, but if the complainer wrote: tail -f *.logs Which involves glob’ing. I assume it allows to monitor several services at the same time. Can systemd do that?
    - tapoxi 2y ago
      Yes, journalctl -u postgres -u redis -f etc
      - mintplant 2y ago
        Or just `journalctl -f` to follow all logs.
    - _flux 2y ago
jordemort 2y ago
systemd-nspawn is actually pretty great, and is mostly just a tool that exposes functionality that systemd already needs for other reasons. IIRC it was originally developed to aid in testing systemd itself, and was initially shipped as an unsupported extra in case other people found it useful. systemd-resolved is the only way I've ever been able to get dockerd to play nice with Tailscale DNS; it's also the best way that I've found to get a system to pick different upstream DNS servers depending on domain. The alternative is hand-rolling it with dnsmasq or something similar. I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet, but for servers with relatively static network configurations, I greatly prefer systemd-networkd over any of the various implementations of ifupdown.
- intelfx 2y ago
  > I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet I don't think there is one. systemd-networkd is explicitly not for dynamic environments like personal laptops or workstations; it has a scope and NetworkManager is largely outside it.
sunshowers 2y ago
I've recently become a fan of the `-x` flag to journalctl, which adds additional context to log entries and even suggests remediations at times. I've identified and fixed several issues on my Linux systems that way. Kind of hard to get that with tail -f!
- rezonant 2y ago
  Ooh nice. What are some examples of things it will catch?
- dumpsterdiver 2y ago
  My goto is journalctl -xf. Best of both worlds :)
- aeonik 2y ago
  How does it work? Can I pipe non-journald logs into the -x flag to take advantage of this lookup?
rezonant 2y ago
> This was a really big pain, yes, but I also remember how I could `tail -f *.logs`. Others point out that journalctl supports `-f` and `-n` for determining how much journal to return, but for me what's great about journalctl is that I do not need to know where the program is logging to, how it rotates logs (or how it doesn't), or anything else. The program does not need special permissions to a log location in /var/log or /usr/var/log or ~/... or anything else, it just outputs using stdout and it gets captured. When I want to look at logs, I just use the same moniker that I use to control the service: its unit name. It is so incredibly nice- I do not want to go back to the bad old days of spelunking through /var/log and friends. > --no-pager To be clear, if you pipe journalctl somewhere, it will automatically skip the pager and just work. `--no-pager` is most useful for when you want to print some unpiped content in an automated script to avoid having a non-existent user exit the pager.
ranger207 2y ago
I remember, and sometimes still have to deal with, programs that _didn't_ dump their logs in /var/log, forcing me to figure out where the log are in the first place
SoftTalker 2y ago
Those old init scripts were pretty boilerplate and not that hard to write. And in the case of OpenBSD (and maybe other BSDs?) the scripts for /etc/rc.d are usually just a few lines. With the traditional init system I knew what was going on, and if I didn't, find and grep were your friends. With systemd I'm googling everything every time. The systemctl man page is over 1,100 lines. OpenBSD's rcctl man page is 119 lines. I'm sure there are use cases for systemd but for running a server that provides a single or a handful of services, it seems like complete overkill.

by agilob 2y ago

superb_dev 2y ago

The follow argument is identical to tail `-f`. `sudo journalctl -f -u <service name>`

rangerelf 2y ago
And what would be the equivalent to, "Oh, I don´t know the name of the log for this process I can see in 'ps aux', let me cd into /var/log and see what filenames I can find ... or grep everything until I can find a couple of words that make some sense so I can keep digging further"? The lack of explorability in journalctl, the "need" to keep everything locked behind their own flavor of tools and magic file types, is what makes the rest of us abhor them.
- iforgotpassword 2y ago
  Huh, isn't grepping journalctl output pretty much the same? It even prefixes messages by the binary name.
  - mdaniel 2y ago
    Pedantically, they're the unit name which only sometimes matches the binary name
    - rezonant 2y ago
      Actually no, they are prefixed by the binary name and the PID number. Technically they are prefixed with: `<date> <host> <binary>[<pid>]`. You can then use `systemctl status <pid>` to identify the unit if you need to. I would imagine this is configurable, and this might be the configuration chosen by my distribution (since I have not changed it myself). It would actually be nice to show the unit name instead of the binary/PID combination, though not strictly necessary. EDIT: Ooh, systemd 239 adds `journalctl -o with-unit` to do this exact thing. There are lots of other formats you can choose from as well. EDIT 2: Unfortunately there's no way to set this as default, you must use `-o with-unit` each time or set up a shell alias :-\
- superb_dev 2y ago
  Well `-u` refers to the unit file, so I would start with `sudo systemctl status` which lists the status of active unit files. I bet I could find the unit name I'm looking for there. If not, then `sudo systemctl list-units` should have it. (and you can grep the output of both) Systemd and Journald are less opaque than I used to think. Even if you don't want to learn the commands, all of its unit files (and the relationships between them) are available through the filesystem. Most of your unit files will be in `/etc/systemd/system`, and the active relationships between units are expressed through soft links.
  - aeonik 2y ago
    How would you exclude certain units from your logs? Let's say that Auditd is spamming your logs, and you just want to exclude those? Can you filter on search? How do you prevent applications from writing to Systemd at all?
    - superb_dev 2y ago
      I would start with `man systemd`, or maybe `journalctl —help`. If you don’t want a service to log to systemd, then don’t run it in systemd?
      - pcthrowaway 2y ago
        I think you can also configure the journal per unit right? Don't remember how exactly, at the very least you can stream logs to a different system. There are also services which will just use their own logging convention for some reason. Honestly it's easier just to go with journalctl for everything you can.
  - ramses0 2y ago
    systemctl does not have the Unix nature.
- ciupicri 2y ago
  # journalctl -f _PID=${your_pid} # option 1 # systemctl status ${your_pid} # option 2 [1]: https://www.freedesktop.org/software/systemd/man/latest/journalctl.html#Examples https://www.freedesktop.org/software/systemd/man/latest/jour... [2]: https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#status%20PATTERN%E2%80%A6%7CPID%E2%80%A6%5D https://www.freedesktop.org/software/systemd/man/latest/syst...
  - Brian_K_White 2y ago
    These absolutely SUCK as answers to that question. They entirely miss the point. They provide a specific answer to a general problem. The problem with systemnd is it assumes that it's possible for all needs to be predicted and accounted for ahead of time. While "look around at directories and files, and grep within them" works after the fact without any special knowledge or tools. The person who wrote the log file did not need to know how someone will maybe try to access it 23 years later on a different OS. It's just a regular file that can be read by anything over any kind of channel on any os. The person finding themselves needing to read the file does not need to have any particular command installed, or installable, or runnable. It does not require the happy path in order to work. I have a joke I always say, often self-deprecating making fun of my own self for the way I do things sometimes, but also when I'm trying to commiserate with a customer so they don't feel intimidated by "the expert" or "the engineer": "37 easy steps!" Every answer that starts with "it's simple, just journalctl ..." is FUCKING 37 easy steps. The very name of the program itself is a trainwreck. journalctl... it takes me 18 seconds just to type it. systemd is great for managing exquisitely washed masses of drone vms. It's utter and complete shit for direct administration, operation, development, debugging, flexibility, or custom integrations.
    - throwaway7356 2y ago
      > journalctl... it takes me 18 seconds just to type it. Sounds like you don't use a keyboard often enough to even remember where the keys are. Not sure I would expect deep (or even shallow) knowledge on UNIX system administration from such a person.
    - robertlagrant 2y ago
- sunshowers 2y ago
  You do have to learn a few new things, yes. But it's not too difficult. We all have to learn new things sometimes. I spent around 15 minutes a few weeks ago learning how to do a few things with journalctl, and I came away from it with a great appreciation for its power.
- rcxdude 2y ago
  systemctl status <pid> will show you the unit and helpfully the last few lines of log from it. Journalctl -u <unit name> will then show you the full logs
- JoshTriplett 2y ago
  > Oh, I don´t know the name of the log for this process I can see in 'ps aux', With services using journald, there's no "name of the log" because everything's in the journal, so that part isn't a problem. Rather than "is this in auth.log or syslog or thisservice.log", it'll always be in the journal. > let me cd into /var/log and see what filenames I can find You can filter journal entries by unit (-u) or by service identifier (-t). Often, though, I find it really useful to be able to see the adjacent entries from other services at the same time, since that can give some indication of what's happening on the system that caused an issue. > or grep everything until I can find a couple of words that make some sense so I can keep digging further journalctl --grep, or more conveniently, journalctl and then / to search. > The lack of explorability in journalctl It's explorable by dozens of different axes, and if all of those aren't sufficient, you always can get the whole thing as text and run any command you like on it, or get it in structured form and do structured queries on it.
deleted 2y ago
[deleted]

hughesjj 2y ago

My 2c, I definitely prefer systemd boot+networkd, fine with journald, don't use the others enough to have an opinion.

iforgotpassword 2y ago
Same. Init system is awesome, love .ini files, networkd is as simple as it can be, journald is fine. Love systemd-nspawn for simple containers as it feels more lightweight than docker. Especially for quick one-offs where I don't need isolated networking you basically have chroot on steroids with "systemd-nspawn -D /path/to/fstree".

lolinder 2y ago

> I could `tail -f *.logs`. I remember how I didn't need to remember about `--no-pager` and `--follow`. journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there. > I remember how I didn't have to google how to find logs between 10 days ago and 4 days ago, because the logs would be in a .tar created by logrotate with a date in the filename. My memory of this time was that every single application had its own unique method of handling logs and its own unique location for storing them. So sure, once you found the logs you didn't have to Google how to find the relevant dates if the application is using logrotate as you describe, but finding the logs in the first place was always a challenge. Systemd is nice in that it provides a single place where all logs go and a single interface for navigating them.

saghm 2y ago
> journalctl supports -f, which by your own account you were already using for tail, so I'm not clear what's worse there Yeah, this is a bit of an odd complaint. If I were this bothered by having to type those, I'd just make an alias to "journalctl --follow --nopager"` and would have forgotten about it years ago.
- stoperaticless 2y ago
  Never tried it, but if the complainer wrote: tail -f *.logs Which involves glob’ing. I assume it allows to monitor several services at the same time. Can systemd do that?
  - tapoxi 2y ago
    Yes, journalctl -u postgres -u redis -f etc
    - mintplant 2y ago
      Or just `journalctl -f` to follow all logs.
  - _flux 2y ago

jordemort 2y ago

systemd-nspawn is actually pretty great, and is mostly just a tool that exposes functionality that systemd already needs for other reasons. IIRC it was originally developed to aid in testing systemd itself, and was initially shipped as an unsupported extra in case other people found it useful. systemd-resolved is the only way I've ever been able to get dockerd to play nice with Tailscale DNS; it's also the best way that I've found to get a system to pick different upstream DNS servers depending on domain. The alternative is hand-rolling it with dnsmasq or something similar. I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet, but for servers with relatively static network configurations, I greatly prefer systemd-networkd over any of the various implementations of ifupdown.

intelfx 2y ago
> I admit I haven't really seen a huge advantage to using systemd-networkd over NetworkManager yet I don't think there is one. systemd-networkd is explicitly not for dynamic environments like personal laptops or workstations; it has a scope and NetworkManager is largely outside it.

sunshowers 2y ago

I've recently become a fan of the `-x` flag to journalctl, which adds additional context to log entries and even suggests remediations at times. I've identified and fixed several issues on my Linux systems that way. Kind of hard to get that with tail -f!

rezonant 2y ago
Ooh nice. What are some examples of things it will catch?
dumpsterdiver 2y ago
My goto is journalctl -xf. Best of both worlds :)
aeonik 2y ago
How does it work? Can I pipe non-journald logs into the -x flag to take advantage of this lookup?

rezonant 2y ago

> This was a really big pain, yes, but I also remember how I could `tail -f *.logs`. Others point out that journalctl supports `-f` and `-n` for determining how much journal to return, but for me what's great about journalctl is that I do not need to know where the program is logging to, how it rotates logs (or how it doesn't), or anything else. The program does not need special permissions to a log location in /var/log or /usr/var/log or ~/... or anything else, it just outputs using stdout and it gets captured. When I want to look at logs, I just use the same moniker that I use to control the service: its unit name. It is so incredibly nice- I do not want to go back to the bad old days of spelunking through /var/log and friends. > --no-pager To be clear, if you pipe journalctl somewhere, it will automatically skip the pager and just work. `--no-pager` is most useful for when you want to print some unpiped content in an automated script to avoid having a non-existent user exit the pager.

ranger207 2y ago

I remember, and sometimes still have to deal with, programs that _didn't_ dump their logs in /var/log, forcing me to figure out where the log are in the first place

SoftTalker 2y ago

Those old init scripts were pretty boilerplate and not that hard to write. And in the case of OpenBSD (and maybe other BSDs?) the scripts for /etc/rc.d are usually just a few lines. With the traditional init system I knew what was going on, and if I didn't, find and grep were your friends. With systemd I'm googling everything every time. The systemctl man page is over 1,100 lines. OpenBSD's rcctl man page is 119 lines. I'm sure there are use cases for systemd but for running a server that provides a single or a handful of services, it seems like complete overkill.