6 ms·

Man this one annoys me. I have this server running a docker container with a specific application. And it writes to a specific filesystem (properly mount binde

Man this one annoys me. I have this server running a docker container with a specific application. And it writes to a specific filesystem (properly mount binded inside the container of course). Sometimes docker starts before the filesystem is mounted. I know systemd can be taught about this but I haven't bothered. Because every time I have to do something in systemd, I have to read some nasty obscure doc. I need know how and where the config should go. I did manage to disable journalctl at least. Because grepping through simple rotated log files is a billion times faster than journalctl. See my comment and the whole thread https://github.com/systemd/systemd/issues/2460#issuecomment-1577671723 https://github.com/systemd/systemd/issues/2460#issuecomment-... I like the concept of systemd. Not the implementation and its leader.

egorfine 1y ago
> I did manage to disable journalctl at least My goodness. Absolutely fuck journald - a solution in search of a problem. I have created a bunch of different scripts to init my instances [1] on all projects. I do it differently from time to time, but one thing they all have in common is that journald gets removed and disabled. [1] https://github.com/egorFiNE/desystemd https://github.com/egorFiNE/desystemd
ptsneves 1y ago
> I know systemd can be taught about this but I haven't bothered. I think After=<your .mount> will work. If you believe it can be taught (and it can) why do you blame your lack knowledge on the tool is not a strong argument against the quality of the tool. > Because grepping through simple rotated log files is a billion times faster than journalctl. `journalctl -D <directory of the journal files> | grep ...` will give you what you want. Systemd is incredibly configurable and that makes its documentation daunting but damn it does everything you want it to do. I used it in embedded systems and it is just amazing. In old times lots of custom programs and management daemons needed to be written. Now it is just a bunch of conf files and it all magically works. The most fair criticism is it does not follow the 'everything is a file philosophy' of Unix, and this makes discoverability and traditional workflows awkward. Even so it is a tool: if it does what you want, but you don't want to spend time understanding it, it is hardly the fault of the tool. I strongly recommend learning it, there will be many Ah-ha moments.
- egorfine 1y ago
  > it does everything you want it to do It does everything no one asked it to. I'm sure they will come up with obscure reasons why the next perfectly working tool has to be destroyed and redone by the only authority - the LP team. Like cron, sudo and yes - logging. > journalctl -D ... will give you what you want Look, I don't need the help of journalctl to grep through text. I can simply grep thru text. > I used it in embedded systems Good luck in a few years when you are flying home on the next Boeing 737-MAX-100800 and it fails mid flight because systemd decided to shut down some service because fuck you that's why. > it does not follow the 'everything is a file philosophy' It does not follow 'everything is a separate simple tool working in concert with others'. systemd is a monolith disguised to look like a set of separate projects. > don't want to spend time understanding it, it is hardly the fault of the tool It is, if we had proper tools for decades and they did work. I'm not a retrograde guy, quite the opposite, but the ideology that LP and the rest are shoving down our throats brings up natural defiance. > there will be many Ah-ha moments No doubts. systemd unit files and systemd-as-PID1 is excellent. It was NOT excellent for the whole time but now it is. The rest? Designed to frustrate and establish dominance, that's it.
- bombela 1y ago
  You can also add fake filesystem parameters to the fstab entries that are parsed by systemd. Here the doc on this. You might be forgiven for having missed it. It's under the section fstab. https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html https://www.freedesktop.org/software/systemd/man/latest/syst... If you had followed my link to the systemd issue, you might have seen the commands I ran, as well as the tests and feedback of everybody on the issue. You might reach the conclusion that journalctl is fundamentally broken beyond repair. edit: added link to systemd doc
godelski 1y ago
> Because grepping through simple rotated log files is a billion times faster than journalctl This is annoying, but there's a "workaround" $ time journalctl | grep "sshd" | wc -l 12622 journalctl 76.04s user 0.71s system 99% cpu 1:17.09 total grep --color=always --no-messages --binary-files=without-match "sshd" 1.28s user 1.69s system 3% cpu 1:17.08 total wc -l 0.00s user 0.00s system 0% cpu 1:17.08 total $ time journalctl > /tmp/all.log && time wc -l /tmp/all.log journalctl > /tmp/all.log 76.05s user 1.22s system 99% cpu 1:17.56 total 16106878 /tmp/all.log wc -l /tmp/all.log 0.03s user 0.20s system 98% cpu 0.236 total # THE SOLUTION $ time journalctl --grep=sshd | wc -l 5790 journalctl --grep=sshd 28.97s user 0.26s system 99% cpu 29.344 total wc -l 0.00s user 0.00s system 0% cpu 29.344 total It's annoying that you need to use the grep flag instead of piping into grep but it is not too hard to switch to that mindset. FWIW, I have gotten slightly faster results using the `--no-pager` flag but it is by such a trivial amount I'll never remember it > Sometimes docker starts before the filesystem is mounted. Look at the output of `systemctl cat docker.service` and you'll see an "After" "Wants" and "Requires" arguments in the unit. You're going to want to edit that (I strongly suggest you use `sudo systemctl edit docker.service`, for reasons stated above) and make sure that it comes after the drive you want mounted. You an set the Requires argument to require that drive so it shouldn't ever start before Alternatively, you can make the drive start earlier. But truthfully, I have no reason to have docker start this early. Here's a link to the target order diagram[0] and Arch wiki[1]. Thing that gets messy is that everyone kinda lazily uses multi-user.target [0] https://www.freedesktop.org/software/systemd/man/latest/bootup.html https://www.freedesktop.org/software/systemd/man/latest/boot... [1] https://wiki.archlinux.org/title/Systemd#Targets https://wiki.archlinux.org/title/Systemd#Targets
- bombela 1y ago
  journalctl --grep is still much slower than grep on simple files. And if you use ripgrep like I prefer, it's even faster still. No really I don't think journactl makes sense in its current form. It's just broken by design. I do like the potential of it. But not the implementation.
  - godelski 1y ago

egorfine 1y ago

> I did manage to disable journalctl at least My goodness. Absolutely fuck journald - a solution in search of a problem. I have created a bunch of different scripts to init my instances [1] on all projects. I do it differently from time to time, but one thing they all have in common is that journald gets removed and disabled. [1] https://github.com/egorFiNE/desystemd https://github.com/egorFiNE/desystemd

ptsneves 1y ago

> I know systemd can be taught about this but I haven't bothered. I think After=<your .mount> will work. If you believe it can be taught (and it can) why do you blame your lack knowledge on the tool is not a strong argument against the quality of the tool. > Because grepping through simple rotated log files is a billion times faster than journalctl. `journalctl -D <directory of the journal files> | grep ...` will give you what you want. Systemd is incredibly configurable and that makes its documentation daunting but damn it does everything you want it to do. I used it in embedded systems and it is just amazing. In old times lots of custom programs and management daemons needed to be written. Now it is just a bunch of conf files and it all magically works. The most fair criticism is it does not follow the 'everything is a file philosophy' of Unix, and this makes discoverability and traditional workflows awkward. Even so it is a tool: if it does what you want, but you don't want to spend time understanding it, it is hardly the fault of the tool. I strongly recommend learning it, there will be many Ah-ha moments.

egorfine 1y ago
> it does everything you want it to do It does everything no one asked it to. I'm sure they will come up with obscure reasons why the next perfectly working tool has to be destroyed and redone by the only authority - the LP team. Like cron, sudo and yes - logging. > journalctl -D ... will give you what you want Look, I don't need the help of journalctl to grep through text. I can simply grep thru text. > I used it in embedded systems Good luck in a few years when you are flying home on the next Boeing 737-MAX-100800 and it fails mid flight because systemd decided to shut down some service because fuck you that's why. > it does not follow the 'everything is a file philosophy' It does not follow 'everything is a separate simple tool working in concert with others'. systemd is a monolith disguised to look like a set of separate projects. > don't want to spend time understanding it, it is hardly the fault of the tool It is, if we had proper tools for decades and they did work. I'm not a retrograde guy, quite the opposite, but the ideology that LP and the rest are shoving down our throats brings up natural defiance. > there will be many Ah-ha moments No doubts. systemd unit files and systemd-as-PID1 is excellent. It was NOT excellent for the whole time but now it is. The rest? Designed to frustrate and establish dominance, that's it.
bombela 1y ago
You can also add fake filesystem parameters to the fstab entries that are parsed by systemd. Here the doc on this. You might be forgiven for having missed it. It's under the section fstab. https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html https://www.freedesktop.org/software/systemd/man/latest/syst... If you had followed my link to the systemd issue, you might have seen the commands I ran, as well as the tests and feedback of everybody on the issue. You might reach the conclusion that journalctl is fundamentally broken beyond repair. edit: added link to systemd doc

godelski 1y ago

> Because grepping through simple rotated log files is a billion times faster than journalctl This is annoying, but there's a "workaround" $ time journalctl | grep "sshd" | wc -l 12622 journalctl 76.04s user 0.71s system 99% cpu 1:17.09 total grep --color=always --no-messages --binary-files=without-match "sshd" 1.28s user 1.69s system 3% cpu 1:17.08 total wc -l 0.00s user 0.00s system 0% cpu 1:17.08 total $ time journalctl > /tmp/all.log && time wc -l /tmp/all.log journalctl > /tmp/all.log 76.05s user 1.22s system 99% cpu 1:17.56 total 16106878 /tmp/all.log wc -l /tmp/all.log 0.03s user 0.20s system 98% cpu 0.236 total # THE SOLUTION $ time journalctl --grep=sshd | wc -l 5790 journalctl --grep=sshd 28.97s user 0.26s system 99% cpu 29.344 total wc -l 0.00s user 0.00s system 0% cpu 29.344 total It's annoying that you need to use the grep flag instead of piping into grep but it is not too hard to switch to that mindset. FWIW, I have gotten slightly faster results using the `--no-pager` flag but it is by such a trivial amount I'll never remember it > Sometimes docker starts before the filesystem is mounted. Look at the output of `systemctl cat docker.service` and you'll see an "After" "Wants" and "Requires" arguments in the unit. You're going to want to edit that (I strongly suggest you use `sudo systemctl edit docker.service`, for reasons stated above) and make sure that it comes after the drive you want mounted. You an set the Requires argument to require that drive so it shouldn't ever start before Alternatively, you can make the drive start earlier. But truthfully, I have no reason to have docker start this early. Here's a link to the target order diagram[0] and Arch wiki[1]. Thing that gets messy is that everyone kinda lazily uses multi-user.target [0] https://www.freedesktop.org/software/systemd/man/latest/bootup.html https://www.freedesktop.org/software/systemd/man/latest/boot... [1] https://wiki.archlinux.org/title/Systemd#Targets https://wiki.archlinux.org/title/Systemd#Targets

bombela 1y ago
journalctl --grep is still much slower than grep on simple files. And if you use ripgrep like I prefer, it's even faster still. No really I don't think journactl makes sense in its current form. It's just broken by design. I do like the potential of it. But not the implementation.
- godelski 1y ago