3 ms·

> Because grepping through simple rotated log files is a billion times faster than journalctl This is annoying, but there's a "workaround" $ time journalctl

> Because grepping through simple rotated log files is a billion times faster than journalctl This is annoying, but there's a "workaround" $ time journalctl | grep "sshd" | wc -l 12622 journalctl 76.04s user 0.71s system 99% cpu 1:17.09 total grep --color=always --no-messages --binary-files=without-match "sshd" 1.28s user 1.69s system 3% cpu 1:17.08 total wc -l 0.00s user 0.00s system 0% cpu 1:17.08 total $ time journalctl > /tmp/all.log && time wc -l /tmp/all.log journalctl > /tmp/all.log 76.05s user 1.22s system 99% cpu 1:17.56 total 16106878 /tmp/all.log wc -l /tmp/all.log 0.03s user 0.20s system 98% cpu 0.236 total # THE SOLUTION $ time journalctl --grep=sshd | wc -l 5790 journalctl --grep=sshd 28.97s user 0.26s system 99% cpu 29.344 total wc -l 0.00s user 0.00s system 0% cpu 29.344 total It's annoying that you need to use the grep flag instead of piping into grep but it is not too hard to switch to that mindset. FWIW, I have gotten slightly faster results using the `--no-pager` flag but it is by such a trivial amount I'll never remember it > Sometimes docker starts before the filesystem is mounted. Look at the output of `systemctl cat docker.service` and you'll see an "After" "Wants" and "Requires" arguments in the unit. You're going to want to edit that (I strongly suggest you use `sudo systemctl edit docker.service`, for reasons stated above) and make sure that it comes after the drive you want mounted. You an set the Requires argument to require that drive so it shouldn't ever start before Alternatively, you can make the drive start earlier. But truthfully, I have no reason to have docker start this early. Here's a link to the target order diagram[0] and Arch wiki[1]. Thing that gets messy is that everyone kinda lazily uses multi-user.target [0] https://www.freedesktop.org/software/systemd/man/latest/bootup.html https://www.freedesktop.org/software/systemd/man/latest/boot... [1] https://wiki.archlinux.org/title/Systemd#Targets https://wiki.archlinux.org/title/Systemd#Targets

bombela 2y ago
journalctl --grep is still much slower than grep on simple files. And if you use ripgrep like I prefer, it's even faster still. No really I don't think journactl makes sense in its current form. It's just broken by design. I do like the potential of it. But not the implementation.
- godelski 2y ago
  > journalctl --grep is still much slower than grep on simple files Idk what to tell you. You had a problem, showed the commands you used and the times it took. So I showed you a different way that took less than half the time to just dump and grep (which you said was faster) My results don't match your conclusion. > if you use ripgrep I've been burned by ripgrep too many times. It's a crazy design choice, to me, to default filter things. Especially to diverge from grep! The only thing I expect grep to ignore are the system hidden files (dotfiles) and anything I explicitly tell it to. I made a git ignore file, not a grep ignore file. I frequently want to grep things I'm ignoring with git. One of my most frequent uses of grep is looking through builds artifacts and logs. Things I'd never want to push. And that's where many people get burned, they think these files just disappeared! The maintainer also has been pretty rude to me about this on HN. I can get we have a different opinion but it's still crazy to think people won't be caught off guard by this behavior. Its name is literally indicating it's a grep replacement. Yeah, I'm surprised its behavior significantly diverges from grep lol
  - burntsushi 2y ago
    > The maintainer also has been pretty rude to me about this on HN. This is AFAIK the only other interaction we've had: https://news.ycombinator.com/item?id=41051587 https://news.ycombinator.com/item?id=41051587 If there are other interactions we've had, feel free to link them. Then others can decide how rude I'm being instead of relying only on your characterization. > but it's still crazy to think people won't be caught off guard by this behavior Straw-manning is also crazy. :-) People have and will absolutely be caught off guard by the behavior. On the flip side, as I said 9 months ago, ripgrep's default behavior is easily one of the most cited positive features of ripgrep aside from its performance. The other crazy thing here is... you don't have to use ripgrep! It is very specifically intended as a departure from traditional grep behavior. Because if you want traditional grep behavior, then you can just use grep. Hence why ripgrep's binary name is not `grep`, unlike the many implementations of POSIX grep. > Its name is literally indicating it's a grep replacement. I also tried to correct this 9 months ago too. See also: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#posix4ever https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos... For anyone else following along at home, if you want ripgrep to search the same files that GNU grep searches, then do `rg -uuu`. Or, if you don't want ripgrep to respect your gitignores but ignore hidden and binary files, then do `rg -u`. It makes sense that folks might be caught off guard by ripgrep's default filtering. This is why I try to mitigate it by stating very clearly that it is going to ignore stuff by default in the first one or two sentences about ripgrep (README, man page, CHANGELOG, project description). I also try to mitigate it by making it very easy to disable this default behavior. These mitigations exist precisely because I know the default behavior can be surprising, in direct contradiction to "but it's still crazy to think people won't be caught off guard by this behavior."
egorfine 2y ago
> you need to use the grep flag instead of piping into grep I don't. It's the journalctl that does. And it can absolutely fuck off with everything and all of it. Log files must be in form of text files. This worked for decades and there is no foreseeable future where this stops working or ceases to be a solution for OS log collection.
egorfine 2y ago
> Look at the output of `systemctl cat docker.service` No. Either the initsystem works in a straightforward way or it doesn't. As soon as we need special commands to just get an impression of what's happening with the service, this init system can - again - fuck off with all that unnecessary complexity. Init must be simple. Unfortunately it isn't anymore. Unfortunately, systemd will not fuck off, it's too late for that. Unfortunately we now have to deal with the consequences of letting LP & co do what they did.
- godelski 2y ago
  > As soon as we need special commands to just get an impression of what's happening with the service, I agree this is bad design. I do not intend to defend `--grep` just was trying to help solve the issue. I 100% believe that this creates an unreasonable expectation of the user and that piping to grep should be expected. Although, my results showed equal times piping to grep and dumping to file then grepping that file. IFF `--grep` is operating in parallel, then I think that's fine that it is faster and I'll take back my critique since it is doing additional functionality and isn't necessary. That situation would be "things work normally, but here's a flag for additional optimization." Is the slowdown the file access? I do notice that it gets "choppy" if I just dump `journalctl --no-pager` but I was doing it over ssh so idk what the bottleneck was. IO is often a pain point (it pains me how often people untar with verbose...).
  - egorfine 2y ago
    > things work normally With text log files. > but here's a flag for additional optimization Which wouldn't be even needed in the first place if that very tool that wants this flag just simply did not exist.

bombela 2y ago

journalctl --grep is still much slower than grep on simple files. And if you use ripgrep like I prefer, it's even faster still. No really I don't think journactl makes sense in its current form. It's just broken by design. I do like the potential of it. But not the implementation.

godelski 2y ago
> journalctl --grep is still much slower than grep on simple files Idk what to tell you. You had a problem, showed the commands you used and the times it took. So I showed you a different way that took less than half the time to just dump and grep (which you said was faster) My results don't match your conclusion. > if you use ripgrep I've been burned by ripgrep too many times. It's a crazy design choice, to me, to default filter things. Especially to diverge from grep! The only thing I expect grep to ignore are the system hidden files (dotfiles) and anything I explicitly tell it to. I made a git ignore file, not a grep ignore file. I frequently want to grep things I'm ignoring with git. One of my most frequent uses of grep is looking through builds artifacts and logs. Things I'd never want to push. And that's where many people get burned, they think these files just disappeared! The maintainer also has been pretty rude to me about this on HN. I can get we have a different opinion but it's still crazy to think people won't be caught off guard by this behavior. Its name is literally indicating it's a grep replacement. Yeah, I'm surprised its behavior significantly diverges from grep lol
- burntsushi 2y ago
  > The maintainer also has been pretty rude to me about this on HN. This is AFAIK the only other interaction we've had: https://news.ycombinator.com/item?id=41051587 https://news.ycombinator.com/item?id=41051587 If there are other interactions we've had, feel free to link them. Then others can decide how rude I'm being instead of relying only on your characterization. > but it's still crazy to think people won't be caught off guard by this behavior Straw-manning is also crazy. :-) People have and will absolutely be caught off guard by the behavior. On the flip side, as I said 9 months ago, ripgrep's default behavior is easily one of the most cited positive features of ripgrep aside from its performance. The other crazy thing here is... you don't have to use ripgrep! It is very specifically intended as a departure from traditional grep behavior. Because if you want traditional grep behavior, then you can just use grep. Hence why ripgrep's binary name is not `grep`, unlike the many implementations of POSIX grep. > Its name is literally indicating it's a grep replacement. I also tried to correct this 9 months ago too. See also: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#posix4ever https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos... For anyone else following along at home, if you want ripgrep to search the same files that GNU grep searches, then do `rg -uuu`. Or, if you don't want ripgrep to respect your gitignores but ignore hidden and binary files, then do `rg -u`. It makes sense that folks might be caught off guard by ripgrep's default filtering. This is why I try to mitigate it by stating very clearly that it is going to ignore stuff by default in the first one or two sentences about ripgrep (README, man page, CHANGELOG, project description). I also try to mitigate it by making it very easy to disable this default behavior. These mitigations exist precisely because I know the default behavior can be surprising, in direct contradiction to "but it's still crazy to think people won't be caught off guard by this behavior."

egorfine 2y ago

> you need to use the grep flag instead of piping into grep I don't. It's the journalctl that does. And it can absolutely fuck off with everything and all of it. Log files must be in form of text files. This worked for decades and there is no foreseeable future where this stops working or ceases to be a solution for OS log collection.

egorfine 2y ago

> Look at the output of `systemctl cat docker.service` No. Either the initsystem works in a straightforward way or it doesn't. As soon as we need special commands to just get an impression of what's happening with the service, this init system can - again - fuck off with all that unnecessary complexity. Init must be simple. Unfortunately it isn't anymore. Unfortunately, systemd will not fuck off, it's too late for that. Unfortunately we now have to deal with the consequences of letting LP & co do what they did.

godelski 2y ago
> As soon as we need special commands to just get an impression of what's happening with the service, I agree this is bad design. I do not intend to defend `--grep` just was trying to help solve the issue. I 100% believe that this creates an unreasonable expectation of the user and that piping to grep should be expected. Although, my results showed equal times piping to grep and dumping to file then grepping that file. IFF `--grep` is operating in parallel, then I think that's fine that it is faster and I'll take back my critique since it is doing additional functionality and isn't necessary. That situation would be "things work normally, but here's a flag for additional optimization." Is the slowdown the file access? I do notice that it gets "choppy" if I just dump `journalctl --no-pager` but I was doing it over ssh so idk what the bottleneck was. IO is often a pain point (it pains me how often people untar with verbose...).
- egorfine 2y ago
  > things work normally With text log files. > but here's a flag for additional optimization Which wouldn't be even needed in the first place if that very tool that wants this flag just simply did not exist.