3 ms·

I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower t

I really want to love journald (it sounds like it's aiming for a good system) but I share the other commenter(s)' frustration here about journald being slower than just pulling out ripgrep on regular files. We have some services at work that log to text files and some to journald. The log volume to file is >> the log volume to journald. Yet `rg query myservice.2026-08-01.log` seems to always wind up being faster and better than something like `journalctl -u myservice.service --since '2026-08-01' --until '2026-08-02' -g 'query'`. (The tab completion and discoverability is also better, I guess)

otterley 2mo ago
What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.
- bombela 2mo ago
  I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020. $ time journalctl > /tmp/all.log real 1m11.364s user 0m52.299s sys 0m6.540s $ time wc -l /tmp/all.log 3659597 /tmp/all.log real 0m0.152s user 0m0.056s sys 0m0.096s $ time journalctl | grep sshd | wc -l 12944 real 0m53.973s user 0m49.535s sys 0m5.210s $ time grep sshd /tmp/all.log | wc -l 12944 real 0m0.429s user 0m0.332s sys 0m0.100s https://github.com/systemd/systemd/issues/2460#issuecomment-1577671723 https://github.com/systemd/systemd/issues/2460#issuecomment-...
  - marginalia_nu 2mo ago
    I'm no fan of journald, but I have some methodological issues with this test. The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either. Running this between calls should flush the changes to disk and then drop the page cache, making for a fairer test. $ sudo sync $ echo 3 | sudo tee /proc/sys/vm/drop_caches
    - soneil 2mo ago
      I tried similar. Each command looped 5 times, ignore the best and worst times, and average the mean three. journalctl >/tmp/all.log 0m59.763s journalctl | grep -c sshd 0m57.767s (otherwise >all is the only example encumbered by write speeds) wc -l /tmp/all.log 0m0.336s grep -c sshd /tmp/all.log 0m1.776s
      - otterley 2mo ago
        You didn't use the query feature of journalctl, so you basically had it do an entire table conversion. Of course that's not going to show its performance value. This is what you should be running for a proper comparison: 1. `echo 3 | sudo tee /proc/sys/vm/drop_caches` 2. `time journalctl -u ssh.service >/dev/null` 3. `journalctl >/tmp/all.log` 4. `echo 3 | sudo tee /proc/sys/vm/drop_caches` 5. `time grep -q sshd /tmp/all.log`
- VGHN7XDuOXPAzol 2mo ago
  Sure, I'll try my best to put some concrete numbers on it: On a file of ~1M lines (350 MB), ripgrep returns a query matching 22k lines in 1.9s. (Separately: We have gzip log rotation for old logs. A log with 1.9M lines (106 MB gzipped) on disk can be searched from cold in 1.3s by ripgrep, whilst still being ergonomic. Maybe a different compressor is better?) On a service unit filter over ~218k lines, `journalctl -u service --since <...> -g query` returns 4.8k matching lines in 4.127s (and this is after having warmed the cache by returning the unfiltered query cold, in 11.9s. I don't have root to clear the disk cache I should have ran the filtered query first but ah well, since the difference is already so large it doesn't matter that the filtered journalctl query gets an unfair advantage) Comparing input rates: ripgrep (plain): 576k line/sec ripgrep (gz): 1.4M line/sec journalctl (-u, --since, -g): 53k line/sec I don't have a great deal of understanding of journald's internals, so perhaps there is some variable here that is unreasonably unfavourable to journald. Transcript user@machine:~/log$ time rg query aservice.log | wc -l 22628 real 0m1.859s user 0m0.109s sys 0m0.909s user@machine:~/log$ wc -l aservice.log 1071606 aservice4.log user@machine:~$ time journalctl --since '2026-08-16' -u myservice@1.service | wc -l 218661 real 0m11.914s user 0m10.729s sys 0m0.798s user@machine:~$ time journalctl --since '2026-08-16' -u myservice@1.service -g query | wc -l 4804 real 0m4.127s user 0m3.727s sys 0m0.186s Probably more painful on the day to day is how `journalctl -fu myservice` seems to stall for on the order of 5 to 10s, sometimes. I can't reproduce at the moment and maybe it only happens on some machines, but if you were interested in 'real-world anecdata' it's something to maybe note.

otterley 2mo ago

What do the metrics look like in practice? seems is a bit too handwavy. I get that impressions matter, but data is actionable.

bombela 2mo ago
I have been complaining about journald abysmal performances for almost as long as I can remember. Here is my latest documented benchmark from 2023, which was slightly better than the one I ran in 2020. $ time journalctl > /tmp/all.log real 1m11.364s user 0m52.299s sys 0m6.540s $ time wc -l /tmp/all.log 3659597 /tmp/all.log real 0m0.152s user 0m0.056s sys 0m0.096s $ time journalctl | grep sshd | wc -l 12944 real 0m53.973s user 0m49.535s sys 0m5.210s $ time grep sshd /tmp/all.log | wc -l 12944 real 0m0.429s user 0m0.332s sys 0m0.100s https://github.com/systemd/systemd/issues/2460#issuecomment-1577671723 https://github.com/systemd/systemd/issues/2460#issuecomment-...
- marginalia_nu 2mo ago
  I'm no fan of journald, but I have some methodological issues with this test. The reads from /tmp/all.log are almost certainly cached since you just wrote the file, and will basically boil down to a memcpy call, rather than actual disk I/O. Speed difference isn't as big as you would think on a modern SSD, but it isn't nothing either. Running this between calls should flush the changes to disk and then drop the page cache, making for a fairer test. $ sudo sync $ echo 3 | sudo tee /proc/sys/vm/drop_caches
  - soneil 2mo ago
    I tried similar. Each command looped 5 times, ignore the best and worst times, and average the mean three. journalctl >/tmp/all.log 0m59.763s journalctl | grep -c sshd 0m57.767s (otherwise >all is the only example encumbered by write speeds) wc -l /tmp/all.log 0m0.336s grep -c sshd /tmp/all.log 0m1.776s
    - otterley 2mo ago
      You didn't use the query feature of journalctl, so you basically had it do an entire table conversion. Of course that's not going to show its performance value. This is what you should be running for a proper comparison: 1. `echo 3 | sudo tee /proc/sys/vm/drop_caches` 2. `time journalctl -u ssh.service >/dev/null` 3. `journalctl >/tmp/all.log` 4. `echo 3 | sudo tee /proc/sys/vm/drop_caches` 5. `time grep -q sshd /tmp/all.log`
VGHN7XDuOXPAzol 2mo ago
Sure, I'll try my best to put some concrete numbers on it: On a file of ~1M lines (350 MB), ripgrep returns a query matching 22k lines in 1.9s. (Separately: We have gzip log rotation for old logs. A log with 1.9M lines (106 MB gzipped) on disk can be searched from cold in 1.3s by ripgrep, whilst still being ergonomic. Maybe a different compressor is better?) On a service unit filter over ~218k lines, `journalctl -u service --since <...> -g query` returns 4.8k matching lines in 4.127s (and this is after having warmed the cache by returning the unfiltered query cold, in 11.9s. I don't have root to clear the disk cache I should have ran the filtered query first but ah well, since the difference is already so large it doesn't matter that the filtered journalctl query gets an unfair advantage) Comparing input rates: ripgrep (plain): 576k line/sec ripgrep (gz): 1.4M line/sec journalctl (-u, --since, -g): 53k line/sec I don't have a great deal of understanding of journald's internals, so perhaps there is some variable here that is unreasonably unfavourable to journald. Transcript user@machine:~/log$ time rg query aservice.log | wc -l 22628 real 0m1.859s user 0m0.109s sys 0m0.909s user@machine:~/log$ wc -l aservice.log 1071606 aservice4.log user@machine:~$ time journalctl --since '2026-08-16' -u myservice@1.service | wc -l 218661 real 0m11.914s user 0m10.729s sys 0m0.798s user@machine:~$ time journalctl --since '2026-08-16' -u myservice@1.service -g query | wc -l 4804 real 0m4.127s user 0m3.727s sys 0m0.186s Probably more painful on the day to day is how `journalctl -fu myservice` seems to stall for on the order of 5 to 10s, sometimes. I can't reproduce at the moment and maybe it only happens on some machines, but if you were interested in 'real-world anecdata' it's something to maybe note.