4 ms·

Why not just have a SQLite file and call it a day? Also, why mmaped file?

pengaru 2mo ago
I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.
- marginalia_nu 2mo ago
  Well there was an ambition, apparently. > Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance > Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
- otterley 2mo ago
  The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-January/004188.html https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
  - pengaru 2mo ago
    FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
    - p_l 2mo ago
      ... Sounds like a signature on a patch that triggers an epic Linus rant on LKML[1] [1] Happened few times, I think RedHat as a whole even got banned from sending changes for a short while
  - giov4 2mo ago
    I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment-777276876 https://github.com/systemd/systemd/issues/15292#issuecomment... It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
    - brohee 2mo ago
      Ah, the Ulrich Drepper school of dealing with reported issues. Time for esystemd ;)
      - deleted 2mo ago
        [deleted]
    - pineapplepizza6 2mo ago
      systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?
      - otterley 2mo ago
        Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.
        redsocksfan45 2mo ago
        [dead]
quotemstr 2mo ago
SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.
- otterley 2mo ago
  Neither DuckDB nor LevelDB existed when journald was created. Not to say it couldn't be done today, but just some historical context.
  - actionfromafar 2mo ago
    LevelDB was released in 2011, so it existed but was very new.
- ElectricalUnion 2mo ago
  No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
  - quotemstr 2mo ago
    Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.) We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
    - dchest 2mo ago
      Write-Ahead Log for... logs? WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread. I believe journald log rotation is basically: close file - open a new one. How is it not completely different?
dmitrygr 2mo ago
Because Poettering didn’t invent SQLite.

I'm not the architect of journald and wasn't really around when these decisions were made, so I can't really speak authoritatively on that particular topic. There was mailing list discussion at the time journald was conceived though, you can find it if you look. https://0pointer.de/blog/projects/the-journal.html https://0pointer.de/blog/projects/the-journal.html might be a good entry-point.

marginalia_nu 2mo ago
Well there was an ambition, apparently. > Performance: journal operations for appending and browsing should be fast in terms of complexity. O(log n) or better is highly advisable, in order to provide for organization-wide log monitoring with good performance > Minimal Footprint: journal data files should be small in disk size, especially in the light that the amount of data generated might be substantially bigger than on classic syslog.
otterley 2mo ago
The mailing list archives are here: https://lists.freedesktop.org/archives/systemd-devel/ https://lists.freedesktop.org/archives/systemd-devel/ It doesn't look like there was an open design review; Lennart Poettering just dropped it in in v38. https://lists.freedesktop.org/archives/systemd-devel/2012-January/004188.html https://lists.freedesktop.org/archives/systemd-devel/2012-Ja...
- pengaru 2mo ago
  FWIW the journal file signature is "LPKSHHRH" for Lennart, Kay Sievers, Harald Hoyer, Red Hat... I presumed it was at least Lennart, Kay, and Harald who collaborated on the design.
  - p_l 2mo ago
    ... Sounds like a signature on a patch that triggers an epic Linus rant on LKML[1] [1] Happened few times, I think RedHat as a whole even got banned from sending changes for a short while
- giov4 2mo ago
  I think this also explains a lot and should not be ignored. https://github.com/systemd/systemd/issues/15292#issuecomment-777276876 https://github.com/systemd/systemd/issues/15292#issuecomment... It seems a recurring (handling) issue but unfortunately it affects multiple linux distro defaults. This is the worse that can collaboratively happen for FOSS in general imho.
  - brohee 2mo ago
    Ah, the Ulrich Drepper school of dealing with reported issues. Time for esystemd ;)
    - deleted 2mo ago
      [deleted]
  - pineapplepizza6 2mo ago
    systemd is not a collaborative project. It is Lennart's personal cathedral project and you can take it or leave it. That's fine for Lennart, the question is if it's so bad then why are the rest of us taking it instead of leaving it?
    - otterley 2mo ago
      Probably because the overall impact is not as bad as extremely vocal people on GitHub and HN would have you believe, and more people like systemd than dislike it.
      - redsocksfan45 2mo ago
        [dead]

quotemstr 2mo ago

SQLite here is okay, but DuckDB or LevelDB would be better. Either way, no need to invent a new storage format.

otterley 2mo ago
Neither DuckDB nor LevelDB existed when journald was created. Not to say it couldn't be done today, but just some historical context.
- actionfromafar 2mo ago
  LevelDB was released in 2011, so it existed but was very new.
ElectricalUnion 2mo ago
No duckdb (or parquet). If you want to avoid writes and write amplification, you really want to avoid re-writing all 122880 rows of a row group every time a single insert happens.
- quotemstr 2mo ago
  Uh, who said anything about writing 122880 rows every time you do a single insert into DuckDB? There's a WAL. Consolidation happens in big chunks. (And it's not like journald log rotation is somehow better than WAL consolidation.) We shouldn't be making momentus choices of data format based on vague and incorrect understandings of data formats.
  - dchest 2mo ago
    Write-Ahead Log for... logs? WAL means it will write the same data at least twice. Similar issue, but even worse, with LevelDB -- it will just delay the inevitable huge rewrites for later. Funny to hear those proposals in the write amplification thread. I believe journald log rotation is basically: close file - open a new one. How is it not completely different?

dmitrygr 2mo ago

Because Poettering didn’t invent SQLite.