6 ms·

Completely agree with this. One of the biggest risks is using journald if you ask me. It makes logs fairly impenetrable if you have a forensic situation to deal

Completely agree with this. One of the biggest risks is using journald if you ask me. It makes logs fairly impenetrable if you have a forensic situation to deal with. Now there did need to be a better solution in the init system space but systemd was possibly the worst answer that could be mustered for it. Systemd is merely what you get when Redhat owns the development community, not the best technical solution.

sliken 6y ago
Heh, it's not that Redhat owned the devel community, it's that several redhat engineers joined the debian technical board and used their bylaws to force a quick vote.
- deleted 6y ago
  [deleted]
- the_why_of_y 6y ago
  You are entitled to your own opinions, but not to your own facts.
  - turminal 6y ago
    Can you provide some links about that? Or an explanation? I am genuinely interested.
    - the_why_of_y 6y ago
      This was the 2014 vote of the Debian Technical Committee: https://lwn.net/Articles/585504/ https://lwn.net/Articles/585504/ Of the 8 voting members, at the time Steve Langasek and Colin Watson were currently employed and Ian Jackson had been previously employed by Canonical. None of the 8 voting members were employed by Red Hat at the time. Here's a comment with more research into affiliations at the time: https://lwn.net/Articles/585026/ https://lwn.net/Articles/585026/
      - turminal 6y ago
        Thanks!
viraptor 6y ago
> It makes logs fairly impenetrable if you have a forensic situation to deal with. You mean with its sealing which can detect corruption, common export formats, the on-disk format clearly documented (https://www.freedesktop.org/wiki/Software/systemd/journal-files/ https://www.freedesktop.org/wiki/Software/systemd/journal-fi...), and independent readers like https://fluentbit.io/documentation/0.14/input/systemd.html https://fluentbit.io/documentation/0.14/input/systemd.html ? What exactly is impenetrable here?
- uncledave 6y ago
  Well it’s more what you do with a node that has been isolated or is suspect. Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. As for forwarding, there is likely a disparity between the last log written to disk and the last log received by the aggregator. What happened in that window is always interesting and always on disk on the node, unless the attacker has nuked it, which neatly leads to... Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. Learned that from years of windows NT. you can possibly recover them from analysing disk blocks though which is even more difficult if you have journald in there again. I prefer rsyslog, plain text and time marking. Much easier to alert on suspicious things when the mark stops arriving. Log journaling has few practical benefits. I. Fact I’m sceptical of any from experience.
  - viraptor 6y ago
    > Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. What's hard about: journalctl -D /mounted/path/var/lib/journald ...usual options > Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. But that's the point - as long as you've been monitoring the signatures, you know which one's been removed/changed.
    - uncledave 6y ago
      That’s fine until it doesn’t work or the journal is corrupt or you’re pulling the journal off the file system as mentioned further down the post I made. I’ve been there and the entire toolset fell to pieces in front of me. Plain text is orders of magnitude easier and safer to deal with and easily recoverable even if only partially which is extremely difficult with journald
reddotX 6y ago
there was Upstart

sliken 6y ago

Heh, it's not that Redhat owned the devel community, it's that several redhat engineers joined the debian technical board and used their bylaws to force a quick vote.

deleted 6y ago
[deleted]
the_why_of_y 6y ago
You are entitled to your own opinions, but not to your own facts.
- turminal 6y ago
  Can you provide some links about that? Or an explanation? I am genuinely interested.
  - the_why_of_y 6y ago
    This was the 2014 vote of the Debian Technical Committee: https://lwn.net/Articles/585504/ https://lwn.net/Articles/585504/ Of the 8 voting members, at the time Steve Langasek and Colin Watson were currently employed and Ian Jackson had been previously employed by Canonical. None of the 8 voting members were employed by Red Hat at the time. Here's a comment with more research into affiliations at the time: https://lwn.net/Articles/585026/ https://lwn.net/Articles/585026/
    - turminal 6y ago
      Thanks!

viraptor 6y ago

> It makes logs fairly impenetrable if you have a forensic situation to deal with. You mean with its sealing which can detect corruption, common export formats, the on-disk format clearly documented (https://www.freedesktop.org/wiki/Software/systemd/journal-files/ https://www.freedesktop.org/wiki/Software/systemd/journal-fi...), and independent readers like https://fluentbit.io/documentation/0.14/input/systemd.html https://fluentbit.io/documentation/0.14/input/systemd.html ? What exactly is impenetrable here?

uncledave 6y ago
Well it’s more what you do with a node that has been isolated or is suspect. Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. As for forwarding, there is likely a disparity between the last log written to disk and the last log received by the aggregator. What happened in that window is always interesting and always on disk on the node, unless the attacker has nuked it, which neatly leads to... Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. Learned that from years of windows NT. you can possibly recover them from analysing disk blocks though which is even more difficult if you have journald in there again. I prefer rsyslog, plain text and time marking. Much easier to alert on suspicious things when the mark stops arriving. Log journaling has few practical benefits. I. Fact I’m sceptical of any from experience.
- viraptor 6y ago
  > Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. What's hard about: journalctl -D /mounted/path/var/lib/journald ...usual options > Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. But that's the point - as long as you've been monitoring the signatures, you know which one's been removed/changed.
  - uncledave 6y ago
    That’s fine until it doesn’t work or the journal is corrupt or you’re pulling the journal off the file system as mentioned further down the post I made. I’ve been there and the entire toolset fell to pieces in front of me. Plain text is orders of magnitude easier and safer to deal with and easily recoverable even if only partially which is extremely difficult with journald

reddotX 6y ago

there was Upstart