4 ms·
> It makes logs fairly impenetrable if you have a forensic situation to deal with. You mean with its sealing which can detect corruption, common export formats
by viraptor 6y ago
> It makes logs fairly impenetrable if you have a forensic situation to deal with.
You mean with its sealing which can detect corruption, common export formats, the on-disk format clearly documented (https://www.freedesktop.org/wiki/Software/systemd/journal-files/ https://www.freedesktop.org/wiki/Software/systemd/journal-fi...), and independent readers like https://fluentbit.io/documentation/0.14/input/systemd.html https://fluentbit.io/documentation/0.14/input/systemd.html ?
What exactly is impenetrable here?
- uncledave 6y agoWell it’s more what you do with a node that has been isolated or is suspect. Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. As for forwarding, there is likely a disparity between the last log written to disk and the last log received by the aggregator. What happened in that window is always interesting and always on disk on the node, unless the attacker has nuked it, which neatly leads to... Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. Learned that from years of windows NT. you can possibly recover them from analysing disk blocks though which is even more difficult if you have journald in there again. I prefer rsyslog, plain text and time marking. Much easier to alert on suspicious things when the mark stops arriving. Log journaling has few practical benefits. I. Fact I’m sceptical of any from experience.
- viraptor 6y ago> Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. What's hard about: journalctl -D /mounted/path/var/lib/journald ...usual options > Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. But that's the point - as long as you've been monitoring the signatures, you know which one's been removed/changed.
- uncledave 6y agoThat’s fine until it doesn’t work or the journal is corrupt or you’re pulling the journal off the file system as mentioned further down the post I made. I’ve been there and the entire toolset fell to pieces in front of me. Plain text is orders of magnitude easier and safer to deal with and easily recoverable even if only partially which is extremely difficult with journald
- Spivak 6y agoI mean the journald format isn’t magic or anything and 99% of what it stores is plain-text. The only issue with “corruption” is that the tooling is bad at handling it. Which is a valid complaint. But in an alternative universe people would be complaining about plain-text because less crashed on any utf decoding issue. Basically every journald using distro immediately forwards the logs to syslog so you can just pretend it doesn’t exist and call it a day. I don’t because journald’s metadata and filtering is super powerful but nothing is stopping you from just grepping like usual.
- deckard1 6y agoit's a bit odd to defend the new kid on the block, journald, and then go and link to fluentbit. I feel like we're missing the elephant in the room, which is that almost everyone is going to forward logs over the network completely bypassing journald/journalctl altogether. At which point you have to ask: who is journald for? In the server environment, you're going to send everything to Splunk or whatever. Which just leaves desktop users to deal with the headache that is journald. Desktop/workstation users don't care about tamper detection, corruption, or any of that nonsense. If there is corruption going on, I doubt they care that their boot logs are invalid. I'm 100% sure they care more that their 3D render that took 30 hours to render is now corrupt, or some other asset or document. And in the rare case they do need to dive into the logs, they aren't going to want to re-learn journalctl again. journald is a solution to a 1990s problem that no longer exists.
- uncledave 6y agoThis is a pretty good description of the problems for sure. Thanks for writing it up.
- viraptor 6y agoI linked to fluentbit only as an independent journal reader, not for what it does as a service. To be clear, I wasn't defending usefulness of sealing and signing. Just the fact that not only it's independently readable, but also you can find out about corruption of it matters to you for forensic purposes raised. Journalctl does provide at least one significant feature though - on a properly configured system "journalctl -u foobar" gives you the logs of foobar. No more chasing which file they live in, no more stdout goes here, stderr there, and logs get split in that special way over there. This is great for desktop users.