3 ms·

Well it’s more what you do with a node that has been isolated or is suspect. Mounting the file system on a scratch node for analysis and then looking at the log

Well it’s more what you do with a node that has been isolated or is suspect. Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. As for forwarding, there is likely a disparity between the last log written to disk and the last log received by the aggregator. What happened in that window is always interesting and always on disk on the node, unless the attacker has nuked it, which neatly leads to... Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. Learned that from years of windows NT. you can possibly recover them from analysing disk blocks though which is even more difficult if you have journald in there again. I prefer rsyslog, plain text and time marking. Much easier to alert on suspicious things when the mark stops arriving. Log journaling has few practical benefits. I. Fact I’m sceptical of any from experience.

viraptor 6y ago
> Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. What's hard about: journalctl -D /mounted/path/var/lib/journald ...usual options > Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. But that's the point - as long as you've been monitoring the signatures, you know which one's been removed/changed.
- uncledave 6y ago
  That’s fine until it doesn’t work or the journal is corrupt or you’re pulling the journal off the file system as mentioned further down the post I made. I’ve been there and the entire toolset fell to pieces in front of me. Plain text is orders of magnitude easier and safer to deal with and easily recoverable even if only partially which is extremely difficult with journald
  - Spivak 6y ago
    I mean the journald format isn’t magic or anything and 99% of what it stores is plain-text. The only issue with “corruption” is that the tooling is bad at handling it. Which is a valid complaint. But in an alternative universe people would be complaining about plain-text because less crashed on any utf decoding issue. Basically every journald using distro immediately forwards the logs to syslog so you can just pretend it doesn’t exist and call it a day. I don’t because journald’s metadata and filtering is super powerful but nothing is stopping you from just grepping like usual.

viraptor 6y ago

> Mounting the file system on a scratch node for analysis and then looking at the logs is a pain in the ass when you have journald in the way. What's hard about: journalctl -D /mounted/path/var/lib/journald ...usual options > Sealing and tamper protection on logs is absolutely no use as any competent attacker will just destroy them outright. But that's the point - as long as you've been monitoring the signatures, you know which one's been removed/changed.

uncledave 6y ago
That’s fine until it doesn’t work or the journal is corrupt or you’re pulling the journal off the file system as mentioned further down the post I made. I’ve been there and the entire toolset fell to pieces in front of me. Plain text is orders of magnitude easier and safer to deal with and easily recoverable even if only partially which is extremely difficult with journald
- Spivak 6y ago
  I mean the journald format isn’t magic or anything and 99% of what it stores is plain-text. The only issue with “corruption” is that the tooling is bad at handling it. Which is a valid complaint. But in an alternative universe people would be complaining about plain-text because less crashed on any utf decoding issue. Basically every journald using distro immediately forwards the logs to syslog so you can just pretend it doesn’t exist and call it a day. I don’t because journald’s metadata and filtering is super powerful but nothing is stopping you from just grepping like usual.