7 ms·

Systemd Forward Secure Sealing of System Logs Makes Little Sense

39 points by louwrentius 12y ago

erhardm 12y ago
Systemd is starting to show the signs of feature creep which IMHO is good. Failing early means the ship can be turned around if there's enough will, or at least (newer) replacements of systemd(in case it goes really off the rails) will learn from others' experience.
- UnoriginalGuy 12y ago
  This was added in 2012.
  - erhardm 12y ago
    You mean the article or the feature? The article is dated by 22/11/2014. The feature may have been added a few months/years ago, but the point still stands about systemd showing signs of feature creep. Note also the DNS resolver issue[0] [0] - http://seclists.org/oss-sec/2014/q4/592 http://seclists.org/oss-sec/2014/q4/592
    - the_mitsuhiko 12y ago
      The feature was added in 2012 or something, whenever the log support came. The DNS issue affects not just systemd but any stub resolver.
      - pdkl95 12y ago
        The DNS issue is worth in systemd's resolver because it is NOT merely a "stub resolver" - it implements a cache. From the same thread: "The DNS specification does not require rewriting of upstream responses to filter out parts for which the queried server is not authoritative. This means that a downstream caching resolver will tend to poison its cache if it adds data from such responses that are not directly in response to the QNAME." http://seclists.org/oss-sec/2014/q4/602 http://seclists.org/oss-sec/2014/q4/602 Adding entries (including additional records, which are common in DNS) without validation is just asking for a DNS poisoning attack.
- ajross 12y ago
  Won't happen. It's turning ugly (really the ship sailed on that with the container stuff), but it's never going to "go off the rails". It's quality software and it works. I... guess I'm sort of in the "anti-systemd" crowd, but in the mildest sense. For the overwhelming majority of use cases, it really just isn't appreciably better. It's better in lots of measurable, small, and ultimately more or less irrelevant ways. Runtime system software maintenance just wasn't a problem worth solving, IMHO. But... it's quality software with solid features and it works. The same logic says we should just shut up and take it. So I do.
- GFK_of_xmaspast 12y ago
  I don't have any dogs in the systemd fight, but it looks like you're moving from 'anger' to 'bargaining' in Kuebler-Ross.
teddyh 12y ago
This starts off looking like a reasonable criticism of systemd, or rather that systemd claims to do more that it (or any alternative) actually can guarantee. It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant). But then it veers off: “This whole observation is not original, in the comments of the systemd author's blogpost, the same argument is made by Andrew Wyatt (two years earlier). The response from the systemd author was to block him.” No links are provided, so we can’t know what any of them actually said or did. This smells like deliberate character assassination. Especially the fact that no references are provided seems fishy to me.
- UnoriginalGuy 12y ago
  > It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant). I'm critical of the article also, but they in fact did. If you click the "Forward Secure Sealing (FSS)" it takes you to a G+ page with the quote on it from Lennart Poettering on Aug 20, 2012.
  - teddyh 12y ago
    It was not stated that that was the source of the quote, and the link was not given anywhere near the quote, but I guess you are right that there was a link… So this quote is from Aug 20, 2012. Has Lennart’s opinion of this changed during the more than two years that have passed since? Does the documentation of systemd make this guarantee? Did the documentation ever make the claim that FSS is an complete replacement for remote logging? Now that I think about it, and have the context of the quote, I think that I (and the blog post author) maybe misunderstood Lennart. FSS is not a complete replacement for remote logging, but: Among other things, remote logging used to be the only way to get immutable logs. If this was your only reason for using remote logging, then FSS is a viable replacement. EDIT: Now that I can actually read what Andrew Wyatt wrote and how he wrote it, I can see why he was blocked. He was condescending and vague on details, and (according to Lennart) Andrew had a habit of doing that. The comments to the blog post actually cover what I wrote in my previous paragraph, so there is no excuse to dig this up now, two years later, and deliberately misunderstand it in order to gain points with the systemd haters. This is FUD.
UnoriginalGuy 12y ago
This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user). Also the author picks apart a quote, but if the assumptions in the previous paragraph is taken at face value (that an attack can have sup-10 second root with automated attacks on the logs), then even remote logging isn't a secure vector as they often aren't sent instantly so an attacker could intercept or alter them. However I suggest that such a scenario is uncommon. Most attackers have to start outside and fight their way in which will generate plenty of logs for either external logging or FSS. > It seems to me that systemd now also wants to play the role as some crude intrusion detection system. FSS isn't a "intrusion detection system." It is an attempt to make logs immutable, nothing more. The nice thing about FSS is that it can be coupled with an actual IDS and the IDS can look for attempts at altering the FSS. > If the systemd author would just have implemented sending log events to an external log server, that would have been way more useful security-wise, I think. It isn't a binary choice. They can implement FSS and remote logging. Also it is open source so you yourself can contribute external logging.
- tw04 12y ago
  >This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user). You mean like a disgruntled system administrator? There are countless scenarios in which an attacker goes from zero access to root within 10 seconds. Predicating your security on that not being a common scenario shows that you've spent next to no time in a real corporate environment. IE: Poettering.
  - matthewmacleod 12y ago
    Nonsense – defence in depth and all of that. FSS is useful because it provides a layer of audibility for users that wouldn't have external log servers in any case, and because it provides some minimal guarantees about the ability to tamper with logs. That's useful, regardless of the fact that it's not a bulletproof solution!
vezzy-fnord 12y ago
An interesting thing to note is that systemd's FSS feature is based on a FSPRG [1] (forward-secure pseudorandom number generator) contributed by Lennart's brother Bertram Poettering, based on the latter's research paper entitled Practical Secure Logging: Seekable Sequential Key Generators [2]. [1] http://cgit.freedesktop.org/systemd/systemd/tree/src/journal/fsprg.c http://cgit.freedesktop.org/systemd/systemd/tree/src/journal... [2] https://eprint.iacr.org/2013/397.pdf https://eprint.iacr.org/2013/397.pdf
- angersock 12y ago
  Oh, goody, so it's homebrew crypto as well?
jimrandomh 12y ago
This is a neat feature, which I didn't know systemd had, and which I recall having wanted in other contexts. The criticism about the 15-minute interval is valid, but that's an implementation detail which, from the sound of it, can be fixed if someone puts in the effort. So while the author may have intended this as a pure criticism, it didn't really come off that way.
the_mitsuhiko 12y ago
So the argument is: it's not good enough, so we need to throw it away?
- click170 12y ago
  No, I think the author's point was that people shouldn't misrepresent FSS as equivalent to logging to a remote server. They can compliment each other, but they are not the same, and thinking they are can lead to serious security snafus.
  - louwrentius 12y ago
    Exactly.
deleted 12y ago
[deleted]
KaiserPro 12y ago
Log signing is nice to have, but ultimately not that reliable. Systemd has quite a broad attack surface, and if the attacker has root, you're fucked anyway. It might be a useful tool for poking local sysadmins. compromising systemd, forcing it to re-sign the logs (assuming that practicable.) Potentially you can have an alert that checks the signature, which might be useful. However you can do something similar now, but comparing log files (local vs remote) although its not that easy or elegant. I've yet to use systemd in anger. I'm sure when we update all 25k machines to it next year we'll become intimately familiar with all its pains.
- Tuna-Fish 12y ago
  > compromising systemd, forcing it to re-sign the logs (assuming that practicable.) This is completely unfeasible. The point of FSS is that you sign the logs, create a new key using your old one, and then permanently throw away the old key. Once this is done, no matter what control you have on the system, you cannot re-sign the logs without the tampering being obvious.
steventhedev 12y ago
I think the systemd complaints really boil down to two very real problems: 1. The authors of systemd aren't dealing with criticism well. They've allowed it to turn into a PR nightmare. Agree or disagree with them on specific issues, but they've allowed this to grow out of hand. There are nearly daily popular posts on HN, Reddit, etc. Most of these are bikeshed arguments, but if you have so many bikesheds that it becomes a daily issue on news aggregators, then you have yourself a culture problem. 2. Perhaps as a result of the above, there haven't been any limitations of the scope of systemd. This means that there is a lot of feature creep as stuff gets added in, and criticism gets swept away without considering it. Unfortunately, I don't see an easy way to fix the culture problem. But it needs to be addressed. EDIT: formatting
- click170 12y ago
  Posts like yours are one of the first steps. At this point we need to address the culture problem and the people contributing to it instead of focusing on the distracting fallout. I've tried to keep an open mind about all things systemd but after finally reading some of the exchanges people have with the systemd folks, I'm surprised systemd isn't more widely rejected as a viable alternative simply because of the attitude of some of the people currently maintaining it. You can't be an butthole to folks trying to offer valid criticisms and expect those same people take your project seriously. That just does not inspire confidence, it inspires skepticism and distrust. I wonder if it's simply a critical mass problem, if once enough devs get peeved they will fork systemd and lead it the way they see fit sans any caustic contributions or contributors.
  - Cyther606 12y ago
    It has been forked [1]. > uselessd (the useless daemon, or the daemon that uses less... depending on your viewpoint) is a project to reduce systemd to a base initd, process supervisor and transactional dependency system, while minimizing intrusiveness and isolationism. Basically, it’s systemd with the superfluous stuff cut out, a (relatively) coherent idea of what it wants to be, support for non-glibc platforms and an approach that aims to minimize complicated design He could use better marketing, but I think removing the kitchen sink aspect of systemd and keeping the service file syntax is a step in the right direction. [1] http://uselessd.darknedgy.net http://uselessd.darknedgy.net
ghshephard 12y ago
No security professional will be bothered to critique "FSS" because anybody worth their salt know that any log ON a server is, by definition, not trustworthy. They've known this for at least 20 years (if not longer), and everyone uses syslogd or some variant thereof. So, in that sense, there is no controversy here, and this feature is, for the most part, completely useless. In the comments below, Andrew Wyatt is absolutely correct and this, "Lennart Poettering" individual doesn't know what they are talking about.
xorcist 12y ago
What I find troublesome with journald is that they ignored everything that was done before in the log signing space. There has been some research done, even an RFC published. They implemented their own solution incompatible with everything else. Any questions about their design was met with, at best, indifference. Since then rsyslog gained a log signing solution, sadly incompatible, but which at least to my eyes look a bit more conservative in design. We don't really have any second opinions on journald. That would be fine if it was just a hobby project, but I find it problematic to include by default in stable enterprise distributions.

39 points by louwrentius 12y ago

Systemd is starting to show the signs of feature creep which IMHO is good. Failing early means the ship can be turned around if there's enough will, or at least (newer) replacements of systemd(in case it goes really off the rails) will learn from others' experience.

UnoriginalGuy 12y ago
This was added in 2012.
- erhardm 12y ago
  You mean the article or the feature? The article is dated by 22/11/2014. The feature may have been added a few months/years ago, but the point still stands about systemd showing signs of feature creep. Note also the DNS resolver issue[0] [0] - http://seclists.org/oss-sec/2014/q4/592 http://seclists.org/oss-sec/2014/q4/592
  - the_mitsuhiko 12y ago
    The feature was added in 2012 or something, whenever the log support came. The DNS issue affects not just systemd but any stub resolver.
    - pdkl95 12y ago
      The DNS issue is worth in systemd's resolver because it is NOT merely a "stub resolver" - it implements a cache. From the same thread: "The DNS specification does not require rewriting of upstream responses to filter out parts for which the queried server is not authoritative. This means that a downstream caching resolver will tend to poison its cache if it adds data from such responses that are not directly in response to the QNAME." http://seclists.org/oss-sec/2014/q4/602 http://seclists.org/oss-sec/2014/q4/602 Adding entries (including additional records, which are common in DNS) without validation is just asking for a DNS poisoning attack.
ajross 12y ago
Won't happen. It's turning ugly (really the ship sailed on that with the container stuff), but it's never going to "go off the rails". It's quality software and it works. I... guess I'm sort of in the "anti-systemd" crowd, but in the mildest sense. For the overwhelming majority of use cases, it really just isn't appreciably better. It's better in lots of measurable, small, and ultimately more or less irrelevant ways. Runtime system software maintenance just wasn't a problem worth solving, IMHO. But... it's quality software with solid features and it works. The same logic says we should just shut up and take it. So I do.
GFK_of_xmaspast 12y ago
I don't have any dogs in the systemd fight, but it looks like you're moving from 'anger' to 'bargaining' in Kuebler-Ross.

teddyh 12y ago

This starts off looking like a reasonable criticism of systemd, or rather that systemd claims to do more that it (or any alternative) actually can guarantee. It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant). But then it veers off: “This whole observation is not original, in the comments of the systemd author's blogpost, the same argument is made by Andrew Wyatt (two years earlier). The response from the systemd author was to block him.” No links are provided, so we can’t know what any of them actually said or did. This smells like deliberate character assassination. Especially the fact that no references are provided seems fishy to me.

UnoriginalGuy 12y ago
> It quotes the author of systemd to prove that the claim is made, but it does not give a link or say from where the quote was taken (so we don’t know the context or age of it, i.e. we don’t know it is still relevant). I'm critical of the article also, but they in fact did. If you click the "Forward Secure Sealing (FSS)" it takes you to a G+ page with the quote on it from Lennart Poettering on Aug 20, 2012.
- teddyh 12y ago
  It was not stated that that was the source of the quote, and the link was not given anywhere near the quote, but I guess you are right that there was a link… So this quote is from Aug 20, 2012. Has Lennart’s opinion of this changed during the more than two years that have passed since? Does the documentation of systemd make this guarantee? Did the documentation ever make the claim that FSS is an complete replacement for remote logging? Now that I think about it, and have the context of the quote, I think that I (and the blog post author) maybe misunderstood Lennart. FSS is not a complete replacement for remote logging, but: Among other things, remote logging used to be the only way to get immutable logs. If this was your only reason for using remote logging, then FSS is a viable replacement. EDIT: Now that I can actually read what Andrew Wyatt wrote and how he wrote it, I can see why he was blocked. He was condescending and vague on details, and (according to Lennart) Andrew had a habit of doing that. The comments to the blog post actually cover what I wrote in my previous paragraph, so there is no excuse to dig this up now, two years later, and deliberately misunderstand it in order to gain points with the systemd haters. This is FUD.

UnoriginalGuy 12y ago

This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user). Also the author picks apart a quote, but if the assumptions in the previous paragraph is taken at face value (that an attack can have sup-10 second root with automated attacks on the logs), then even remote logging isn't a secure vector as they often aren't sent instantly so an attacker could intercept or alter them. However I suggest that such a scenario is uncommon. Most attackers have to start outside and fight their way in which will generate plenty of logs for either external logging or FSS. > It seems to me that systemd now also wants to play the role as some crude intrusion detection system. FSS isn't a "intrusion detection system." It is an attempt to make logs immutable, nothing more. The nice thing about FSS is that it can be coupled with an actual IDS and the IDS can look for attempts at altering the FSS. > If the systemd author would just have implemented sending log events to an external log server, that would have been way more useful security-wise, I think. It isn't a binary choice. They can implement FSS and remote logging. Also it is open source so you yourself can contribute external logging.

tw04 12y ago
>This article is predicated on the assumption that an attacker goes from zero access to full root within 10 seconds. While there are vectors which have allowed such rapid escalation, they're less common now than historically thanks to Linux's security looking more like an onion rather than just a binary state (root or user). You mean like a disgruntled system administrator? There are countless scenarios in which an attacker goes from zero access to root within 10 seconds. Predicating your security on that not being a common scenario shows that you've spent next to no time in a real corporate environment. IE: Poettering.
- matthewmacleod 12y ago
  Nonsense – defence in depth and all of that. FSS is useful because it provides a layer of audibility for users that wouldn't have external log servers in any case, and because it provides some minimal guarantees about the ability to tamper with logs. That's useful, regardless of the fact that it's not a bulletproof solution!

vezzy-fnord 12y ago

An interesting thing to note is that systemd's FSS feature is based on a FSPRG [1] (forward-secure pseudorandom number generator) contributed by Lennart's brother Bertram Poettering, based on the latter's research paper entitled Practical Secure Logging: Seekable Sequential Key Generators [2]. [1] http://cgit.freedesktop.org/systemd/systemd/tree/src/journal/fsprg.c http://cgit.freedesktop.org/systemd/systemd/tree/src/journal... [2] https://eprint.iacr.org/2013/397.pdf https://eprint.iacr.org/2013/397.pdf

angersock 12y ago
Oh, goody, so it's homebrew crypto as well?

jimrandomh 12y ago

This is a neat feature, which I didn't know systemd had, and which I recall having wanted in other contexts. The criticism about the 15-minute interval is valid, but that's an implementation detail which, from the sound of it, can be fixed if someone puts in the effort. So while the author may have intended this as a pure criticism, it didn't really come off that way.

the_mitsuhiko 12y ago

So the argument is: it's not good enough, so we need to throw it away?

click170 12y ago
No, I think the author's point was that people shouldn't misrepresent FSS as equivalent to logging to a remote server. They can compliment each other, but they are not the same, and thinking they are can lead to serious security snafus.
- louwrentius 12y ago
  Exactly.

deleted 12y ago

[deleted]

KaiserPro 12y ago

Log signing is nice to have, but ultimately not that reliable. Systemd has quite a broad attack surface, and if the attacker has root, you're fucked anyway. It might be a useful tool for poking local sysadmins. compromising systemd, forcing it to re-sign the logs (assuming that practicable.) Potentially you can have an alert that checks the signature, which might be useful. However you can do something similar now, but comparing log files (local vs remote) although its not that easy or elegant. I've yet to use systemd in anger. I'm sure when we update all 25k machines to it next year we'll become intimately familiar with all its pains.

Tuna-Fish 12y ago
> compromising systemd, forcing it to re-sign the logs (assuming that practicable.) This is completely unfeasible. The point of FSS is that you sign the logs, create a new key using your old one, and then permanently throw away the old key. Once this is done, no matter what control you have on the system, you cannot re-sign the logs without the tampering being obvious.

steventhedev 12y ago

I think the systemd complaints really boil down to two very real problems: 1. The authors of systemd aren't dealing with criticism well. They've allowed it to turn into a PR nightmare. Agree or disagree with them on specific issues, but they've allowed this to grow out of hand. There are nearly daily popular posts on HN, Reddit, etc. Most of these are bikeshed arguments, but if you have so many bikesheds that it becomes a daily issue on news aggregators, then you have yourself a culture problem. 2. Perhaps as a result of the above, there haven't been any limitations of the scope of systemd. This means that there is a lot of feature creep as stuff gets added in, and criticism gets swept away without considering it. Unfortunately, I don't see an easy way to fix the culture problem. But it needs to be addressed. EDIT: formatting

click170 12y ago
Posts like yours are one of the first steps. At this point we need to address the culture problem and the people contributing to it instead of focusing on the distracting fallout. I've tried to keep an open mind about all things systemd but after finally reading some of the exchanges people have with the systemd folks, I'm surprised systemd isn't more widely rejected as a viable alternative simply because of the attitude of some of the people currently maintaining it. You can't be an butthole to folks trying to offer valid criticisms and expect those same people take your project seriously. That just does not inspire confidence, it inspires skepticism and distrust. I wonder if it's simply a critical mass problem, if once enough devs get peeved they will fork systemd and lead it the way they see fit sans any caustic contributions or contributors.
- Cyther606 12y ago
  It has been forked [1]. > uselessd (the useless daemon, or the daemon that uses less... depending on your viewpoint) is a project to reduce systemd to a base initd, process supervisor and transactional dependency system, while minimizing intrusiveness and isolationism. Basically, it’s systemd with the superfluous stuff cut out, a (relatively) coherent idea of what it wants to be, support for non-glibc platforms and an approach that aims to minimize complicated design He could use better marketing, but I think removing the kitchen sink aspect of systemd and keeping the service file syntax is a step in the right direction. [1] http://uselessd.darknedgy.net http://uselessd.darknedgy.net

ghshephard 12y ago

No security professional will be bothered to critique "FSS" because anybody worth their salt know that any log ON a server is, by definition, not trustworthy. They've known this for at least 20 years (if not longer), and everyone uses syslogd or some variant thereof. So, in that sense, there is no controversy here, and this feature is, for the most part, completely useless. In the comments below, Andrew Wyatt is absolutely correct and this, "Lennart Poettering" individual doesn't know what they are talking about.

xorcist 12y ago

What I find troublesome with journald is that they ignored everything that was done before in the log signing space. There has been some research done, even an RFC published. They implemented their own solution incompatible with everything else. Any questions about their design was met with, at best, indifference. Since then rsyslog gained a log signing solution, sadly incompatible, but which at least to my eyes look a bit more conservative in design. We don't really have any second opinions on journald. That would be fine if it was just a hobby project, but I find it problematic to include by default in stable enterprise distributions.