27 ms·

System Down: A systemd-journald exploit

385 points by gmueckl 8y ago

ape4 8y ago
Top comment from reddit thread: FWIW distros that use -fstack-clash-protection to compile systemd, including recent Fedora and OpenSUSE, aren't vulnerable. https://www.reddit.com/r/linux/comments/aeac8g/systemd_earns_three_cves_can_be_used_to_gain/ https://www.reddit.com/r/linux/comments/aeac8g/systemd_earns...
- detaro 8y ago
  It says so in the article too.
- nailer 8y ago
  Also in the article: > To the best of our knowledge, all systemd-based Linux distributions are vulnerable, but SUSE Linux Enterprise 15, openSUSE Leap 15.0, and Fedora 28 and 29 are not exploitable because their user space is compiled with GCC's -fstack-clash-protection.
  - krylon 8y ago
    Does that include openSuse Tumbleweed?
- userbinator 8y ago
  I'm of two minds when it comes to mitigations like this. On the one hand, it helps decrease the number of exploits. On the other hand, it encourages/tolerates the proliferation of low-quality code.
  - loeg 8y ago
    Stack clash protection is a necessary implementation detail of the C language. It's a shame it's even optional. C17 § 6.5.16.1 Simple assignment, Semantics, (3): > If the value being stored in an object is read from another object that overlaps in any way the storage of the first object, then the overlap shall be exact and the two objects shall have qualified or unqualified versions of a compatible type; otherwise, the behavior is undefined. Also listed under J.2 "Undefined Behavior," with reference to the same section. The same clause is present in C11 and C99 under the same section.
    - saagarjha 8y ago
      > otherwise, the behavior is undefined I don't see how this necessitates stack protection at all, at least from a standards perspective.
      - loeg 8y ago
        Ok, here's the logic: 0. Your premise is that standard C does not require a runtime to provide any protection from stack memory growing into the heap. 1. Standard C defines automatic object storage and heap object storage. 2. It is implementation defined where heap objects are allocated in memory. As a result, the type and compatibility of arbitrary heap memory locations are unpredictable to developers. 3. It is implementation defined where stack objects are allocated in memory. As a result, the type and compatibility of arbitrary stack memory locations are unpredictable to developers. 4. Standard C says that mutated objects can't overlap [more or less]. 5. Optimizing compilers don't allocate memory for stack objects that aren't at least partially mutated. 6. Standard C defines no way for developers to determine how deep their stack is, or where the top of the heap is. 7. From (2), (3), (5), (6), it is impossible for developers to determine whether or not they would violate (4). 8. As a result of (7) and (0), any use of both heap and stack objects is UB. A C runtime compliant to your definition in (0) could start the heap from 0x100, growing up, and the stack at 0x104, growing down. 8.a. Similarly, any use of static or other lifetime, and stack objects, is UB. 9. It logically follows from 8 that, under your premise (0), it is always UB to the stack and other kinds of data memory. I guess you could make the case that the C committee hate everyone and all stack objects are UB. But my read of the C standard is that the intent is that any use of stack objects is not in fact required to be UB. If stack objects were always UB, I think it would be called out explicitly or not specified in the standard. So yes, I think any standard-conforming C implementation has to prevent stack clash one way or another. That could take the form of an alternate stack memory that is entirely separate from heap memory; it doesn't have to be probing and guard pages.
        saagarjha 8y ago
        > I guess you could make the case that the C committee hate everyone and all stack objects are UB. Honestly, I think that this is the logical conclusion, or at least, every compiler/machine implementation is subtly wrong; the standards glosses over what should happen when the stack is too large, for example, so technically any use of the stack is not guaranteed to be valid because you might just be out of space. The C standard is for a "head-in-the-clouds" abstract machine, so it doesn't concern itself with pedestrian issues like lack of memory.
gmueckl 8y ago
These are three long standing exploitable bugs in systemd's journald which can be used to gain local root privileges. Most systemd based distributions are vulnerable and, from the looks of it, may have been for years.
- nightfly 8y ago
  link?
  - mgalgs 8y ago
    I believe he's giving a synopsis of TFA
deleted 8y ago
[deleted]
insertcredit 8y ago
I refuse to use systemd to this day. It's unbelievably complex and became established through political power play rather than any sort of merit. Which is without a doubt not what I expected to see in the Linux ecosystem.
- isatty 8y ago
  I do too, for personal use (OpenRC on Gentoo user through and through), but due to the adoption of systemd by popular distributions I don’t see an alternative for work. Unfortunately.
  - geggam 8y ago
    I moved all my personal things to OpenBSD due to the Linux community move away from unix standards
    - DyslexicAtheist 8y ago
      seconded. the Linux fanboy community (which I've no doubt been part of until today and still defend bc of the FOSS spirit) is just getting too much. Linux is a victim of its own success. Bloat and inconsistency everywhere and compared to openBSD they rather move fast and break things rather than chose coherent design. Linux Security as it exists today is an oxymoron. Not to mention the community that consists in BSD of people who know what they do and can be trusted with auditing code. Linux code audits are a joke ... they rather hurl abuse at security guys (e.g. "masturbating monkeys") or outright ignore them ... (no time to discuss esthetics and coherent design because it's all about shipping fast at all cost - the same and only way they do in their day jobs ...)
    - smhenderson 8y ago
      That's what I did as well. But I also continue to use Slackware which doesn't have any current plans to move to SystemD. It was the first Linux distro I ever used and I still love it to this day. I've used many different systems for work over the years but I've kept at least one Slackware laptop or desktop going since early 1995 when I first found a Slackware CDROM in the back of a book about learning Linux. Although it gets new software, bug fixes and minor updates here and there it's remarkably similar today to what it was then. And I find it Just Works, I've had very little trouble with it over the years. I think I like it too because it always felt like the most traditional Unix like distro I've ever used and also more BSD than System V which I also generally prefer.
AdmiralAsshat 8y ago
First CVE I've seen to reference a System of a Down song. (Every subsection starts with an SOAD song quote)
- nailer 8y ago
  Off topic, but SOAD were really one of the most enduring metal acts of the early 2000s. If you remember them, Needle Drop recently did a classic review of Toxicity - enjoy the nostalgia! https://www.youtube.com/watch?v=-jI1ofec02A https://www.youtube.com/watch?v=-jI1ofec02A
- rconti 8y ago
  I'm not a SOAD fan, but good to see someone at my former employer still has a sense of humor :)
Twirrim 8y ago
I might be reading that article wrong, but it reads like they're critical of the use of alloca? I've never really written C or the like, what is dangerous about alloca?
- speps 8y ago
  Variable size allocation from the stack... That's one dangerous trick. [1] https://stackoverflow.com/questions/1018853/why-is-the-use-of-alloca-not-considered-good-practice https://stackoverflow.com/questions/1018853/why-is-the-use-o...
- pilif 8y ago
  It allocates on the stack which is inherently of limited size and allocations grow in the direction of the “juicy” stuff. As so often with the C library, there is little bounds checking going on either (in fact, the behavior once you overflow the stack is explicitly undefined, so who knows what the hell is going to happen) If you somehow let the user specify the amount of bytes you’re going to allocate on the stack, then that’s an exploitable issue.
  - Twirrim 8y ago
    Ah, OK. So it is truly horrifying that it's in heavy use in journald, where users inevitably have that sort of power.
- deleted 8y ago
  [deleted]
segfaultbuserr 8y ago
Yet another proof for the following: 1. It's reasonable to claim that amd64 (x86_64) is more secure than x86. x86_64 has larger address space, thus higher ASLR entropy. The exploit needs 10 minutes to crack ASLR on x86, but 70 minutes on amd64. If some alert systems have been deploy on the server (attacks need to keep crashing systemd-journald in this process), it buys time. In other cases, it makes exploitation infeasible. 2. CFLAGS hardening works, in addition to ASLR, it's the last line of defense for all C programs. As long as there are still C programs running, patching all memory corruption bugs is impossible. Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. All hardening flags should be turned on by all distributions, unless there is a special reason. Fedora turned "-fstack-clash-protection" on since Fedora 28 (https://fedoraproject.org/wiki/Changes/HardeningFlags28 https://fedoraproject.org/wiki/Changes/HardeningFlags28). If you are releasing a C program on Linux, please consider the following, -D_FORTIFY_SOURCE=2 glibc hardening -Wp,-D_GLIBCXX_ASSERTIONS glibc++ hardening -fstack-protector-strong stack smash protection -fstack-clash-protection stack clash protection -fPIE -pie better ASLR protection -Wl,-z,noexecstack don't allow code on stack -Wl,-z,relro ELF hardening -Wl,-z,now ELF hardening Major Linux distributions, including Fedora, Debian, Arch Linux, openSUSE are already doing it. Similarly, Firefox and Chromium are using many of these flags too. Unfortunately, Debian did not use `-fstack-clash-protection` and got hit by the exploit, because it was only added since GCC 8. For a more comprehensive review, check * Recommended compiler and linker flags for GCC: https://developers.redhat.com/blog/2018/03/21/compiler-and-linker-flags-gcc/ https://developers.redhat.com/blog/2018/03/21/compiler-and-l... * Debian Hardening https://wiki.debian.org/Hardening https://wiki.debian.org/Hardening
- devereaux 8y ago
  What is the performance overhead for using these flags?
  - segfaultbuserr 8y ago
    It's very hard to find solid numbers, because these flags are introduced and popularized at different times. But I'll try to give you my best summary. Long shory short, these flags are considered essentially and most web browsers and distributions have already enabled them without problems. Let's see... -D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS These two are considered very cheap. It enables some sanity checks for buffer overflow in glibc and glibc++, respectively. See https://access.redhat.com/blogs/766093/posts/3606481 https://access.redhat.com/blogs/766093/posts/3606481 and -fstack-protector-strong Also known as stack canary. The earliest version is simply called "-fstack-protector", but the coverage was bad (only "protects" <2% functions), there was also "-fstack-protector-full", adding checks to ALL functions, but the performance overhead is unacceptable. The "-strong" variant is an upgraded version, it uses a heuristic when determining if a function needs stack protectors, provides much better code coverage and a tradeoff between security and performance. It's relatively expensive, but is considered essential to prevent attackers from smashing the stack, and is enabled in most web browser engines. See https://lwn.net/Articles/584225/ https://lwn.net/Articles/584225/ -fstack-clash-protection stack clash protection Introduced in GCC 8.0 after the Stack Clash attack was discovered, see https://securingsoftware.blogspot.com/2017/12/stack-clash-protection-in-gcc.html https://securingsoftware.blogspot.com/2017/12/stack-clash-pr... -fPIE -pie better ASLR protection PIE is Position Independent-Executable, Address Space Layout Randomization needs it to achieve its full level of protection. It has been around for a long time, but only saw a mass adoption recently, possibly due to higher performance impact on 32-bit x86, but favorable on 64-bit systems. Red Hat has a performance analysis. * https://access.redhat.com/blogs/766093/posts/1975803 https://access.redhat.com/blogs/766093/posts/1975803 Arch Linux has also benchmarked its performance impact before deciding to enable it. * https://github.com/pid1/test-sec-flags/wiki https://github.com/pid1/test-sec-flags/wiki -Wl,-z,noexecstack don't allow code on stack It just disables the executable stack, no overhead. And normally the stack is NOT executable, unless under some special conditions like linking assembly code, due to bad default settings in GNU toolchains for historical reasons. This explicitly tells the linker: don't allow an executable stack, no matter what. -Wl,-z,relro ELF hardening -Wl,-z,now ELF hardening It marks the GOT table as readonly, and force the system to resolve all function references to shared libraries at startup time, instead of runtime. It has some impact on startup time, but mostly negligible. See https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%20techniques/exploitation%20techniques/2011/06/02/relro-relocation-read-only/ https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%....
lousken 8y ago
Was systemd ever properly audited? That's exactly why i use devuan as my home server.
- tinus_hn 8y ago
  Was init?
  - insertcredit 8y ago
    The complexity of Sysvinit is orders of magnitude less than that of systemd.
    - dijit 8y ago
      The "sysvinit" binary package has had a total of 2 bug reports tagged "security" filed in Debian's bug tracker. That's over the entire history of the package that goes back to at least 2004. If I extend the search to all packages built from the "sysvinit" source (which includes packages like initscripts and sysvinit-utils), the count increases to 8. (source: https://www.debian.org/Bugs https://www.debian.org/Bugs. I'm not linking to exact queries since they take quite some time and HN has a tendency of taking down Debian's bug tracker)
      - nailer 8y ago
        Also crond, atd, and various log manipulation tools that systemd replaces.
        zzzcpan 8y ago
        It doesn't replace them, it just doesn't use them.
        dijit 8y ago
        It supplants them with the same functionality. That's literally the definition of replacement.
jononor 8y ago
Are there static analysis tools that catch such problems? User controlled buffer length for alloca.
sus_007 8y ago
If you're curious to know further critics regarding Systemd, then without-systemd[1] has got you covered. [1] : http://without-systemd.org/wiki/index.php/Main_Page http://without-systemd.org/wiki/index.php/Main_Page
- blablabla123 8y ago
  Yeah I know that one. It puzzles me how systemd has such a broad support among distributions and end users. It just redoes ages of system management in such new ways that they don't exactly fit the picture. We all know that some stuff of Linux sucks but this can't seriously be the solution. Solving things the windows ways. ;)
  - yarrel 8y ago
    SystemD doesn't have broad support among end users. It has weary resignation and a few people who don't know how grep works.
    - adontz 8y ago
      Honestly, if systemd is really that bad, why everyone adopted it? We can blame Red Hat as Lennard's employer for forcing systemd into their family of distributions (RedHat, CentOS, Fedora, CoreOS). But why Debian/Ubuntu? Why ArchLinux and Gentoo which were never ever targeted at "people who don't know how grep works"? Why SUSE? Either systemd brings more good, than harm; or we have to blame Illuminati and Masonry.
      - deleted 8y ago
        [deleted]
      - insertcredit 8y ago
        You are using logical fallacies in your argument. First, not _everyone_ has adopted it (loaded language). Google, which controls the vast majority of Linux systems on the planet, has not. GNU has not. Others [1] have not. Second, the critique against systemd is substantial and solid enough to stand on its own regardless of popularity. Popularity does not imply quality, you should read "Worse is better" by Richard Gabriel. Politics, network effects and an octopus-like architecture that imposes itself via ever-increasing interdependencies are reasonable explanations to systemd adoption. For a distribution provider or package maintainer, it has gotten to the point where it's easier to go along with systemd than try and fight it, since the latter option means extra work. This is really a sad state of affairs. [1] http://without-systemd.org/wiki/index.php/Main_Page http://without-systemd.org/wiki/index.php/Main_Page
dabockster 8y ago
I knew it was a bad exploit when I saw stpcpy() being used in the code. Always use the “n” versions (eg stpncpy()) and measure your sizes!
- kevin_thibedeau 8y ago
  stpncpy() is still brain dead for copying null-terminated strings. Regardless of your viewpoint on truncating strings, the zero-pad behavior is wasteful and rarely necessary.
  - dabockster 8y ago
    My point stands. C/C++ aren’t memory safe. You can’t just copy stuff around in memory and expect the compiler to fix it for you when the sizes don’t match. This isn’t Python or Node. I’m seriously curious as to why the non-n versions are still allowed to compile. The dangers seem way too real.
    - loeg 8y ago
      C and C++ give you low level tools, in a variety of senses. For backwards-compatibility reasons, which is a big part of the value of languages like C, C++, Go, and Java, removing anything is difficult or impossible. We barely got rid of gets(3) in C11, which is and has always been impossible to use safely. In contrast, it is at least possible to use the non-n versions of string routines safely. In addition, the 'n' versions have deleterious side effects that make their adoption unappealing: * zero padding to n * no nul termination for strings of length n or longer The BSD 'l' versions (strlcpy, etc) don't have either of those first two problems, but do have: * not in standard C, nor POSIX; * as a result, glibc still refuses to implement them In addition, some people will inevitably complain that both 'n' and 'l' variants inevitably truncate source strings if they don't fit, and that therefore no one should use either of them, you should just perfectly calculate lengths and use the unchecked ones without making mistakes. I can't tell if these people are delusional or trolling, but realistically, programmers make mistakes, and using unchecked string routines is a common source of buffer overflow; this is usually a worse problem than string truncation. Your point doesn't stand.
kevin_thibedeau 8y ago
Is Coverity being run against systemd? Would it not have found these issues?
- uniformlyrandom 8y ago
  Usefulness of static code analysis beyond linting is greatly overstated (by people selling static code analysis tools).
  - pjmlp 8y ago
    As proven by a large majority of C developers ignoring lint since 1979.
  - kevin_thibedeau 8y ago
    Coverity isn't useless. It finds these kinds of buffer overflow errors easily. It's shameful that a prominent keystone FOSS project is using such outdated coding practices in the first place. Not using the free tooling available for such projects is doubly so.
    - tssva 8y ago
      Systemd uses Simmle LGTM and QL for static code analysis which is a good thing since Coverty Scan is currently down without an ETA for restoration.
      - techslave 8y ago
        the flaws have existed for years.
zzzcpan 8y ago
I suppose prlimit'ing stack size for systemd-journald could work against this exploit too, since clash protection works, but without recompiling anything.
admax88q 8y ago
Jesus Christ stop using C.
pdkl95 8y ago
> If we send a large "native" message to /run/systemd/journal/socket ... the maximum size of a "native" entry is 768MB Why does journal allow such large messages over a socket? That alone might be a denial-of-service attack; are real messages delayed/blocked if someone sends spawns a bunch of processes that send 768MB of junk to that socket? commit c4aa09b06f835c91cea9e021df4c3605cff2318d Date: Mon Apr 8 20:32:03 2013 +0200 ... -#define ENTRY_SIZE_MAX (1024*1024*64) -#define DATA_SIZE_MAX (1024*1024*64) ... +#define ENTRY_SIZE_MAX (1024*1024*768) +#define DATA_SIZE_MAX (1024*1024*768) WTF? Why would you need to increase the max size so much? What are you intending to send over that socket?! Oh. From the commit[1], the missing lines at the 2nd "...": +/* Make sure not to make this smaller than the maximum coredump + * size. See COREDUMP_MAX in coredump.c */ Why would you send coredumps over a socket? Just write them to a file and send the file's path to journald. Increasing the max message size 1200% just to avoid writing a core file is crazy. [1] https://github.com/systemd/systemd/commit/c4aa09b06f835c91cea9e021df4c3605cff2318d https://github.com/systemd/systemd/commit/c4aa09b06f835c91ce...
- xena 8y ago
  Sockets are files internally to Linux.
- int0x80 8y ago
  Another big question is why does it use alloca at all. Considering that it is well known in the C world to be problematic and discouraged. For this very reason. - Are you performance-constrained? NO. - Is calling malloc/free such a big deal, instead of alloca? It is not. Yes. That neat join funcion will need to get an extra parameter and free the returned pointer when it's done with it. This is C. If you want neat convenient string joining functions use a different language.
  - newnewpdro 8y ago
    > - Are you performance-constrained? NO. A significant portion of issues filed against journald are performance related. It's undesirable to have a bunch of mallocs and frees in the hot path of logging. It's not inappropriate to use the stack where possible, obviously in this case user-controlled input was being trusted where it shouldn't have. There have been many problems of this sort in journald, where the nature of the user-controlled input has the ability to bring journald to its knees in a global manner. Prior to the process metadata caching, you could cause journald to go absolutely nuts querying the sending processes metadata from /proc by sending it messages full of newlines. Every newline-separated field is treated as a separate message with its own metadata to be acquired. The routines for fetching all that stuff from /proc granularly allocated and freed the data via libc as well, it was very painful. Things are better now, but you still can't log particularly high rates through journald compared to a simple O_APPEND style logger like syslogd. Programs synchronously logging debug messages quickly become slowed down by journald pinning a CPU. You can still cause a world of hurt by spamming journald with tiny emergency messages, each of which forces a sync of the journal. But a user can also run `while true; do sync; done`
egberts1 8y ago
I still use ISC DHCP client due to enterprise DHCP using some esoteric DHCP Options that systemd cannot do to this day.
kazinator 8y ago
https://wiki.sei.cmu.edu/confluence/display/c/MEM05-C.+Avoid+large+stack+allocations https://wiki.sei.cmu.edu/confluence/display/c/MEM05-C.+Avoid... > This compliant solution replaces the VLA with a call to malloc(). If malloc() fails, the return value can be checked to prevent the program from terminating abnormally. Wishful thinking. On overcommit-enabled VM systems (pretty much every GNU/Linux desktop and server by default), malloc will cheerfully return a valid-looking pointer to virtual memory even if the system is out of memory. Your process will get a fatal signal when later it tries to use the memory. It will not be prevented from terminating abnormally. Any page of the memory could blow up. Maybe the first three pages get a frame just fine, but when the process touches the fourth, it faults. The reason to use malloc instead of the stack is that it goes awry only when the system is out of memory, not when the stack is out of room.
- geofft 8y ago
  But the only failure case there is a DoS. You cannot get memory corruption from overcommit the way you can from a stack clash, and therefore you cannot get arbitrary code execution. That seems like a huge reason to prefer overcommit.
  - kazinator 8y ago
    Stack clashes are protected with guard pages. Guard pages can be skipped, but not by code like strjoina which fills in all of the bytes. https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt https://www.qualys.com/2017/06/19/stack-clash/stack-clash.tx... The reporters do claim to have a root exploit that they will publish later, which will be interesting (if it still applies on systems with robust guard pages). The strings are copied into the alloca area from the lower address to the higher, so initially, the stpcpy calls might be corrupting memory belonging to an adjacent mapping and not to the stack. Eventually the loop has to cross back into the stack, and it cannot do so without writing into a guard page. Situations where that is still exploitable can be contrived. For instance, the program sets up a SIGSEGV handler to execute on an alternate stack using sigaltstack. When the stack guard page is hit, that handler executes successfully, the program continues somehow (e.g. by a recovering longjmp that rewinds the stack) and then falls victim to the corruption. One thing we can do is after calling alloca, walk over the memory in reverse order (higher address to lower) in strides of 4096 bytes, and touch a byte at every increment. This way we hit the guard page (on any system that has a page size of at least 4096).
_emacsomancer_ 8y ago
systemd is interesting as an innovative init/daemon-manager(+), but I remain convinced it's not ideal that it is de facto Linux default init. Surely the default init should be something more manageable (i.e. with fewer attack surfaces) like runit or OpenRC (in combination with something, perhaps runit) or s6 etc.
- chousuke 8y ago
  This attack doesn't touch the init system's attack surface though. It breaks journald, which is a separate program that you don't even have to run (syslog daemons still work) Calling this a bug in the init system is like assigning fault for bugs in eg. rsyslog to sysvinit
  - insertcredit 8y ago
    This is disingenuous, systemd-journald is the default in every systemd-using distribution I am aware of. The philosophy of systemd is all about tight coupling and forcing its singular vision on end users. When that vision falls apart you can not claim it is not really a systemd problem because in theory you could have gone out of your way and done something that is not encouraged.
  - _emacsomancer_ 8y ago
    But you know that my post is a criticism of the systemd ecosystem, or, more specifically, its status as the de facto. All my systems that run systemd run journald; all of my systems that run journald run systemd. Because those are the choices the distro devs made. My systems which don't run systemd are immune to these vulnerabilities.
  - gmueckl 8y ago
    No, you cannot run systemd without journald. Journald can forward messages to another syslog implementation. Its launch is even hardcoded into systemd very early in the boot process IIRC. The intent is to capture early log messages in memory inside journald before a writable /var is available.
- zanny 8y ago
eikenberry 8y ago
I don't see any systems with security releases ready. Did they not notify the effected systems before making it public? I thought that was standard procedure these days.
- kasabali 8y ago
  It's in the linked page: 2018-11-26: Advisory sent to Red Hat Product Security (as recommended by https://github.com/systemd/systemd/blob/master/docs/CONTRIBUTING.md#security-vulnerability-reports https://github.com/systemd/systemd/blob/master/docs/CONTRIBU...). 2018-12-26: Advisory and patches sent to linux-distros@...nwall. 2019-01-09: Coordinated Release Date (6:00 PM UTC).
upofadown 8y ago
>On a Debian stable (9.5), our proof of concept wins this race and gains eip control after a dozen tries (systemd automatically restarts journald after each crash): Seems like a good example of why automatically restarting things that crash is a bad idea.
- loeg 8y ago
  As with anything, there are tradeoffs. A dozen is too short to really make a difference, but one thing that can be done at the service management layer without abandoning restart entirely is a delayed restart, or exponential backoff. You could also imagine a per-service option for auto restart. Paranoid organizations with plenty of on-call engineers could disable auto-restart, if they were convinced the engineers wouldn't just rig up their own auto-restart to avoid the call.
NelsonMinar 8y ago
Who could have guessed that rewriting 30 years of accumulated Unix systems would be so complicated?
z3t4 8y ago
How can you check if you systemd is patched ? This sounds very serious.
throw2016 8y ago
There is a pattern of designing for the most niche use cases and imposing complexity on everyone by default. Shouldn't it be the other way round? Letting those with advanced needs accept the complexity? Presumably they have the know how. There are multiple examples of this including the binary journal for those with extreme auditing needs, optimizing for laptop wifi networks when Linux is predominantly used in servers, the ironically named 'predictable network names' that are anything but predictable. And these issues being hand-waved away because of some fringe use case with the additional overhead nonchalantly imposed on all users.
codedokode 8y ago
I think that putting a hard limit on alloca allocation size, somewhere around 1-10 Kb, could help prevent this type of exploits. Also, it is possible to write a function that would choose between alloca() and malloc()/free() depending on the size.
posix_me_less 8y ago
It's been two weeks since the GNU/Linux distributors were notified of this, and still no fixes are available in Redhat/Centos/Debian. Any ideas about how to mitigate this until official fixes are available? Would it help to block autorestart of journald? Any idea about how to that?

385 points by gmueckl 8y ago

ape4 8y ago

Top comment from reddit thread: FWIW distros that use -fstack-clash-protection to compile systemd, including recent Fedora and OpenSUSE, aren't vulnerable. https://www.reddit.com/r/linux/comments/aeac8g/systemd_earns_three_cves_can_be_used_to_gain/ https://www.reddit.com/r/linux/comments/aeac8g/systemd_earns...

detaro 8y ago
It says so in the article too.
nailer 8y ago
Also in the article: > To the best of our knowledge, all systemd-based Linux distributions are vulnerable, but SUSE Linux Enterprise 15, openSUSE Leap 15.0, and Fedora 28 and 29 are not exploitable because their user space is compiled with GCC's -fstack-clash-protection.
- krylon 8y ago
  Does that include openSuse Tumbleweed?
userbinator 8y ago
I'm of two minds when it comes to mitigations like this. On the one hand, it helps decrease the number of exploits. On the other hand, it encourages/tolerates the proliferation of low-quality code.
- loeg 8y ago
  Stack clash protection is a necessary implementation detail of the C language. It's a shame it's even optional. C17 § 6.5.16.1 Simple assignment, Semantics, (3): > If the value being stored in an object is read from another object that overlaps in any way the storage of the first object, then the overlap shall be exact and the two objects shall have qualified or unqualified versions of a compatible type; otherwise, the behavior is undefined. Also listed under J.2 "Undefined Behavior," with reference to the same section. The same clause is present in C11 and C99 under the same section.
  - saagarjha 8y ago
    > otherwise, the behavior is undefined I don't see how this necessitates stack protection at all, at least from a standards perspective.
    - loeg 8y ago
      Ok, here's the logic: 0. Your premise is that standard C does not require a runtime to provide any protection from stack memory growing into the heap. 1. Standard C defines automatic object storage and heap object storage. 2. It is implementation defined where heap objects are allocated in memory. As a result, the type and compatibility of arbitrary heap memory locations are unpredictable to developers. 3. It is implementation defined where stack objects are allocated in memory. As a result, the type and compatibility of arbitrary stack memory locations are unpredictable to developers. 4. Standard C says that mutated objects can't overlap [more or less]. 5. Optimizing compilers don't allocate memory for stack objects that aren't at least partially mutated. 6. Standard C defines no way for developers to determine how deep their stack is, or where the top of the heap is. 7. From (2), (3), (5), (6), it is impossible for developers to determine whether or not they would violate (4). 8. As a result of (7) and (0), any use of both heap and stack objects is UB. A C runtime compliant to your definition in (0) could start the heap from 0x100, growing up, and the stack at 0x104, growing down. 8.a. Similarly, any use of static or other lifetime, and stack objects, is UB. 9. It logically follows from 8 that, under your premise (0), it is always UB to the stack and other kinds of data memory. I guess you could make the case that the C committee hate everyone and all stack objects are UB. But my read of the C standard is that the intent is that any use of stack objects is not in fact required to be UB. If stack objects were always UB, I think it would be called out explicitly or not specified in the standard. So yes, I think any standard-conforming C implementation has to prevent stack clash one way or another. That could take the form of an alternate stack memory that is entirely separate from heap memory; it doesn't have to be probing and guard pages.
      - saagarjha 8y ago
        > I guess you could make the case that the C committee hate everyone and all stack objects are UB. Honestly, I think that this is the logical conclusion, or at least, every compiler/machine implementation is subtly wrong; the standards glosses over what should happen when the stack is too large, for example, so technically any use of the stack is not guaranteed to be valid because you might just be out of space. The C standard is for a "head-in-the-clouds" abstract machine, so it doesn't concern itself with pedestrian issues like lack of memory.

gmueckl 8y ago

These are three long standing exploitable bugs in systemd's journald which can be used to gain local root privileges. Most systemd based distributions are vulnerable and, from the looks of it, may have been for years.

nightfly 8y ago
link?
- mgalgs 8y ago
  I believe he's giving a synopsis of TFA

deleted 8y ago

[deleted]

insertcredit 8y ago

I refuse to use systemd to this day. It's unbelievably complex and became established through political power play rather than any sort of merit. Which is without a doubt not what I expected to see in the Linux ecosystem.

isatty 8y ago
I do too, for personal use (OpenRC on Gentoo user through and through), but due to the adoption of systemd by popular distributions I don’t see an alternative for work. Unfortunately.
- geggam 8y ago
  I moved all my personal things to OpenBSD due to the Linux community move away from unix standards
  - DyslexicAtheist 8y ago
    seconded. the Linux fanboy community (which I've no doubt been part of until today and still defend bc of the FOSS spirit) is just getting too much. Linux is a victim of its own success. Bloat and inconsistency everywhere and compared to openBSD they rather move fast and break things rather than chose coherent design. Linux Security as it exists today is an oxymoron. Not to mention the community that consists in BSD of people who know what they do and can be trusted with auditing code. Linux code audits are a joke ... they rather hurl abuse at security guys (e.g. "masturbating monkeys") or outright ignore them ... (no time to discuss esthetics and coherent design because it's all about shipping fast at all cost - the same and only way they do in their day jobs ...)
  - smhenderson 8y ago
    That's what I did as well. But I also continue to use Slackware which doesn't have any current plans to move to SystemD. It was the first Linux distro I ever used and I still love it to this day. I've used many different systems for work over the years but I've kept at least one Slackware laptop or desktop going since early 1995 when I first found a Slackware CDROM in the back of a book about learning Linux. Although it gets new software, bug fixes and minor updates here and there it's remarkably similar today to what it was then. And I find it Just Works, I've had very little trouble with it over the years. I think I like it too because it always felt like the most traditional Unix like distro I've ever used and also more BSD than System V which I also generally prefer.

AdmiralAsshat 8y ago

First CVE I've seen to reference a System of a Down song. (Every subsection starts with an SOAD song quote)

nailer 8y ago
Off topic, but SOAD were really one of the most enduring metal acts of the early 2000s. If you remember them, Needle Drop recently did a classic review of Toxicity - enjoy the nostalgia! https://www.youtube.com/watch?v=-jI1ofec02A https://www.youtube.com/watch?v=-jI1ofec02A
rconti 8y ago
I'm not a SOAD fan, but good to see someone at my former employer still has a sense of humor :)

Twirrim 8y ago

I might be reading that article wrong, but it reads like they're critical of the use of alloca? I've never really written C or the like, what is dangerous about alloca?

speps 8y ago
Variable size allocation from the stack... That's one dangerous trick. [1] https://stackoverflow.com/questions/1018853/why-is-the-use-of-alloca-not-considered-good-practice https://stackoverflow.com/questions/1018853/why-is-the-use-o...
pilif 8y ago
It allocates on the stack which is inherently of limited size and allocations grow in the direction of the “juicy” stuff. As so often with the C library, there is little bounds checking going on either (in fact, the behavior once you overflow the stack is explicitly undefined, so who knows what the hell is going to happen) If you somehow let the user specify the amount of bytes you’re going to allocate on the stack, then that’s an exploitable issue.
- Twirrim 8y ago
  Ah, OK. So it is truly horrifying that it's in heavy use in journald, where users inevitably have that sort of power.
deleted 8y ago
[deleted]

segfaultbuserr 8y ago

Yet another proof for the following: 1. It's reasonable to claim that amd64 (x86_64) is more secure than x86. x86_64 has larger address space, thus higher ASLR entropy. The exploit needs 10 minutes to crack ASLR on x86, but 70 minutes on amd64. If some alert systems have been deploy on the server (attacks need to keep crashing systemd-journald in this process), it buys time. In other cases, it makes exploitation infeasible. 2. CFLAGS hardening works, in addition to ASLR, it's the last line of defense for all C programs. As long as there are still C programs running, patching all memory corruption bugs is impossible. Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. All hardening flags should be turned on by all distributions, unless there is a special reason. Fedora turned "-fstack-clash-protection" on since Fedora 28 (https://fedoraproject.org/wiki/Changes/HardeningFlags28 https://fedoraproject.org/wiki/Changes/HardeningFlags28). If you are releasing a C program on Linux, please consider the following, -D_FORTIFY_SOURCE=2 glibc hardening -Wp,-D_GLIBCXX_ASSERTIONS glibc++ hardening -fstack-protector-strong stack smash protection -fstack-clash-protection stack clash protection -fPIE -pie better ASLR protection -Wl,-z,noexecstack don't allow code on stack -Wl,-z,relro ELF hardening -Wl,-z,now ELF hardening Major Linux distributions, including Fedora, Debian, Arch Linux, openSUSE are already doing it. Similarly, Firefox and Chromium are using many of these flags too. Unfortunately, Debian did not use `-fstack-clash-protection` and got hit by the exploit, because it was only added since GCC 8. For a more comprehensive review, check * Recommended compiler and linker flags for GCC: https://developers.redhat.com/blog/2018/03/21/compiler-and-linker-flags-gcc/ https://developers.redhat.com/blog/2018/03/21/compiler-and-l... * Debian Hardening https://wiki.debian.org/Hardening https://wiki.debian.org/Hardening

devereaux 8y ago
What is the performance overhead for using these flags?
- segfaultbuserr 8y ago
  It's very hard to find solid numbers, because these flags are introduced and popularized at different times. But I'll try to give you my best summary. Long shory short, these flags are considered essentially and most web browsers and distributions have already enabled them without problems. Let's see... -D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS These two are considered very cheap. It enables some sanity checks for buffer overflow in glibc and glibc++, respectively. See https://access.redhat.com/blogs/766093/posts/3606481 https://access.redhat.com/blogs/766093/posts/3606481 and -fstack-protector-strong Also known as stack canary. The earliest version is simply called "-fstack-protector", but the coverage was bad (only "protects" <2% functions), there was also "-fstack-protector-full", adding checks to ALL functions, but the performance overhead is unacceptable. The "-strong" variant is an upgraded version, it uses a heuristic when determining if a function needs stack protectors, provides much better code coverage and a tradeoff between security and performance. It's relatively expensive, but is considered essential to prevent attackers from smashing the stack, and is enabled in most web browser engines. See https://lwn.net/Articles/584225/ https://lwn.net/Articles/584225/ -fstack-clash-protection stack clash protection Introduced in GCC 8.0 after the Stack Clash attack was discovered, see https://securingsoftware.blogspot.com/2017/12/stack-clash-protection-in-gcc.html https://securingsoftware.blogspot.com/2017/12/stack-clash-pr... -fPIE -pie better ASLR protection PIE is Position Independent-Executable, Address Space Layout Randomization needs it to achieve its full level of protection. It has been around for a long time, but only saw a mass adoption recently, possibly due to higher performance impact on 32-bit x86, but favorable on 64-bit systems. Red Hat has a performance analysis. * https://access.redhat.com/blogs/766093/posts/1975803 https://access.redhat.com/blogs/766093/posts/1975803 Arch Linux has also benchmarked its performance impact before deciding to enable it. * https://github.com/pid1/test-sec-flags/wiki https://github.com/pid1/test-sec-flags/wiki -Wl,-z,noexecstack don't allow code on stack It just disables the executable stack, no overhead. And normally the stack is NOT executable, unless under some special conditions like linking assembly code, due to bad default settings in GNU toolchains for historical reasons. This explicitly tells the linker: don't allow an executable stack, no matter what. -Wl,-z,relro ELF hardening -Wl,-z,now ELF hardening It marks the GOT table as readonly, and force the system to resolve all function references to shared libraries at startup time, instead of runtime. It has some impact on startup time, but mostly negligible. See https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%20techniques/exploitation%20techniques/2011/06/02/relro-relocation-read-only/ https://blog.osiris.cyber.nyu.edu/exploitation%20mitigation%....

lousken 8y ago

Was systemd ever properly audited? That's exactly why i use devuan as my home server.

tinus_hn 8y ago
Was init?
- insertcredit 8y ago
  The complexity of Sysvinit is orders of magnitude less than that of systemd.
  - dijit 8y ago
    The "sysvinit" binary package has had a total of 2 bug reports tagged "security" filed in Debian's bug tracker. That's over the entire history of the package that goes back to at least 2004. If I extend the search to all packages built from the "sysvinit" source (which includes packages like initscripts and sysvinit-utils), the count increases to 8. (source: https://www.debian.org/Bugs https://www.debian.org/Bugs. I'm not linking to exact queries since they take quite some time and HN has a tendency of taking down Debian's bug tracker)
    - nailer 8y ago
      Also crond, atd, and various log manipulation tools that systemd replaces.
      - zzzcpan 8y ago
        It doesn't replace them, it just doesn't use them.
        dijit 8y ago
        It supplants them with the same functionality. That's literally the definition of replacement.

jononor 8y ago

Are there static analysis tools that catch such problems? User controlled buffer length for alloca.

sus_007 8y ago

If you're curious to know further critics regarding Systemd, then without-systemd[1] has got you covered. [1] : http://without-systemd.org/wiki/index.php/Main_Page http://without-systemd.org/wiki/index.php/Main_Page

blablabla123 8y ago
Yeah I know that one. It puzzles me how systemd has such a broad support among distributions and end users. It just redoes ages of system management in such new ways that they don't exactly fit the picture. We all know that some stuff of Linux sucks but this can't seriously be the solution. Solving things the windows ways. ;)
- yarrel 8y ago
  SystemD doesn't have broad support among end users. It has weary resignation and a few people who don't know how grep works.
  - adontz 8y ago
    Honestly, if systemd is really that bad, why everyone adopted it? We can blame Red Hat as Lennard's employer for forcing systemd into their family of distributions (RedHat, CentOS, Fedora, CoreOS). But why Debian/Ubuntu? Why ArchLinux and Gentoo which were never ever targeted at "people who don't know how grep works"? Why SUSE? Either systemd brings more good, than harm; or we have to blame Illuminati and Masonry.
    - deleted 8y ago
      [deleted]
    - insertcredit 8y ago
      You are using logical fallacies in your argument. First, not _everyone_ has adopted it (loaded language). Google, which controls the vast majority of Linux systems on the planet, has not. GNU has not. Others [1] have not. Second, the critique against systemd is substantial and solid enough to stand on its own regardless of popularity. Popularity does not imply quality, you should read "Worse is better" by Richard Gabriel. Politics, network effects and an octopus-like architecture that imposes itself via ever-increasing interdependencies are reasonable explanations to systemd adoption. For a distribution provider or package maintainer, it has gotten to the point where it's easier to go along with systemd than try and fight it, since the latter option means extra work. This is really a sad state of affairs. [1] http://without-systemd.org/wiki/index.php/Main_Page http://without-systemd.org/wiki/index.php/Main_Page

dabockster 8y ago

I knew it was a bad exploit when I saw stpcpy() being used in the code. Always use the “n” versions (eg stpncpy()) and measure your sizes!

kevin_thibedeau 8y ago
stpncpy() is still brain dead for copying null-terminated strings. Regardless of your viewpoint on truncating strings, the zero-pad behavior is wasteful and rarely necessary.
- dabockster 8y ago
  My point stands. C/C++ aren’t memory safe. You can’t just copy stuff around in memory and expect the compiler to fix it for you when the sizes don’t match. This isn’t Python or Node. I’m seriously curious as to why the non-n versions are still allowed to compile. The dangers seem way too real.
  - loeg 8y ago
    C and C++ give you low level tools, in a variety of senses. For backwards-compatibility reasons, which is a big part of the value of languages like C, C++, Go, and Java, removing anything is difficult or impossible. We barely got rid of gets(3) in C11, which is and has always been impossible to use safely. In contrast, it is at least possible to use the non-n versions of string routines safely. In addition, the 'n' versions have deleterious side effects that make their adoption unappealing: * zero padding to n * no nul termination for strings of length n or longer The BSD 'l' versions (strlcpy, etc) don't have either of those first two problems, but do have: * not in standard C, nor POSIX; * as a result, glibc still refuses to implement them In addition, some people will inevitably complain that both 'n' and 'l' variants inevitably truncate source strings if they don't fit, and that therefore no one should use either of them, you should just perfectly calculate lengths and use the unchecked ones without making mistakes. I can't tell if these people are delusional or trolling, but realistically, programmers make mistakes, and using unchecked string routines is a common source of buffer overflow; this is usually a worse problem than string truncation. Your point doesn't stand.

kevin_thibedeau 8y ago

Is Coverity being run against systemd? Would it not have found these issues?

uniformlyrandom 8y ago
Usefulness of static code analysis beyond linting is greatly overstated (by people selling static code analysis tools).
- pjmlp 8y ago
  As proven by a large majority of C developers ignoring lint since 1979.
- kevin_thibedeau 8y ago
  Coverity isn't useless. It finds these kinds of buffer overflow errors easily. It's shameful that a prominent keystone FOSS project is using such outdated coding practices in the first place. Not using the free tooling available for such projects is doubly so.
  - tssva 8y ago
    Systemd uses Simmle LGTM and QL for static code analysis which is a good thing since Coverty Scan is currently down without an ETA for restoration.
    - techslave 8y ago
      the flaws have existed for years.

zzzcpan 8y ago

I suppose prlimit'ing stack size for systemd-journald could work against this exploit too, since clash protection works, but without recompiling anything.

admax88q 8y ago

Jesus Christ stop using C.

pdkl95 8y ago

> If we send a large "native" message to /run/systemd/journal/socket ... the maximum size of a "native" entry is 768MB Why does journal allow such large messages over a socket? That alone might be a denial-of-service attack; are real messages delayed/blocked if someone sends spawns a bunch of processes that send 768MB of junk to that socket? commit c4aa09b06f835c91cea9e021df4c3605cff2318d Date: Mon Apr 8 20:32:03 2013 +0200 ... -#define ENTRY_SIZE_MAX (1024*1024*64) -#define DATA_SIZE_MAX (1024*1024*64) ... +#define ENTRY_SIZE_MAX (1024*1024*768) +#define DATA_SIZE_MAX (1024*1024*768) WTF? Why would you need to increase the max size so much? What are you intending to send over that socket?! Oh. From the commit[1], the missing lines at the 2nd "...": +/* Make sure not to make this smaller than the maximum coredump + * size. See COREDUMP_MAX in coredump.c */ Why would you send coredumps over a socket? Just write them to a file and send the file's path to journald. Increasing the max message size 1200% just to avoid writing a core file is crazy. [1] https://github.com/systemd/systemd/commit/c4aa09b06f835c91cea9e021df4c3605cff2318d https://github.com/systemd/systemd/commit/c4aa09b06f835c91ce...

xena 8y ago
Sockets are files internally to Linux.
int0x80 8y ago
Another big question is why does it use alloca at all. Considering that it is well known in the C world to be problematic and discouraged. For this very reason. - Are you performance-constrained? NO. - Is calling malloc/free such a big deal, instead of alloca? It is not. Yes. That neat join funcion will need to get an extra parameter and free the returned pointer when it's done with it. This is C. If you want neat convenient string joining functions use a different language.
- newnewpdro 8y ago
  > - Are you performance-constrained? NO. A significant portion of issues filed against journald are performance related. It's undesirable to have a bunch of mallocs and frees in the hot path of logging. It's not inappropriate to use the stack where possible, obviously in this case user-controlled input was being trusted where it shouldn't have. There have been many problems of this sort in journald, where the nature of the user-controlled input has the ability to bring journald to its knees in a global manner. Prior to the process metadata caching, you could cause journald to go absolutely nuts querying the sending processes metadata from /proc by sending it messages full of newlines. Every newline-separated field is treated as a separate message with its own metadata to be acquired. The routines for fetching all that stuff from /proc granularly allocated and freed the data via libc as well, it was very painful. Things are better now, but you still can't log particularly high rates through journald compared to a simple O_APPEND style logger like syslogd. Programs synchronously logging debug messages quickly become slowed down by journald pinning a CPU. You can still cause a world of hurt by spamming journald with tiny emergency messages, each of which forces a sync of the journal. But a user can also run `while true; do sync; done`

egberts1 8y ago

I still use ISC DHCP client due to enterprise DHCP using some esoteric DHCP Options that systemd cannot do to this day.

kazinator 8y ago

https://wiki.sei.cmu.edu/confluence/display/c/MEM05-C.+Avoid+large+stack+allocations https://wiki.sei.cmu.edu/confluence/display/c/MEM05-C.+Avoid... > This compliant solution replaces the VLA with a call to malloc(). If malloc() fails, the return value can be checked to prevent the program from terminating abnormally. Wishful thinking. On overcommit-enabled VM systems (pretty much every GNU/Linux desktop and server by default), malloc will cheerfully return a valid-looking pointer to virtual memory even if the system is out of memory. Your process will get a fatal signal when later it tries to use the memory. It will not be prevented from terminating abnormally. Any page of the memory could blow up. Maybe the first three pages get a frame just fine, but when the process touches the fourth, it faults. The reason to use malloc instead of the stack is that it goes awry only when the system is out of memory, not when the stack is out of room.

geofft 8y ago
But the only failure case there is a DoS. You cannot get memory corruption from overcommit the way you can from a stack clash, and therefore you cannot get arbitrary code execution. That seems like a huge reason to prefer overcommit.
- kazinator 8y ago
  Stack clashes are protected with guard pages. Guard pages can be skipped, but not by code like strjoina which fills in all of the bytes. https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt https://www.qualys.com/2017/06/19/stack-clash/stack-clash.tx... The reporters do claim to have a root exploit that they will publish later, which will be interesting (if it still applies on systems with robust guard pages). The strings are copied into the alloca area from the lower address to the higher, so initially, the stpcpy calls might be corrupting memory belonging to an adjacent mapping and not to the stack. Eventually the loop has to cross back into the stack, and it cannot do so without writing into a guard page. Situations where that is still exploitable can be contrived. For instance, the program sets up a SIGSEGV handler to execute on an alternate stack using sigaltstack. When the stack guard page is hit, that handler executes successfully, the program continues somehow (e.g. by a recovering longjmp that rewinds the stack) and then falls victim to the corruption. One thing we can do is after calling alloca, walk over the memory in reverse order (higher address to lower) in strides of 4096 bytes, and touch a byte at every increment. This way we hit the guard page (on any system that has a page size of at least 4096).

_emacsomancer_ 8y ago

systemd is interesting as an innovative init/daemon-manager(+), but I remain convinced it's not ideal that it is de facto Linux default init. Surely the default init should be something more manageable (i.e. with fewer attack surfaces) like runit or OpenRC (in combination with something, perhaps runit) or s6 etc.

chousuke 8y ago
This attack doesn't touch the init system's attack surface though. It breaks journald, which is a separate program that you don't even have to run (syslog daemons still work) Calling this a bug in the init system is like assigning fault for bugs in eg. rsyslog to sysvinit
- insertcredit 8y ago
  This is disingenuous, systemd-journald is the default in every systemd-using distribution I am aware of. The philosophy of systemd is all about tight coupling and forcing its singular vision on end users. When that vision falls apart you can not claim it is not really a systemd problem because in theory you could have gone out of your way and done something that is not encouraged.
- _emacsomancer_ 8y ago
  But you know that my post is a criticism of the systemd ecosystem, or, more specifically, its status as the de facto. All my systems that run systemd run journald; all of my systems that run journald run systemd. Because those are the choices the distro devs made. My systems which don't run systemd are immune to these vulnerabilities.
- gmueckl 8y ago
  No, you cannot run systemd without journald. Journald can forward messages to another syslog implementation. Its launch is even hardcoded into systemd very early in the boot process IIRC. The intent is to capture early log messages in memory inside journald before a writable /var is available.
zanny 8y ago

eikenberry 8y ago

I don't see any systems with security releases ready. Did they not notify the effected systems before making it public? I thought that was standard procedure these days.

kasabali 8y ago
It's in the linked page: 2018-11-26: Advisory sent to Red Hat Product Security (as recommended by https://github.com/systemd/systemd/blob/master/docs/CONTRIBUTING.md#security-vulnerability-reports https://github.com/systemd/systemd/blob/master/docs/CONTRIBU...). 2018-12-26: Advisory and patches sent to linux-distros@...nwall. 2019-01-09: Coordinated Release Date (6:00 PM UTC).

upofadown 8y ago

>On a Debian stable (9.5), our proof of concept wins this race and gains eip control after a dozen tries (systemd automatically restarts journald after each crash): Seems like a good example of why automatically restarting things that crash is a bad idea.

loeg 8y ago
As with anything, there are tradeoffs. A dozen is too short to really make a difference, but one thing that can be done at the service management layer without abandoning restart entirely is a delayed restart, or exponential backoff. You could also imagine a per-service option for auto restart. Paranoid organizations with plenty of on-call engineers could disable auto-restart, if they were convinced the engineers wouldn't just rig up their own auto-restart to avoid the call.

NelsonMinar 8y ago

Who could have guessed that rewriting 30 years of accumulated Unix systems would be so complicated?

z3t4 8y ago

How can you check if you systemd is patched ? This sounds very serious.

throw2016 8y ago

There is a pattern of designing for the most niche use cases and imposing complexity on everyone by default. Shouldn't it be the other way round? Letting those with advanced needs accept the complexity? Presumably they have the know how. There are multiple examples of this including the binary journal for those with extreme auditing needs, optimizing for laptop wifi networks when Linux is predominantly used in servers, the ironically named 'predictable network names' that are anything but predictable. And these issues being hand-waved away because of some fringe use case with the additional overhead nonchalantly imposed on all users.

codedokode 8y ago

I think that putting a hard limit on alloca allocation size, somewhere around 1-10 Kb, could help prevent this type of exploits. Also, it is possible to write a function that would choose between alloca() and malloc()/free() depending on the size.

posix_me_less 8y ago

It's been two weeks since the GNU/Linux distributors were notified of this, and still no fixes are available in Redhat/Centos/Debian. Any ideas about how to mitigate this until official fixes are available? Would it help to block autorestart of journald? Any idea about how to that?