18 ms·

The Dirty Pipe Vulnerability

697 points by max_k 5y ago

jesprenj 5y ago
This affects kernels from 5.8 and was fixed in 5.16.11, 5.15.25 and 5.10.102. Exploit code is public and available on the linked page.
- raesene9 5y ago
  < 5.8 not being affected is probably a saving grace for quite a few enterprises as I'd expect that LTS distributions may not have got that version included as yet.
  - gchamonlive 5y ago
    CentOS 7 is already at 5.10 so it should affect lots of production systems
    - emrvb 5y ago
      *blinks* *stares at kernel-3.10.0-1160.59.1.el7*
      - samus 5y ago
        Is this a smartphone? I'm on 3.18!
    - LinuxBender 5y ago
      Are you by chance using a 3rd party kernel repo such as ElRepo to work around a limitation? Or could someone at your org be compiling a custom kernel?
- greyface- 5y ago
  Debian stable (bullseye) is still vulnerable: https://security-tracker.debian.org/tracker/CVE-2022-0847 https://security-tracker.debian.org/tracker/CVE-2022-0847
  - deng 5y ago
    That page is not up-to-date, fix was released today: https://lists.debian.org/debian-security-announce/2022/msg00059.html https://lists.debian.org/debian-security-announce/2022/msg00...
    - greyface- 5y ago
      It wasn't available via `apt-get update && apt-get dist-upgrade` as of when I drafted that comment, but I confirm that 5.10.92-2 seems to be released now.
      - deng 5y ago
        Well the fix was released ~30 minutes ago, so that checks out. ;-) The security-tracker site is now updated as well.
      - bill_mcgonigle 5y ago
        Others, note that the new archive name is 'stable-security'. You might need to update your pins if you upgraded from Buster and you're not seeing the update now. I put in a pull request to add it to the release notes.
- baggy_trough 5y ago
  How about Ubuntu?
  - zenexer 5y ago
    The relevant CVE page returns a 500 error: https://github.com/canonical-web-and-design/ubuntu.com/issues/11324 https://github.com/canonical-web-and-design/ubuntu.com/issue... 21.10 appears to be lacking the patch.
    - baggy_trough 5y ago
      The CVE page returns now, with a whole bunch of "needs triage". https://ubuntu.com/security/CVE-2022-0847 https://ubuntu.com/security/CVE-2022-0847
- baggy_trough 5y ago
  It's disturbing that despite prior disclosure on distro lists, Ubuntu doesn't have an update available, with public exploits circulating now.
  - deleted 5y ago
    [deleted]
abofh 5y ago
Wow, awesome debugging - very impressed.
egberts1 5y ago
Extreme Debugger Par Excellence! What a supérioritégrandeur!
kgraves 5y ago
Google's Fuchsia/Zircon cannot come fast enough.
- amelius 5y ago
  Because they use formal methods preventing this kind of thing from happening?
  - kgraves 5y ago
    Even with your formal methods strawman, vulnerabilities like these are still possible in Linux and C. We need to move on.
    - freemint 5y ago
      You can formally verify C code against a spec though.
- k4rli 5y ago
  I wouldn't expect additional security from introducing an entirely new OS/kernel. Just unknown RCEs and other vulnerabilities waiting to be discovered.
  - kgraves 5y ago
    Just like Linux so no change there, We still need to move on.
- throwaway889900 5y ago
  Why not seL4 then?
nickelpro 5y ago
Excellent work and excellent write up Max. A feather in your cap to be proud of for sure.
piratejon 5y ago
Wow, almost 10 months from the first reported file corruption until identification as an exploitable bug.
- Ensorceled 5y ago
  I'll bite, why the "Wow"? It was a random, intermittent file corruption that didn't cause real harm to the authors organization and was, clearly, very tricky to track down.
  - piratejon 5y ago
    I don't have a basis for how long this might take. As the author mentions "All bugs become shallow once they can be reproduced.", but only after spending probably the largest amount of time waiting for new incident reports to come in, and then analyzing the reports (e.g. to determine most incidents occurred on the last day of the month), and hours staring at application and kernel code. It's very impressive, but certainly the largest amount of time in the 10 month duration was not actually debugging. The "moment of extraordinary clarity" probably sprung out of years of experience.
    - Ensorceled 5y ago
      Ah, I guess my thinking is that they didn't really focus on it. It was annoying but not high priority ... until they started to get an inkling of what was actually going on.
    - silverfox17 5y ago
      Agreed, about 99% of admins I know would not be able to identify this error, and most likely most Hacker News reads. The last sentence on your post is very true.
      - lazide 5y ago
        If not 99.999% I’ve worked with (and been) a dev for several decades, and I can count on one hand the number of folks who would have a chance of figuring this out, and 2 fingers the number of folks who WOULD. Of course, most never try to optimize or go so deep like this that they would ever need to, so there is that!
db48x 5y ago
C needs to die. Pro tip for language designers: require all fields to be initialized any time an object is created. Really impressive debugging too.
- kevincox 5y ago
  I love Rust, but would it have prevented this problem? IIUC there was no memory corruption at the language level here. This was really just a logic error.
  - pdw 5y ago
    You can't accidentally leave a field of a struct uninitialized in Rust or in other sane languages.
  - db48x 5y ago
    Yes, it would have. Some code creates an instance of some struct, but doesn’t set the flags field to zero. It thus keeps whatever value happened to be in that spot in memory, an essentially random set of bits. Rust would force you to either explicitly name the flags field and give it a value, or use `..Default::default()` to initialize all remaining fields automatically. Anything else would be a compile–time error. The fix: +++ b/lib/iov_iter.c @@ -414,6 +414,7 @@ static size_t copy_page_to_iter_pipe(struct page \*page, size_t offset, size_t by return 0; buf->ops = &page_cache_pipe_buf_ops; + buf->flags = 0; get_page(page); buf->page = page; buf->offset = offset;
    - amelius 5y ago
      Wouldn't Lint have caught the error too?
    - kevincox 5y ago
      Ah thanks for explaining. I misunderstood the root cause and didn't read the patch. Rust definitely would have helped here. Or even just enforcing modern C practices such as overwriting the whole struct so that non-specified values would have been set to zero (although explicit is better than zero).
parmezan 5y ago
It has been less than a month after fixes emerged for kernels and your PoC exploit has already been released into the public. Should you not have waited at least a bit longer (for example 2 months) before disclosing this vulnerability so that people/companies can keep up with patching? Don't they need more time to patch their servers and legacy etc before this becomes yet another log4j exploitation fest? That is if this really is the new dirty cow vuln. I get responsible disclosure is important, but should we not give people some more opportunity to patch, which will always take some time? Just curious. Also, nice work and interesting find!
- nickelpro 5y ago
  Once the commit is in the kernel tree it's effectively public for those looking to exploit it. Combing recent commits for bug fixes for the platform you're targeting is exploitation 101. The announcement only serves to let the rest of the public know about this and incentivize them to upgrade.
- staticassertion 5y ago
  It's the absolute opposite. It's insane that this commit wasn't flagged as a patch for a major vulnerability. Why am I finding out about this now? Why is it now my job to comb through commits looking for hidden patches? It puts me, as a defender, at an insane disadvantage. Attackers have the time, incentives, and skills to look at commits for vulns. I don't. I don't get paid for every commit I look at, I don't get value out of it. This backwards process pushed by Greg KH and others upstream needs to die ASAP.
  - weberer 5y ago
    Personally, I just enable automatic security updates and forget about it.
- amluto 5y ago
  Max did everything right here, and in this case I’m not sure the distribution process exists to have done better. (Thanks Max for handling this well and politely and for putting up with everyone’s conflicting opinions.)
MayeulC 5y ago
Wouldn't this allow modifying a cached version of /sbin/su to nop the password check? This seems really easy to exploit for privilege escalation.
- max_k 5y ago
  Yes. But you can also inject code into libc.so.6, and all running processes will have it.
  - staticassertion 5y ago
    Or /etc/passwd
- freemint 5y ago
  Yes it would. That is implied because writing arbitrary files means you can also edit the permission systems
blinkingled 5y ago
The offending commit was authored by Christoph Hellwig and possibly reviewed by Al Viro both of whom combined are close to 100% of Linux filesystems and VFS knowledge. Point being with the level of complexity you're just going to live the fact that they'll always be bugs. VFS/Page Cache/FS layers represent incredible complexity and cross dependencies - but the good news is code is very mature by now and should not see changes like this too often.
- dncornholio 5y ago
  And the reason for the commit was to have 'nicer code'. The code was working perfectly fine before someone decided it was not nice enough?
  - max_k 5y ago
    Your post sounds like it's a bad thing, but "nicer" code is easier to maintain, i.e. there will be fewer bugs (and fewer vulnerabilities). This bug is an exception of the rule - shit happens. But refactoring code to be "nicer" prevents more bugs than it causes. Two patches were involved in making this bug happen, and minus the bug, I value both of them (and their authors).
    - dncornholio 5y ago
      I might have sounded harsh but I think shit happens is not the way to look at this. Don't claim I'm a better developer, but I always try to shy away from making things look nicer. Experience have thought me, deal with problems when it is a problem. Dealing with could be problems can be a deep, very deep rabbit hole. The commit message gave me the feeling that we should have just trust the author. https://github.com/torvalds/linux/commit/f6dd975583bd8ce088400648fd9819e4691c8958 https://github.com/torvalds/linux/commit/f6dd975583bd8ce0884...
      - max_k 5y ago
        There's no bug in that commit, the commit is correct, it only makes the bug exploitable. The buggy commit is older, it's https://github.com/torvalds/linux/commit/241699cd72a8489c9446ae3910ddd243e9b9061b https://github.com/torvalds/linux/commit/241699cd72a8489c944... but not exploitable. > I always try to shy away from making things look nicer That's understandable, though from my experience, lots of old bugs can be found while refactoring code, even at the (small) risk of introducing new bugs.
gaius_baltar 5y ago
Fix was already merged to Android, however, there are millions of devices that will never be updated. The nice question: can this be used for temp-rooting? Vulnerabilities can be a blessing sometimes...
- max_k 5y ago
  Yes. I have a working exploit, but havn't published it (yet).
- rfoo 5y ago
  > there are millions of devices that will never be updated Luckily, almost all (if not just all) these millions of devices which will never be updated never ever received the vulnerable version in the first place. The bug was only introduced in 5.8 and due to how hardware vendors work phones are still stuck in 4.19 ages (or better, 5.4. but no 5.10 besides Pixel 6)
- _rdvw 5y ago
  I maintain a ROM for primarily older devices, the big feature is automated kernel CVE patching. My patcher was able to patch the 15 affected devices I support, and I'll have builds up in the next few days. https://gitlab.com/divested-mobile/divestos-build/-/commit/54dbcd9e43ed5106fdeacbd63a709438dc31a677 https://gitlab.com/divested-mobile/divestos-build/-/commit/5...
- WhyNotHugo 5y ago
  > The nice question: can this be used for temp-rooting? Vulnerabilities can be a blessing sometimes... Based on the description, sounds like it should be quite possible.
bananabiscuit 5y ago
I’m curious how git bisect was applied here. Wouldn’t you have to compile the whole kernel somehow and then run your test program using that kernel? Is that really what was done here?
- CasperDern 5y ago
  The kernel is relatively easy to compile and install, so I would think that's exactly what they did.
- rfoo 5y ago
  Yes? This is faster and easier than you may think it to be. Building a reasonably small kernel only takes ~a minute. People usually have fully automated git-bisect-run scripts for build & test in qemu.
  - bananabiscuit 5y ago
    Oh, interesting, did not know it could be so fast.
    - gengkev 5y ago
      For me, at least, there's an important difference missing from the debate over the term "C/C++": compiling C code is always much faster than you would expect, but compiling C++ code is always much slower than you would expect...
      - db48x 5y ago
        And yet there's an ongoing effort to optimize the kernel compile time by rearranging all of the headers. On a modern machine with plenty of cores a kernel build is pretty quick, but they're talking about slicing 20% or more off the top. It's always slower than we'd like.
    - cjbprime 5y ago
      Perhaps more importantly than being fast, it is scriptable. ("git bisect run" can take a shell command to run and interpret the exit code of, so you could script everything including the kernel recompiles and walk away for a few hours.)
sublimefire 5y ago
An example that needs to be in the textbooks. A detailed explanation and a timeline along with the code snippets. It succinctly shows you the complexities involved. Kudos to Max for putting it all into the post. > Blaming the Linux kernel (i.e. somebody else’s code) for data corruption must be the last resort. ^^^ I can only image the stress levels at this point.
- xcambar 5y ago
  The magic for me was the two little C programs that demonstrated the bug. Circa 10 lines of C. Beautiful.
  - rmetzler 5y ago
    Yes, being able to replicate the issue in a small piece of code is a very good thing. First it helps to triage whether the bug is in your own code or in other code. You also then can use it like the author and find the commit which introduced the bug. And you can use it as a test case to verify if the bug is closed.
- moltke 5y ago
  I've personally found bugs in unpopular kernel APIs. I spent days thinking it was my code until I went and read the Linux implementation.
  - girvo 5y ago
    ESP-IDF has so many bugs that it’s often the first thing to blame when we hit issues, even if it is our code after all haha
- misnome 5y ago
  Yes, this is an extremely well written and to the point writeup.
  - deutschew 5y ago
    usually I dont read too deeply into CVE because they are too complex but this article made me go holy sh- wish more would be written like this
jwilk 5y ago
> A ZIP file is just a container for .gz files That doesn't sound right.
- greyface- 5y ago
  Both PKZIP and gzip use DEFLATE: https://en.wikipedia.org/wiki/Deflate https://en.wikipedia.org/wiki/Deflate
- jl6 5y ago
  Yeah, a gzip file is itself a container for a DEFLATE stream. Gzip files can contain metadata such as timestamps, and comments.
- zenexer 5y ago
  GZIP (.gz) and PKZIP (.zip) are both containers for DEFLATE. GZIP is barely a container with minimal metadata, whereas PKZIP supports quite a bit of metadata. Although you can’t quite concatenate GZIP streams to get a PKZIP file, it’s pretty close—if I recall correctly, you just chop off the GZIP header.
  - zenexer 5y ago
    I'm past the edit period, but: > if I recall correctly, you just chop off the GZIP header. ...to get the raw DEFLATE stream, that is. You still need to attach any necessary metadata for PKZIP, which Max mentions. Their approach for converting between the two is pretty clever: it's so elegant and simple that it seems obvious, but I never would have thought of it. Very nifty, @max_k!
staticassertion 5y ago
Another example of a vulnerability that is purposefully obfuscated in the commit log. It is an insane practice that needs to die. The Linux kernel maintainers have been doing this for decades and it's now a standard practice for upstream. This gives attackers an advantage (they are incentivized to read commits and can easily see the vuln) and defenders a huge disadvantage. Now I have to rush to patch whereas attackers have had this entire time to build their POCs and exploit systems. End this ridiculous practice.
- amluto 5y ago
  Do you have actual evidence of that in a case like this? (This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objections won’t help.)
  - staticassertion 5y ago
    Evidence of what, exactly? I can find you lots of evidence for hiding vulns, they don't even hide it - I'm sure Greg will admit to as much. Evidence of this being helpful to attackers and not defenders? IDK, talk to anyone who does Linux kernel exploit development. edit: There you go, Greg linked his policy, which explicitly notes this.
  - rfoo 5y ago
    Not OP, but please do try to influence this policy if you can: 1. The commit message [1] does not mention any security implication. This is reasonable, because the patch is usually released to the public earlier and it makes sense to do some obfuscation, to deter patch-gappers. But note that this approach is not a controversy-free one. 2. But there is also no security announcement in stable release notes or any similar stuff. I don't know how to provide evidence of "something simply does not exist". 3. Check the timeline in the blog post. The bug being fixed in stable release (5.6.11 on 2022-02-23) marks the end of upstream's handling of this bug. Max then had to send the bug details to linux-distros list to kick off (another separate process) distro maintainers' response. If what you are maintaining is not a distro, good luck. Is this wrong-sounding enough? [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9d2231c5d74e13b2a0546fee6737ee4446017903 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
itvision 5y ago
This if f*cking scary. Such a simple code, so dangerous and it works. You can trivially add an extra root user via /etc/{passwd|shadow}. There are tons of options how to p0wn a system. Please update your devices ASAP!
- pabs3 5y ago
  Those unsupported devices probably don't run Linux 5.8 or later, they are likely on older versions. It would be really useful to have this vuln on them though, it would help with getting root so you can get control of your own device and install your own choice of OS.
  - itvision 5y ago
    You're right, I though kernel 5.8 is a lot older than it actually is. I've edited my post. Sorry!
- lazide 5y ago
  Eh, it’s a limited subset of kernel versions (ones unlikely to be used in those devices), and requires local execution privileges and access to the file system. Linux in general has had numerous security issues (as has every other OS), often requiring far less access. Does it need patching? Of course. It’s not a privilege escalation remote code execution issue though, and even if it was, it would be on a tiny fraction of running devices right now.
  - itvision 5y ago
    > and even if it was, it would be on a tiny fraction of running devices right now. That's correct and I misjudged the situation. Sorry!
- deleted 5y ago
  [deleted]
orwin 5y ago
Does this have a cvss yet? It seems really powerful and easy to exploit. And by easy to exploit I'm talking beginner CTF easy.
ZYinMD 5y ago
I think you're quite gifted in story telling, you could be thriller book writer.
reasonabl_human 5y ago
Crazy. Just successfully pwnd my homelab box in the garage.. Exciting for the implications of opening many locked down consumer devices out there. Nightmare for the greater cyber sec world...
alanbernstein 5y ago
Dirty pipe.. how about "sewerbleed"?
- mjevans 5y ago
  The exploit involves DIRTY (should be written back to disk) memory pages attached to a PIPE between processes.
  - alanbernstein 5y ago
    Yes... sewers are dirty pipes. "sewerbleed" is funnier than "dirty pipe", and it matches https://en.wikipedia.org/wiki/Heartbleed https://en.wikipedia.org/wiki/Heartbleed.
    - mjevans 5y ago
      We all got the reference you were making; the problem is 'heart' 'bleed' is based around what could be considered the heart (rather than as is normally said, the brain) of a computer 'bleeding' data from one context to another. In both cases the researchers chose sort of punny names that were also self descriptive and obvious once you read how to produce the exploit. 'Dirty Pipe' is literally the recipe for this exploit / corruption. Maybe your name seems funny to you for some reason that isn't obvious / shared.
qwertox 5y ago
What a poster child. Deserves some kind of award.
- pantalaimon 5y ago
  I like how they casually mention that they have basically written their entire stack themselves.
cryptonector 5y ago
There was a never-shipped bug in Solaris back around.. I want to say 2006? I don't remember exactly when, but there was a bug where block writes in a socketpair pipe could get swapped. I ended up writing a program that wrote entire blocks where each block was a repeated block counter, that way I could look for swapped blocks, and then also use that for the bug report. The application that was bit hard by this bug was ssh. Writing [repeated, if needed] monotonically increasing counters like this is a really good testing technique.
Dowwie 5y ago
>Memory bandwidth is saved by employing the splice() system call to feed data directly from the hard disk into the HTTP connection, without passing the kernel/userspace boundary (“zero-copy”). What are the memory savings of this splicing approach as compared to streaming [through userspace]?
- db48x 5y ago
  67% savings. If an application reads a 1MB file the normal way, the kernel creates 1MB of buffers in the file system cache to hold the data. Then it copies the data to another 1MB of buffers which are owned by the application. If the application then writes the data out to a network socket, the kernel has to allocate another 1MB buffer to hold the data while it is being sent. If the application were processing the data in some way, then it would be worth it. Otherwise it is better to skip all of that work.
- max_k 5y ago
  What does "streaming buffers" mean? splice() avoids copying data from kernel to userspace and back; it stays in the kernel, and often isn't even copied at all, only page references are passed around.
DonHopkins 5y ago
Once I fell victim to The Dirty Bong Vulnerability, when the cat knocked the bong over onto my Dell laptop's keyboard. Fortunately I had the extended warranty, and the nice repairwoman just smelled it, laughed at me, and cheerfully replaced the keyboard for free. No way Apple would have ever done that.
cookiengineer 5y ago
Amazing write-up! This is a super example of a responsible disclosure. I mean, compiling 17 kernels alone takes so long that most people would've given up in between.
- mort96 5y ago
  Depends entirely on what sort of hardware you have. IIRC, I usually spend around 5 minutes when compiling Linux on my desktop, so not instant but not horrible. The agonizing part would be to have to manually install, boot and test those kernels, or to create a setup involving virtual machines which does that automatically to use `git bisect run`. But yeah, incredibly impressive persistence.
- Unklejoe 5y ago
  > most people would've given up in between Nah, that's the most fun part. Once you have one kernel that works and one that doesn't, you can be pretty sure that you'll eventually find the cause of the bug. The part where I would have given up is the "trying to reproduce" part.
  - bombcar 5y ago
    Yeah, 17 loops is a small enough number that is honestly probably not bother figuring out how to automate git bisect and a test. The real deal was tracking it down and creating a reproducible test.
AviationAtom 5y ago
Since so many distros seem to lag a good ways behind on packages, and this vulnerability (in it's easiest exploited form) was introduced in kernel 5.8, it would seem a fair amount of Linux installs wouldn't actually be vulnerable to this. Is that somewhat correct?
- bombcar 5y ago
  Yes - ish. Depends on the distro. Ubuntu 20.04 has 5.4, for example, and I suspect many use that.
aetherspawn 5y ago
The sort of bug that could have been caught by unit tests I suppose.
mltony 5y ago
10 years ago I found even more outrageous bug in Windows 8. I was working in MSFT back than and I was writing a tool that produced 10 GB of data in TSV format , that I wanted to stream into gzip so that later this file would be sent over the network. When the other side received the file they would gunzip it successfully, but inside there would be mostly correct TSV data with some chunks of random binary garbage. Turned out that pipe operator was somehow causing this. As a responsible citizen I tried to report it to the right team and there I ran into problems. Apparently no one in Windows wants to deal with bugs. IIt was ridiculously hard to report this while being an employee, I can't imagine anyone being able to report similar bugs from outside. And even though I reported that bug I saw no activity in it when I was leaving the company. However I just tried to quickly reproduce it on Windows 10 and it wouldn't reproduce. Maybe I forgot some details of that bug or maybe indeed they fixed this by now.
- wombat-man 5y ago
  Worked there too at one point. It can be a struggle to find the right feature team. Once you do, if you can get it triaged, unless it’s high sev high priority it’s getting kicked to the next time period. Glad it looks like they got around to it though.
- Terry_Roll 5y ago
  > Maybe I forgot some details of that bug or maybe indeed they fixed this by now. There are lots of things which have been fixed in Windows 10, I'd go so far to say 1903 (19H1) is where things started to settle down, but even the latest versions are not perfect. When the Israeli/Palestinian conflict broke out in 2019, some of the US military computers started playing up for about a week, after the US vetoed something at the UN level regarding this conflict. So MS still has a long way to go to get things secure.
dcow 5y ago
The sample code does not demonstrate how to get the page to be flagged dirty so that the kernel actually writes it back to disk. Did I miss something?
- bombcar 5y ago
  I assume you’d trigger a write some other way - if using this to mess with the shadow file, say, change your password at the same time to flush the file.
egberts1 5y ago
Reminds me of SunOS 4.1.3 where you simply type in ‘+’ about 127 times at the “Login:” prompt and PRESTO-CHANGO … you get a root shell prompt.
- bill_mcgonigle 5y ago
  Who says closed source doesn't have benefits?
bill_mcgonigle 5y ago
Thank you for not selling this to the "industry"!
legalcorrection 5y ago
>Let me briefly introduce how our log server works: In the CM4all hosting environment, all web servers (running our custom open source HTTP server) send UDP multicast datagrams with metadata about each HTTP request. These are received by the log servers running Pond, our custom open source in-memory database. A nightly job splits all access logs of the previous day into one per hosted web site, each compressed with zlib. Via HTTP, all access logs of a month can be downloaded as a single .gz file. Using a trick (which involves Z_SYNC_FLUSH), we can just concatenate all gzipped daily log files without having to decompress and recompress them, which means this HTTP request consumes nearly no CPU. Memory bandwidth is saved by employing the splice() system call to feed data directly from the hard disk into the HTTP connection, without passing the kernel/userspace boundary (“zero-copy”). Windows users can’t handle .gz files, but everybody can extract ZIP files. A ZIP file is just a container for .gz files, so we could use the same method to generate ZIP files on-the-fly; all we needed to do was send a ZIP header first, then concatenate all .gz file contents as usual, followed by the central directory (another kind of header). Just want to say, these people are running a pretty impressive operation. Very thoroughly engineered system they have there.

697 points by max_k 5y ago

This affects kernels from 5.8 and was fixed in 5.16.11, 5.15.25 and 5.10.102. Exploit code is public and available on the linked page.

raesene9 5y ago
< 5.8 not being affected is probably a saving grace for quite a few enterprises as I'd expect that LTS distributions may not have got that version included as yet.
- gchamonlive 5y ago
  CentOS 7 is already at 5.10 so it should affect lots of production systems
  - emrvb 5y ago
    *blinks* *stares at kernel-3.10.0-1160.59.1.el7*
    - samus 5y ago
      Is this a smartphone? I'm on 3.18!
  - LinuxBender 5y ago
    Are you by chance using a 3rd party kernel repo such as ElRepo to work around a limitation? Or could someone at your org be compiling a custom kernel?
greyface- 5y ago
Debian stable (bullseye) is still vulnerable: https://security-tracker.debian.org/tracker/CVE-2022-0847 https://security-tracker.debian.org/tracker/CVE-2022-0847
- deng 5y ago
  That page is not up-to-date, fix was released today: https://lists.debian.org/debian-security-announce/2022/msg00059.html https://lists.debian.org/debian-security-announce/2022/msg00...
  - greyface- 5y ago
    It wasn't available via `apt-get update && apt-get dist-upgrade` as of when I drafted that comment, but I confirm that 5.10.92-2 seems to be released now.
    - deng 5y ago
      Well the fix was released ~30 minutes ago, so that checks out. ;-) The security-tracker site is now updated as well.
    - bill_mcgonigle 5y ago
      Others, note that the new archive name is 'stable-security'. You might need to update your pins if you upgraded from Buster and you're not seeing the update now. I put in a pull request to add it to the release notes.
baggy_trough 5y ago
How about Ubuntu?
- zenexer 5y ago
  The relevant CVE page returns a 500 error: https://github.com/canonical-web-and-design/ubuntu.com/issues/11324 https://github.com/canonical-web-and-design/ubuntu.com/issue... 21.10 appears to be lacking the patch.
  - baggy_trough 5y ago
    The CVE page returns now, with a whole bunch of "needs triage". https://ubuntu.com/security/CVE-2022-0847 https://ubuntu.com/security/CVE-2022-0847
baggy_trough 5y ago
It's disturbing that despite prior disclosure on distro lists, Ubuntu doesn't have an update available, with public exploits circulating now.
- deleted 5y ago
  [deleted]

abofh 5y ago

Wow, awesome debugging - very impressed.

egberts1 5y ago

Extreme Debugger Par Excellence! What a supérioritégrandeur!

kgraves 5y ago

Google's Fuchsia/Zircon cannot come fast enough.

amelius 5y ago
Because they use formal methods preventing this kind of thing from happening?
- kgraves 5y ago
  Even with your formal methods strawman, vulnerabilities like these are still possible in Linux and C. We need to move on.
  - freemint 5y ago
    You can formally verify C code against a spec though.
k4rli 5y ago
I wouldn't expect additional security from introducing an entirely new OS/kernel. Just unknown RCEs and other vulnerabilities waiting to be discovered.
- kgraves 5y ago
  Just like Linux so no change there, We still need to move on.
throwaway889900 5y ago
Why not seL4 then?

nickelpro 5y ago

Excellent work and excellent write up Max. A feather in your cap to be proud of for sure.

piratejon 5y ago

Wow, almost 10 months from the first reported file corruption until identification as an exploitable bug.

Ensorceled 5y ago
I'll bite, why the "Wow"? It was a random, intermittent file corruption that didn't cause real harm to the authors organization and was, clearly, very tricky to track down.
- piratejon 5y ago
  I don't have a basis for how long this might take. As the author mentions "All bugs become shallow once they can be reproduced.", but only after spending probably the largest amount of time waiting for new incident reports to come in, and then analyzing the reports (e.g. to determine most incidents occurred on the last day of the month), and hours staring at application and kernel code. It's very impressive, but certainly the largest amount of time in the 10 month duration was not actually debugging. The "moment of extraordinary clarity" probably sprung out of years of experience.
  - Ensorceled 5y ago
    Ah, I guess my thinking is that they didn't really focus on it. It was annoying but not high priority ... until they started to get an inkling of what was actually going on.
  - silverfox17 5y ago
    Agreed, about 99% of admins I know would not be able to identify this error, and most likely most Hacker News reads. The last sentence on your post is very true.
    - lazide 5y ago
      If not 99.999% I’ve worked with (and been) a dev for several decades, and I can count on one hand the number of folks who would have a chance of figuring this out, and 2 fingers the number of folks who WOULD. Of course, most never try to optimize or go so deep like this that they would ever need to, so there is that!

db48x 5y ago

C needs to die. Pro tip for language designers: require all fields to be initialized any time an object is created. Really impressive debugging too.

kevincox 5y ago
I love Rust, but would it have prevented this problem? IIUC there was no memory corruption at the language level here. This was really just a logic error.
- pdw 5y ago
  You can't accidentally leave a field of a struct uninitialized in Rust or in other sane languages.
- db48x 5y ago
  Yes, it would have. Some code creates an instance of some struct, but doesn’t set the flags field to zero. It thus keeps whatever value happened to be in that spot in memory, an essentially random set of bits. Rust would force you to either explicitly name the flags field and give it a value, or use `..Default::default()` to initialize all remaining fields automatically. Anything else would be a compile–time error. The fix: +++ b/lib/iov_iter.c @@ -414,6 +414,7 @@ static size_t copy_page_to_iter_pipe(struct page \*page, size_t offset, size_t by return 0; buf->ops = &page_cache_pipe_buf_ops; + buf->flags = 0; get_page(page); buf->page = page; buf->offset = offset;
  - amelius 5y ago
    Wouldn't Lint have caught the error too?
  - kevincox 5y ago
    Ah thanks for explaining. I misunderstood the root cause and didn't read the patch. Rust definitely would have helped here. Or even just enforcing modern C practices such as overwriting the whole struct so that non-specified values would have been set to zero (although explicit is better than zero).

parmezan 5y ago

It has been less than a month after fixes emerged for kernels and your PoC exploit has already been released into the public. Should you not have waited at least a bit longer (for example 2 months) before disclosing this vulnerability so that people/companies can keep up with patching? Don't they need more time to patch their servers and legacy etc before this becomes yet another log4j exploitation fest? That is if this really is the new dirty cow vuln. I get responsible disclosure is important, but should we not give people some more opportunity to patch, which will always take some time? Just curious. Also, nice work and interesting find!

nickelpro 5y ago
Once the commit is in the kernel tree it's effectively public for those looking to exploit it. Combing recent commits for bug fixes for the platform you're targeting is exploitation 101. The announcement only serves to let the rest of the public know about this and incentivize them to upgrade.
staticassertion 5y ago
It's the absolute opposite. It's insane that this commit wasn't flagged as a patch for a major vulnerability. Why am I finding out about this now? Why is it now my job to comb through commits looking for hidden patches? It puts me, as a defender, at an insane disadvantage. Attackers have the time, incentives, and skills to look at commits for vulns. I don't. I don't get paid for every commit I look at, I don't get value out of it. This backwards process pushed by Greg KH and others upstream needs to die ASAP.
- weberer 5y ago
  Personally, I just enable automatic security updates and forget about it.
amluto 5y ago
Max did everything right here, and in this case I’m not sure the distribution process exists to have done better. (Thanks Max for handling this well and politely and for putting up with everyone’s conflicting opinions.)

MayeulC 5y ago

Wouldn't this allow modifying a cached version of /sbin/su to nop the password check? This seems really easy to exploit for privilege escalation.

max_k 5y ago
Yes. But you can also inject code into libc.so.6, and all running processes will have it.
- staticassertion 5y ago
  Or /etc/passwd
freemint 5y ago
Yes it would. That is implied because writing arbitrary files means you can also edit the permission systems

blinkingled 5y ago

The offending commit was authored by Christoph Hellwig and possibly reviewed by Al Viro both of whom combined are close to 100% of Linux filesystems and VFS knowledge. Point being with the level of complexity you're just going to live the fact that they'll always be bugs. VFS/Page Cache/FS layers represent incredible complexity and cross dependencies - but the good news is code is very mature by now and should not see changes like this too often.

dncornholio 5y ago
And the reason for the commit was to have 'nicer code'. The code was working perfectly fine before someone decided it was not nice enough?
- max_k 5y ago
  Your post sounds like it's a bad thing, but "nicer" code is easier to maintain, i.e. there will be fewer bugs (and fewer vulnerabilities). This bug is an exception of the rule - shit happens. But refactoring code to be "nicer" prevents more bugs than it causes. Two patches were involved in making this bug happen, and minus the bug, I value both of them (and their authors).
  - dncornholio 5y ago
    I might have sounded harsh but I think shit happens is not the way to look at this. Don't claim I'm a better developer, but I always try to shy away from making things look nicer. Experience have thought me, deal with problems when it is a problem. Dealing with could be problems can be a deep, very deep rabbit hole. The commit message gave me the feeling that we should have just trust the author. https://github.com/torvalds/linux/commit/f6dd975583bd8ce088400648fd9819e4691c8958 https://github.com/torvalds/linux/commit/f6dd975583bd8ce0884...
    - max_k 5y ago
      There's no bug in that commit, the commit is correct, it only makes the bug exploitable. The buggy commit is older, it's https://github.com/torvalds/linux/commit/241699cd72a8489c9446ae3910ddd243e9b9061b https://github.com/torvalds/linux/commit/241699cd72a8489c944... but not exploitable. > I always try to shy away from making things look nicer That's understandable, though from my experience, lots of old bugs can be found while refactoring code, even at the (small) risk of introducing new bugs.

gaius_baltar 5y ago

Fix was already merged to Android, however, there are millions of devices that will never be updated. The nice question: can this be used for temp-rooting? Vulnerabilities can be a blessing sometimes...

max_k 5y ago
Yes. I have a working exploit, but havn't published it (yet).
rfoo 5y ago
> there are millions of devices that will never be updated Luckily, almost all (if not just all) these millions of devices which will never be updated never ever received the vulnerable version in the first place. The bug was only introduced in 5.8 and due to how hardware vendors work phones are still stuck in 4.19 ages (or better, 5.4. but no 5.10 besides Pixel 6)
_rdvw 5y ago
I maintain a ROM for primarily older devices, the big feature is automated kernel CVE patching. My patcher was able to patch the 15 affected devices I support, and I'll have builds up in the next few days. https://gitlab.com/divested-mobile/divestos-build/-/commit/54dbcd9e43ed5106fdeacbd63a709438dc31a677 https://gitlab.com/divested-mobile/divestos-build/-/commit/5...
WhyNotHugo 5y ago
> The nice question: can this be used for temp-rooting? Vulnerabilities can be a blessing sometimes... Based on the description, sounds like it should be quite possible.

bananabiscuit 5y ago

I’m curious how git bisect was applied here. Wouldn’t you have to compile the whole kernel somehow and then run your test program using that kernel? Is that really what was done here?

CasperDern 5y ago
The kernel is relatively easy to compile and install, so I would think that's exactly what they did.
rfoo 5y ago
Yes? This is faster and easier than you may think it to be. Building a reasonably small kernel only takes ~a minute. People usually have fully automated git-bisect-run scripts for build & test in qemu.
- bananabiscuit 5y ago
  Oh, interesting, did not know it could be so fast.
  - gengkev 5y ago
    For me, at least, there's an important difference missing from the debate over the term "C/C++": compiling C code is always much faster than you would expect, but compiling C++ code is always much slower than you would expect...
    - db48x 5y ago
      And yet there's an ongoing effort to optimize the kernel compile time by rearranging all of the headers. On a modern machine with plenty of cores a kernel build is pretty quick, but they're talking about slicing 20% or more off the top. It's always slower than we'd like.
  - cjbprime 5y ago
    Perhaps more importantly than being fast, it is scriptable. ("git bisect run" can take a shell command to run and interpret the exit code of, so you could script everything including the kernel recompiles and walk away for a few hours.)

sublimefire 5y ago

An example that needs to be in the textbooks. A detailed explanation and a timeline along with the code snippets. It succinctly shows you the complexities involved. Kudos to Max for putting it all into the post. > Blaming the Linux kernel (i.e. somebody else’s code) for data corruption must be the last resort. ^^^ I can only image the stress levels at this point.

xcambar 5y ago
The magic for me was the two little C programs that demonstrated the bug. Circa 10 lines of C. Beautiful.
- rmetzler 5y ago
  Yes, being able to replicate the issue in a small piece of code is a very good thing. First it helps to triage whether the bug is in your own code or in other code. You also then can use it like the author and find the commit which introduced the bug. And you can use it as a test case to verify if the bug is closed.
moltke 5y ago
I've personally found bugs in unpopular kernel APIs. I spent days thinking it was my code until I went and read the Linux implementation.
- girvo 5y ago
  ESP-IDF has so many bugs that it’s often the first thing to blame when we hit issues, even if it is our code after all haha
misnome 5y ago
Yes, this is an extremely well written and to the point writeup.
- deutschew 5y ago
  usually I dont read too deeply into CVE because they are too complex but this article made me go holy sh- wish more would be written like this

jwilk 5y ago

> A ZIP file is just a container for .gz files That doesn't sound right.

greyface- 5y ago
Both PKZIP and gzip use DEFLATE: https://en.wikipedia.org/wiki/Deflate https://en.wikipedia.org/wiki/Deflate
jl6 5y ago
Yeah, a gzip file is itself a container for a DEFLATE stream. Gzip files can contain metadata such as timestamps, and comments.
zenexer 5y ago
GZIP (.gz) and PKZIP (.zip) are both containers for DEFLATE. GZIP is barely a container with minimal metadata, whereas PKZIP supports quite a bit of metadata. Although you can’t quite concatenate GZIP streams to get a PKZIP file, it’s pretty close—if I recall correctly, you just chop off the GZIP header.
- zenexer 5y ago
  I'm past the edit period, but: > if I recall correctly, you just chop off the GZIP header. ...to get the raw DEFLATE stream, that is. You still need to attach any necessary metadata for PKZIP, which Max mentions. Their approach for converting between the two is pretty clever: it's so elegant and simple that it seems obvious, but I never would have thought of it. Very nifty, @max_k!

staticassertion 5y ago

Another example of a vulnerability that is purposefully obfuscated in the commit log. It is an insane practice that needs to die. The Linux kernel maintainers have been doing this for decades and it's now a standard practice for upstream. This gives attackers an advantage (they are incentivized to read commits and can easily see the vuln) and defenders a huge disadvantage. Now I have to rush to patch whereas attackers have had this entire time to build their POCs and exploit systems. End this ridiculous practice.

amluto 5y ago
Do you have actual evidence of that in a case like this? (This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objections won’t help.)
- staticassertion 5y ago
  Evidence of what, exactly? I can find you lots of evidence for hiding vulns, they don't even hide it - I'm sure Greg will admit to as much. Evidence of this being helpful to attackers and not defenders? IDK, talk to anyone who does Linux kernel exploit development. edit: There you go, Greg linked his policy, which explicitly notes this.
- rfoo 5y ago
  Not OP, but please do try to influence this policy if you can: 1. The commit message [1] does not mention any security implication. This is reasonable, because the patch is usually released to the public earlier and it makes sense to do some obfuscation, to deter patch-gappers. But note that this approach is not a controversy-free one. 2. But there is also no security announcement in stable release notes or any similar stuff. I don't know how to provide evidence of "something simply does not exist". 3. Check the timeline in the blog post. The bug being fixed in stable release (5.6.11 on 2022-02-23) marks the end of upstream's handling of this bug. Max then had to send the bug details to linux-distros list to kick off (another separate process) distro maintainers' response. If what you are maintaining is not a distro, good luck. Is this wrong-sounding enough? [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9d2231c5d74e13b2a0546fee6737ee4446017903 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

itvision 5y ago

This if f*cking scary. Such a simple code, so dangerous and it works. You can trivially add an extra root user via /etc/{passwd|shadow}. There are tons of options how to p0wn a system. Please update your devices ASAP!

pabs3 5y ago
Those unsupported devices probably don't run Linux 5.8 or later, they are likely on older versions. It would be really useful to have this vuln on them though, it would help with getting root so you can get control of your own device and install your own choice of OS.
- itvision 5y ago
  You're right, I though kernel 5.8 is a lot older than it actually is. I've edited my post. Sorry!
lazide 5y ago
Eh, it’s a limited subset of kernel versions (ones unlikely to be used in those devices), and requires local execution privileges and access to the file system. Linux in general has had numerous security issues (as has every other OS), often requiring far less access. Does it need patching? Of course. It’s not a privilege escalation remote code execution issue though, and even if it was, it would be on a tiny fraction of running devices right now.
- itvision 5y ago
  > and even if it was, it would be on a tiny fraction of running devices right now. That's correct and I misjudged the situation. Sorry!
deleted 5y ago
[deleted]

orwin 5y ago

Does this have a cvss yet? It seems really powerful and easy to exploit. And by easy to exploit I'm talking beginner CTF easy.

ZYinMD 5y ago

I think you're quite gifted in story telling, you could be thriller book writer.

reasonabl_human 5y ago

Crazy. Just successfully pwnd my homelab box in the garage.. Exciting for the implications of opening many locked down consumer devices out there. Nightmare for the greater cyber sec world...

alanbernstein 5y ago

Dirty pipe.. how about "sewerbleed"?

mjevans 5y ago
The exploit involves DIRTY (should be written back to disk) memory pages attached to a PIPE between processes.
- alanbernstein 5y ago
  Yes... sewers are dirty pipes. "sewerbleed" is funnier than "dirty pipe", and it matches https://en.wikipedia.org/wiki/Heartbleed https://en.wikipedia.org/wiki/Heartbleed.
  - mjevans 5y ago
    We all got the reference you were making; the problem is 'heart' 'bleed' is based around what could be considered the heart (rather than as is normally said, the brain) of a computer 'bleeding' data from one context to another. In both cases the researchers chose sort of punny names that were also self descriptive and obvious once you read how to produce the exploit. 'Dirty Pipe' is literally the recipe for this exploit / corruption. Maybe your name seems funny to you for some reason that isn't obvious / shared.

qwertox 5y ago

What a poster child. Deserves some kind of award.

pantalaimon 5y ago
I like how they casually mention that they have basically written their entire stack themselves.

cryptonector 5y ago

There was a never-shipped bug in Solaris back around.. I want to say 2006? I don't remember exactly when, but there was a bug where block writes in a socketpair pipe could get swapped. I ended up writing a program that wrote entire blocks where each block was a repeated block counter, that way I could look for swapped blocks, and then also use that for the bug report. The application that was bit hard by this bug was ssh. Writing [repeated, if needed] monotonically increasing counters like this is a really good testing technique.

Dowwie 5y ago

>Memory bandwidth is saved by employing the splice() system call to feed data directly from the hard disk into the HTTP connection, without passing the kernel/userspace boundary (“zero-copy”). What are the memory savings of this splicing approach as compared to streaming [through userspace]?

db48x 5y ago
67% savings. If an application reads a 1MB file the normal way, the kernel creates 1MB of buffers in the file system cache to hold the data. Then it copies the data to another 1MB of buffers which are owned by the application. If the application then writes the data out to a network socket, the kernel has to allocate another 1MB buffer to hold the data while it is being sent. If the application were processing the data in some way, then it would be worth it. Otherwise it is better to skip all of that work.
max_k 5y ago
What does "streaming buffers" mean? splice() avoids copying data from kernel to userspace and back; it stays in the kernel, and often isn't even copied at all, only page references are passed around.

DonHopkins 5y ago

Once I fell victim to The Dirty Bong Vulnerability, when the cat knocked the bong over onto my Dell laptop's keyboard. Fortunately I had the extended warranty, and the nice repairwoman just smelled it, laughed at me, and cheerfully replaced the keyboard for free. No way Apple would have ever done that.

cookiengineer 5y ago

Amazing write-up! This is a super example of a responsible disclosure. I mean, compiling 17 kernels alone takes so long that most people would've given up in between.

mort96 5y ago
Depends entirely on what sort of hardware you have. IIRC, I usually spend around 5 minutes when compiling Linux on my desktop, so not instant but not horrible. The agonizing part would be to have to manually install, boot and test those kernels, or to create a setup involving virtual machines which does that automatically to use `git bisect run`. But yeah, incredibly impressive persistence.
Unklejoe 5y ago
> most people would've given up in between Nah, that's the most fun part. Once you have one kernel that works and one that doesn't, you can be pretty sure that you'll eventually find the cause of the bug. The part where I would have given up is the "trying to reproduce" part.
- bombcar 5y ago
  Yeah, 17 loops is a small enough number that is honestly probably not bother figuring out how to automate git bisect and a test. The real deal was tracking it down and creating a reproducible test.

AviationAtom 5y ago

Since so many distros seem to lag a good ways behind on packages, and this vulnerability (in it's easiest exploited form) was introduced in kernel 5.8, it would seem a fair amount of Linux installs wouldn't actually be vulnerable to this. Is that somewhat correct?

bombcar 5y ago
Yes - ish. Depends on the distro. Ubuntu 20.04 has 5.4, for example, and I suspect many use that.

aetherspawn 5y ago

The sort of bug that could have been caught by unit tests I suppose.

mltony 5y ago

10 years ago I found even more outrageous bug in Windows 8. I was working in MSFT back than and I was writing a tool that produced 10 GB of data in TSV format , that I wanted to stream into gzip so that later this file would be sent over the network. When the other side received the file they would gunzip it successfully, but inside there would be mostly correct TSV data with some chunks of random binary garbage. Turned out that pipe operator was somehow causing this. As a responsible citizen I tried to report it to the right team and there I ran into problems. Apparently no one in Windows wants to deal with bugs. IIt was ridiculously hard to report this while being an employee, I can't imagine anyone being able to report similar bugs from outside. And even though I reported that bug I saw no activity in it when I was leaving the company. However I just tried to quickly reproduce it on Windows 10 and it wouldn't reproduce. Maybe I forgot some details of that bug or maybe indeed they fixed this by now.

wombat-man 5y ago
Worked there too at one point. It can be a struggle to find the right feature team. Once you do, if you can get it triaged, unless it’s high sev high priority it’s getting kicked to the next time period. Glad it looks like they got around to it though.
Terry_Roll 5y ago
> Maybe I forgot some details of that bug or maybe indeed they fixed this by now. There are lots of things which have been fixed in Windows 10, I'd go so far to say 1903 (19H1) is where things started to settle down, but even the latest versions are not perfect. When the Israeli/Palestinian conflict broke out in 2019, some of the US military computers started playing up for about a week, after the US vetoed something at the UN level regarding this conflict. So MS still has a long way to go to get things secure.

dcow 5y ago

The sample code does not demonstrate how to get the page to be flagged dirty so that the kernel actually writes it back to disk. Did I miss something?

bombcar 5y ago
I assume you’d trigger a write some other way - if using this to mess with the shadow file, say, change your password at the same time to flush the file.

egberts1 5y ago

Reminds me of SunOS 4.1.3 where you simply type in ‘+’ about 127 times at the “Login:” prompt and PRESTO-CHANGO … you get a root shell prompt.

bill_mcgonigle 5y ago
Who says closed source doesn't have benefits?

bill_mcgonigle 5y ago

Thank you for not selling this to the "industry"!

legalcorrection 5y ago

>Let me briefly introduce how our log server works: In the CM4all hosting environment, all web servers (running our custom open source HTTP server) send UDP multicast datagrams with metadata about each HTTP request. These are received by the log servers running Pond, our custom open source in-memory database. A nightly job splits all access logs of the previous day into one per hosted web site, each compressed with zlib. Via HTTP, all access logs of a month can be downloaded as a single .gz file. Using a trick (which involves Z_SYNC_FLUSH), we can just concatenate all gzipped daily log files without having to decompress and recompress them, which means this HTTP request consumes nearly no CPU. Memory bandwidth is saved by employing the splice() system call to feed data directly from the hard disk into the HTTP connection, without passing the kernel/userspace boundary (“zero-copy”). Windows users can’t handle .gz files, but everybody can extract ZIP files. A ZIP file is just a container for .gz files, so we could use the same method to generate ZIP files on-the-fly; all we needed to do was send a ZIP header first, then concatenate all .gz file contents as usual, followed by the central directory (another kind of header). Just want to say, these people are running a pretty impressive operation. Very thoroughly engineered system they have there.