8 ms·

To me, the biggest part of this story is: 1. Over two years ago, this was apparently detected automatically by the syzkaller kernel fuzzer, and automatically r

To me, the biggest part of this story is: 1. Over two years ago, this was apparently detected automatically by the syzkaller kernel fuzzer, and automatically reported on its public mailing list. [1] 2. Over a year and a half ago, it was apparently fixed in the upstream kernel. [2] 3. It was apparently never merged back to various "stable" kernels, leading to the recent CVE. [3] So you might read that and think "Ok, probably a rare mistake"... ...but instead: 4. This is apparently a _super_ common sequence of events, with kernel vulnerabilities getting lost in the shuffle, or otherwise not backported to "stable" kernels for a variety of reasons like the patch no cleanly longer applies. Dmitry Vyukov (original author of syzkaller fuzzer that found this 2 years ago) gave a very interesting talk on how frequently this happens a couple weeks ago at the Linux Maintainer's Summit, along with some discussion of how to change kernel dev processes to try to dramatically improve things: slides: https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf https://linuxplumbersconf.org/event/4/contributions/554/atta... video: https://youtu.be/a2Nv-KJyqPk?t=5239 https://youtu.be/a2Nv-KJyqPk?t=5239 --- [1] https://twitter.com/dvyukov/status/1180195777680986113 https://twitter.com/dvyukov/status/1180195777680986113 [2] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/drivers/android/binder.c?h=linux-4.14.y&id=7a3cee43e935b9d526ad07f20bf005ba7e74d05b https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux... [3] https://mobile.twitter.com/grsecurity/status/1180059539233804288 https://mobile.twitter.com/grsecurity/status/118005953923380...

yjftsjthsd-h 7y ago
So should we be fuzzing the stable branches separately?
- ebiggers 7y ago
  syzbot is already fuzzing the latest two stable kernels and has found hundreds of bugs, including lots of use-after-frees. All these bugs are listed here: - https://syzkaller.appspot.com/linux-4.14 https://syzkaller.appspot.com/linux-4.14 - https://syzkaller.appspot.com/linux-4.19 https://syzkaller.appspot.com/linux-4.19 As far as I know, no one is doing anything with the syzbot bugs against stable kernels directly, since no company using Linux is paying anyone to do it as their job. But some are getting fixed; e.g., some get reported against mainline too, then fixed and backported.
  - panpanna 7y ago
    How complex it is to test all known issues against all current kernels? A weekly report with some easy to understand graphs would probably convince more people to work on these bugs.
    - jsjohnst 7y ago
      Time and cost, same as it would be to do it across all kernel versions, not just current ones. Theoretically could be done pretty simply via a CI/CD pipeline if someone wrote solid test cases for the issues found by the fuzzier.
      - panpanna 7y ago
        Thats my point! Make this part of the kernel regression and also run it on the old kernels.
jumpingmice 7y ago
The Linux kernel is a glaring example software malfunction due to its combination of moderate defect density and incredible extent, along with a culture intolerant of competence. People who became subsystem maintainers because they happened to be hanging around a mailing list in the 90s are still gatekeepers of important subsystems despite their now-decades-long records of continuous malfeasance. Patches that demonstrably improve the health of the project are rejected if they would reduce the powers of these gatekeepers. We should look at the whole project as a cautionary tale of the kind of leveraged destruction that some programmers of modest ability but extreme confidence can wreak on our industry. It's bad enough that syzbot finds fifty serious bugs per hour, but I'll relay a personal anecdote. Earlier this year I wagered a colleague that I could open up the source of the 4.10 kernel (the one that was once current in Ubuntu 16) and find an obvious defect in less than an hour. It actually only took me about 15 minutes, to find a deadlock in the squashfs that was triggered by kmalloc failure and an error path via goto, which of course nobody should ever use. And while I'm reading it I'm just thinking to myself that this is the worst program I've ever seen and it would never pass a code review at my workplace, but it's out there right now running on billions of computers.
- dleslie 7y ago
  Care to name and shame with supporting evidence?
- Hello71 7y ago
  it's fairly well known that small kmallocs do not fail, and that there are many, many instances in the filesystem code which assume that small kmallocs do not fail. there have been two LWN articles on this exact subject.
  - girvo 7y ago
    And goto being used for error handling is pretty stock standard across most C codebases I’ve worked on or seen over the years, so I’m not sure what the particular gripe is there
    - 7y ago
edoo 7y ago
Could this be an issue of not appropriately identifying the impact of the bug? If it was reported by an automated tool and easy to fix perhaps the developer failed to fully investigate the problem, failing to realize it is a critical vulnerability and have the fix back ported.
- hannob 7y ago
  KASAN identified it as a use after free bug. That makes it very plausibly a security vuln. What more do you want to automate? The problem is there are so many of those.
dmix 7y ago
The failures of the Linux core team to properly prioritize security is quite well known. A lot of people have poked the bear by trying to bring this up, also with specific real examples, and got a tongue lashing from the team and moved on to other things. I'm amazed the GRSecurity people have managed to do it for so long. Even if merging their stuff mainline legitimately wasn't practical, I've seen plenty of snark and dismissiveness from the Linux team towards them and others. And GRSEC does actively bring in CVEs into their kernel patches all the time and get paid via sponsors to do so. I'm sure going through old CVEs is a great way to find "zero days" and/or relapses after old patches. Or even just following the work GRSec does there's probably plenty of stuff for a highly motivated company like NSO to exploit.
- telanis 7y ago
  GRSec is the primary reason the Android devices from BlackBerry have never, to my knowledge, been rooted (despite their many flaws). It's crazy that it's not more accepted.
  - dmix 7y ago
    I personally wouldn't trust a company that openly bragged it built a system to provide local police and Intel agencies with real time access to Blackberry messaging flowing across an entire city in 2010 for G20. In addition to sharing their "master" encryption key for a number of years: https://www.theverge.com/2016/4/14/11434926/blackberry-encryption-master-key-broken-canada-rcmp-surveillance https://www.theverge.com/2016/4/14/11434926/blackberry-encry... Also AFAIK Blackberry only provided a hardened kernel with a single device in 2015 called Priv. I haven't heard anything from them since... maybe someone could correct me here.
    - westmeal 7y ago
      The new Android devices also have hardened kernels but it doesn't really matter phones are insecure as fuck in other ways.
      - 7y ago
mister_hn 7y ago
The biggest problem is that instead of vendors submitting drivers for their devices in the mainline kernel and profit for all the fixes being done there, everyone makes a fork and there continues to work. Obviously, when merging updates from mainline kernel in the forked one, something is discarded or lost.
- clarry 7y ago
  > The biggest problem is that instead of vendors submitting drivers for their devices in the mainline kernel and profit for all the fixes being done there, everyone makes a fork The other half of the problem is the companies that actually use these garbage dump forks and build products on top of them. For me, getting the SoCs and chips we use running on latest upstream kernels was a high priority in platform bringup. I only used SoC vedors' garbage dump SDKs for quick testing & some reference. And chip vendors' drivers I ported straight to upstream git version. Of course this isn't how it goes in companies where "shit to market" is top priority.
makomk 7y ago
I'm pretty sure your undersnding is wrong. This was backported to all the applicable stable kernel versions almost two years ago. The problem is that it never made its way from those into the vendor-specific kernels that were actually shipping on mamy people's devices, because vendors are terrible.

yjftsjthsd-h 7y ago

So should we be fuzzing the stable branches separately?

ebiggers 7y ago
syzbot is already fuzzing the latest two stable kernels and has found hundreds of bugs, including lots of use-after-frees. All these bugs are listed here: - https://syzkaller.appspot.com/linux-4.14 https://syzkaller.appspot.com/linux-4.14 - https://syzkaller.appspot.com/linux-4.19 https://syzkaller.appspot.com/linux-4.19 As far as I know, no one is doing anything with the syzbot bugs against stable kernels directly, since no company using Linux is paying anyone to do it as their job. But some are getting fixed; e.g., some get reported against mainline too, then fixed and backported.
- panpanna 7y ago
  How complex it is to test all known issues against all current kernels? A weekly report with some easy to understand graphs would probably convince more people to work on these bugs.
  - jsjohnst 7y ago
    Time and cost, same as it would be to do it across all kernel versions, not just current ones. Theoretically could be done pretty simply via a CI/CD pipeline if someone wrote solid test cases for the issues found by the fuzzier.
    - panpanna 7y ago
      Thats my point! Make this part of the kernel regression and also run it on the old kernels.

jumpingmice 7y ago

The Linux kernel is a glaring example software malfunction due to its combination of moderate defect density and incredible extent, along with a culture intolerant of competence. People who became subsystem maintainers because they happened to be hanging around a mailing list in the 90s are still gatekeepers of important subsystems despite their now-decades-long records of continuous malfeasance. Patches that demonstrably improve the health of the project are rejected if they would reduce the powers of these gatekeepers. We should look at the whole project as a cautionary tale of the kind of leveraged destruction that some programmers of modest ability but extreme confidence can wreak on our industry. It's bad enough that syzbot finds fifty serious bugs per hour, but I'll relay a personal anecdote. Earlier this year I wagered a colleague that I could open up the source of the 4.10 kernel (the one that was once current in Ubuntu 16) and find an obvious defect in less than an hour. It actually only took me about 15 minutes, to find a deadlock in the squashfs that was triggered by kmalloc failure and an error path via goto, which of course nobody should ever use. And while I'm reading it I'm just thinking to myself that this is the worst program I've ever seen and it would never pass a code review at my workplace, but it's out there right now running on billions of computers.

dleslie 7y ago
Care to name and shame with supporting evidence?
Hello71 7y ago
it's fairly well known that small kmallocs do not fail, and that there are many, many instances in the filesystem code which assume that small kmallocs do not fail. there have been two LWN articles on this exact subject.
- girvo 7y ago
  And goto being used for error handling is pretty stock standard across most C codebases I’ve worked on or seen over the years, so I’m not sure what the particular gripe is there
  - 7y ago

edoo 7y ago

Could this be an issue of not appropriately identifying the impact of the bug? If it was reported by an automated tool and easy to fix perhaps the developer failed to fully investigate the problem, failing to realize it is a critical vulnerability and have the fix back ported.

hannob 7y ago
KASAN identified it as a use after free bug. That makes it very plausibly a security vuln. What more do you want to automate? The problem is there are so many of those.

dmix 7y ago

The failures of the Linux core team to properly prioritize security is quite well known. A lot of people have poked the bear by trying to bring this up, also with specific real examples, and got a tongue lashing from the team and moved on to other things. I'm amazed the GRSecurity people have managed to do it for so long. Even if merging their stuff mainline legitimately wasn't practical, I've seen plenty of snark and dismissiveness from the Linux team towards them and others. And GRSEC does actively bring in CVEs into their kernel patches all the time and get paid via sponsors to do so. I'm sure going through old CVEs is a great way to find "zero days" and/or relapses after old patches. Or even just following the work GRSec does there's probably plenty of stuff for a highly motivated company like NSO to exploit.

telanis 7y ago
GRSec is the primary reason the Android devices from BlackBerry have never, to my knowledge, been rooted (despite their many flaws). It's crazy that it's not more accepted.
- dmix 7y ago
  I personally wouldn't trust a company that openly bragged it built a system to provide local police and Intel agencies with real time access to Blackberry messaging flowing across an entire city in 2010 for G20. In addition to sharing their "master" encryption key for a number of years: https://www.theverge.com/2016/4/14/11434926/blackberry-encryption-master-key-broken-canada-rcmp-surveillance https://www.theverge.com/2016/4/14/11434926/blackberry-encry... Also AFAIK Blackberry only provided a hardened kernel with a single device in 2015 called Priv. I haven't heard anything from them since... maybe someone could correct me here.
  - westmeal 7y ago
    The new Android devices also have hardened kernels but it doesn't really matter phones are insecure as fuck in other ways.
    - 7y ago

mister_hn 7y ago

The biggest problem is that instead of vendors submitting drivers for their devices in the mainline kernel and profit for all the fixes being done there, everyone makes a fork and there continues to work. Obviously, when merging updates from mainline kernel in the forked one, something is discarded or lost.

clarry 7y ago
> The biggest problem is that instead of vendors submitting drivers for their devices in the mainline kernel and profit for all the fixes being done there, everyone makes a fork The other half of the problem is the companies that actually use these garbage dump forks and build products on top of them. For me, getting the SoCs and chips we use running on latest upstream kernels was a high priority in platform bringup. I only used SoC vedors' garbage dump SDKs for quick testing & some reference. And chip vendors' drivers I ported straight to upstream git version. Of course this isn't how it goes in companies where "shit to market" is top priority.

makomk 7y ago

I'm pretty sure your undersnding is wrong. This was backported to all the applicable stable kernel versions almost two years ago. The problem is that it never made its way from those into the vendor-specific kernels that were actually shipping on mamy people's devices, because vendors are terrible.