8 ms·

Another example of a vulnerability that is purposefully obfuscated in the commit log. It is an insane practice that needs to die. The Linux kernel maintainers h

Another example of a vulnerability that is purposefully obfuscated in the commit log. It is an insane practice that needs to die. The Linux kernel maintainers have been doing this for decades and it's now a standard practice for upstream. This gives attackers an advantage (they are incentivized to read commits and can easily see the vuln) and defenders a huge disadvantage. Now I have to rush to patch whereas attackers have had this entire time to build their POCs and exploit systems. End this ridiculous practice.

amluto 5y ago
Do you have actual evidence of that in a case like this? (This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objections won’t help.)
- staticassertion 5y ago
  Evidence of what, exactly? I can find you lots of evidence for hiding vulns, they don't even hide it - I'm sure Greg will admit to as much. Evidence of this being helpful to attackers and not defenders? IDK, talk to anyone who does Linux kernel exploit development. edit: There you go, Greg linked his policy, which explicitly notes this.
- rfoo 5y ago
  Not OP, but please do try to influence this policy if you can: 1. The commit message [1] does not mention any security implication. This is reasonable, because the patch is usually released to the public earlier and it makes sense to do some obfuscation, to deter patch-gappers. But note that this approach is not a controversy-free one. 2. But there is also no security announcement in stable release notes or any similar stuff. I don't know how to provide evidence of "something simply does not exist". 3. Check the timeline in the blog post. The bug being fixed in stable release (5.6.11 on 2022-02-23) marks the end of upstream's handling of this bug. Max then had to send the bug details to linux-distros list to kick off (another separate process) distro maintainers' response. If what you are maintaining is not a distro, good luck. Is this wrong-sounding enough? [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9d2231c5d74e13b2a0546fee6737ee4446017903 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
  - amluto 5y ago
    #1 is intentional, for better or for worse. It’s certainly well-intentioned too, although the intentions may be based on wrong assumptions. #2: upstream makes no general effort to identify security bugs as such. Obviously this one was known to be a security bug, but the general policy (see #1) is to avoid announcing it. #3: In any embargo situation, if you’re not on the distribution list, you don’t get notified. This is unavoidable. oss-security nominally handles everyone else, but it’s very spotty. Sometimes I wish there was a Linux kernel security advisory process, but this would need funding or a dedicated volunteer.
    - rfoo 5y ago
      TBH the thing annoyed me most in this story is the "Someone had to start the disclosure process on linux-distros again and if they didn't no one would know"-part. There are certainly silent bug fixes where the author intentionally (or not) does not post to linux-distros or any other maillists even after stable release. It would take an hour to dig a good example tho. (Okay, maybe 10 minutes if I'm going to read Brad Spengler's rants) I guess a Linux kernel security advisory process is needed to fix this, but yeah :(
      - amluto 5y ago
        For what it’s worth, linux-distros has its own opinions that are not necessarily compatible with those of the upstream kernel.
    - mjw1007 5y ago
      If only there was some kind of foundation with a revenue of $177 million last year which had an interest in Linux's success.
      - nisa 5y ago
        they are busy doing blockchain projects :)
    - sirdarckcat 5y ago
- rocqua 5y ago
  This is about the commit that fixed the bug, not the commit that introduced the bug. The accusation is not that linux developers intentionally introduced a vulnerability. Instead it is that linux developers hid that a commit fixed a vulnerability. Linux does this to prevent people from learning that the vulnerability exists.
  - wtarreau 5y ago
    > Linux does this to prevent people from learning that the vulnerability exists No, not at all, just to leave time to users to deploy the fix before everyone jumps on exploits. This is important because every single backported patch is a candidate for an exploit already, and it's only a matter of time before any of them is exploited. Reason why embargoes have to stay short. It takes some time to figure whether a bug may have security impacts. It takes much less time once this is figured, to develop an exploit. By the way it could have really happened that the fix for data corruption would have been merged first, and only later the author figured there was a security impact. And the patch wouldn't have been any different. That's why leaving 1-2 weeks for the fix to flow via distros to users, and having the author post a complete article is by far the best solution for everyone.
    - roddux 5y ago
      Nobody is arguing that users having a 1-2 week patch window is a bad thing. However, this frankly seems incompatible with open-source projects. Silently patching issues does not work in practice; it frequently leads to missed fixes, misapplied patches and other incompatibility woes. The situation with backports and LTS releases showcases this well— the only truly well-supported kernel is latest. Everything else is a patchwork of best-effort fixes, not all of which may have been applied correctly. Brad Spengler of grsecurity fame talks frequently about this (primarily via Twitter): https://twitter.com/spendergrsec https://twitter.com/spendergrsec
      - wtarreau 5y ago
        Not really. As you say there's an extremely difficult balance with opensource and not exposing everyone at once. You can't get a fix deployed everywhere without it being public first or it ends up in a total unfixable mess. But if the fix is public and gives too many info (exploit procedure) then you put everyone in danger until the fix flows to users. Thus the only solution is to have a public fix describing the bug and not necessarily all the details, while distros prepare their update, and everyone discloses the trouble at the same time. Those who need early notification MUST ABSOLUTELY BE on linux-distros. There's no other way around. As soon as the patch is published, the risk is non-nul and a race is started between those who look for candidate fixes and those who have to distribute fixes to end users. This is not about silently patching or hiding bugs, quite the opposite, it's about making them public as quickly as possible so that the fix can be picked, but without the unneeded elements that help vandals damage shared systems before these systems have a chance to be updated. Then it is useful that the reporter communicates about their finding, this often helps improve general security by documenting how certain classes of bugs turn to security issues (Max did an awesome job here, probably the best such bug report in the last few years). And distros need to publish more details as well in their advisories, so details are not "hidden", they're just delayed during tha embargo. Those who are not notified AND who do not follow stable are simply irresponsible. But I don't think there are that many doing that nowadays, possibly just a few hero admins in small companies trying to impress their boss with their collection of carefully selected patches (that render their machine even more vulnerable and tend to make them vocal when such issues happen). In addition it's important to keep in mind that some bugs are discovered as being exploitable long after being fixed. That's why one MUST ABSOLUTELY NOT rely on the commit message alone to decide whether they are vulnerable or not, since it's quite common not to know upfront. I remember a year or two ago someone from Google's security team reported a bug on haproxy that could cause a crash in the HPACK decoder. That was extremely embarrassing as it could allow anyone to remotely crash haproxy. We had to release the fix indicating that the bug was critical and that the risk of crashing when facing bad data was real, without explaining how to crash it (since like a kernel it's a components many people tend to forget to upgrade). Then after the fix was merged, I was still discussing with the reporter and asked "do you think it could further be abused for RCE?". He said "let me check". A week later he came back saying "good news, I succeeded". No way to get that info in the commit message even if we wanted to, since that was too late. Yet the issue was important. Speaking of Brad, I personally think that grsec ought to be on linux-distros, but maybe they prefer not to appear as "tainted" by early notifications, or maybe they're having some fun finding other issues themselves. We even proposed Brad to be on the security list, because he has the skills to help a lot and improve the security there. He could have interesting writeups for some of the bugs, and it would probably change his perception of what happens there. Maybe one day he'll accept (still keeping hope :-)).
gregkh 5y ago
I've described how we (the kernel security team) handles this type of things many times, and even summarized it in the past here: http://www.kroah.com/log/blog/2018/02/05/linux-kernel-release-model/ http://www.kroah.com/log/blog/2018/02/05/linux-kernel-releas... Scroll down to the section entitled "Security" for the details. If you wish to disagree with how we handle all of this, wonderful, we will be glad to discuss it on the mailing lists. Just don't try to rehash all the same old arguments again, as that's not going to work at all. Also, this was fixed in a public kernel last week, what prevented you from updating your kernel already? Did you need more time to test the last release? Edit: It was fixed in a public release 12 days ago.
- staticassertion 5y ago
  I'm well aware of your policy. Yes, I disagree. Also, mailing lists suck and I'll continue to comment wherever I please about the matter. > Just don't try to rehash all the same old arguments again, as that's not going to work at all. No shit. People have been trying to explain all of this to you for decades lol I'm not stupid enough to think I'll succeed where they've failed.
  - bombcar 5y ago
    What’s amusing is a few years back there WAS a security bug so critical that they DID handle it in a special and strange way.
    - gregkh 5y ago
      Yes, and because of that, we created a new process for those types of issues (i.e. broken hardware problems that need more coordination.) That process is documented at https://www.kernel.org/doc/html/latest/process/embargoed-hardware-issues.html https://www.kernel.org/doc/html/latest/process/embargoed-har... if you are curious. It's been working semi-well, and gives us a way to deal with longer embargo times (like months instead of weeks and days), but it does not integrate well into the linux-distro-like way of working just yet, which is an issue that hopefully will be resolved sometime in the future if the linux-distro members wish it to be.
bell-cot 5y ago
Attackers with the resources and patience to read and deeply analyze all the commits, over time... those guys were fairly likely to notice the bug back when it was introduced. Plain vs. obscure comments on the patch don't much matter to them. Low-resource and lower-skill attackers - "/* fix vuln. introduced in prior commit 123456789 */" could be quite useful to them.
- staticassertion 5y ago
  I don't think you understand how attackers work. Attackers don't just crawl code at random. Starting with known crashes or obfuscated commits is always faster.
  - 3np 5y ago
    There are both kinds, and also those who do both.
hvidgaard 5y ago
What should they do instead? You have to rush to patch in any case. If the maintainers start to label commits with "security patch" the logical step is that it doesn't require immediate action when the label is not there. Never mind that the bug might actually be exploitable but undiscovered by white hats. If you do not want to rush to patch more than you have to, use a LTS kernel and know that updates matter and should be applied asap regardless of the reason for the patch.
- staticassertion 5y ago
  > What should they do instead? When someone submits a patch for a vulnerability label the commit with that information. > You have to rush to patch in any case. The difference is how much of a head start attackers have. Attackers are incentivized to read commits for obfuscated vulns - asking defenders to do that is just adding one more thing to our plates. That's a huge difference. > the logical step is that it doesn't require immediate action when the label is not there. So I can go about my patch cycle as normal. > Never mind that the bug might actually be exploitable but undiscovered by white hats. OK? So? First of all, it's usually really obvious when a bug might be exploitable, or at least it would be if we didn't have commits obfuscating the details. Second, I'm not suggesting that you only apply security labeled patches.
  - roddux 5y ago
    Don't know why your other comment got downvoted. Silently patching bugs has left many LTS kernels vulnerable to old bugs, because they weren't tagged as security fixes. Also leads to other issues..: https://grsecurity.net/the_life_of_a_bad_security_fix https://grsecurity.net/the_life_of_a_bad_security_fix See also: https://twitter.com/spendergrsec https://twitter.com/spendergrsec
    - staticassertion 5y ago
      Not just downvoted. Flagged lol
  - sirdarckcat 5y ago
camgunz 5y ago
Can you say what you're hoping to do? LK devs tag security fixes with "[SECURITY]" and then what? You would merge individual [SECURITY] commits into your tree? Currently the situation is that you can just follow development/stable trees right (e.g. [0])? Why would you only want the security patches (of which there look to be a lot just in the last couple weeks). Are you looking to not apply a patch because LK devs haven't marked it as a security patch? [0]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=v5.10.103 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
- staticassertion 5y ago
  Assume I patch my Linux boxes once a month. I see a commit where an attacker has a trivial privesc. I read the commit, see if it's relevant to me, and potentially decide to do an out of cycle patch. As in, instead of updating next month I'll just update now.
  - camgunz 5y ago
    Gotcha. Yeah it does seem like there's some space between the overpromising "I am a Linux Kernel Dev and I proclaim this patch is/is not a security patch" and the underpromising "I am a Linux Kernel Dev and have no knowledge of whether or not this is a security patch". It doesn't seem unreasonable to mark it somehow when you know. On the other hand, just on that page I linked, there's... a lot of issues in there I would consider patching for security reasons. I don't know how reasonable it is, given the existing kernel development model, to tag this stuff in the commit. The LTS branches pull in from a lot of other branches, so like, which ones do you follow? When Vijayanand Jitta patches a UAF bug in their tree, it might be hanging out on the internet for a while for hackers to see before it ever gets into a kernel tree you might consider merging from. I guess what I'm saying here is that it seems like a lot to ask that if I find a bug, I: - don't discuss it publicly in any way - perform independent research to determine whether there are security implications - if there are, ask everyone else to keep the fix secret until it lands in the release trees with a [SECURITY] tag - accept all the blame if I'm ever wrong, even once That too is a lot of overhead and responsibility. So I'm sympathetic to their argument of "honestly, you should just assume these are all security vulns". So maybe this is just a perspective thing? Like, there are a lot of commits, they can't all be security issues right? Well of course they can be! This is C after all. Like in that list, there's dozens of things I think should probably have a SECURITY tag. Over 14 days, let's just call that 2 patches a day. I'm not patching twice a day; it's hard for me to imagine anyone would, or would want to devote mental bandwidth to getting that down to a manageable rate ("I don't run that ethernet card", etc.) So for me, I actually kind of like the weekly batching? It feels pragmatic and a pretty good balancing of kernel dev/sysadmin needs. Can I envision a system that gave end-users more information? Yeah definitely, but not one that wouldn't ask LK devs to do a lot more work. Which I guess is a drawn out way of saying "feel free to write your own OS" or "consider OpenBSD" or "get involved in Rust in the kernel" or "try to move safer/microkernel designs forward" :).
rocqua 5y ago
What is your threat model / situation that you care about attackers who reverse engineer patches, but are not in the small circle of people who would be informed before hand. To me, it seems like the average corporate security team is not going to worry about these kinds of attackers. Security for state secrets might, but they seem likely to be clued in early by Linux developers. I'm probably missing something tho.
- staticassertion 5y ago
  > What is your threat model / situation that you care about attackers who reverse engineer patches, but are not in the small circle of people who would be informed before hand. Virtually every single Linux user. I think what you're missing is how commonplace and straightforward it is for attackers to review these commits and how uncommon it is for someone to be on the receiving end of an embargo. Most exploits are for N days, meaning that they're for vulnerabilities that have a patch out for them. Knowing that there's a patch is universally critical for all defenders. For context, my company will be posting about a kernel (then) 0day one of our security researchers discovered. You can read other Linux kernel exploitation work we've done here: https://www.graplsecurity.com/blog https://www.graplsecurity.com/blog
  - rocqua 5y ago
    By threat model I mean, who are you worried about attacking you. I get that every linux user could be attacked. But why would someone with the relevant knowledge that could pull this off attack a given linux user? Why are you worried about it? (Not trying to be sarcastic, trying to get a sense of what threats you are worried about).
    - staticassertion 5y ago
      My point is that this is basically just how exploits work for Linux, so it's pretty universal unless your main concern is 0days. As for me personally, I run a company that uses Linux in production. We happen to explicitly do research into Linux kernel security (we'll be publishing tomorrow on a 0day we had reported) https://www.graplsecurity.com/blog https://www.graplsecurity.com/blog
titzer 5y ago
This is why stable branches are a thing. I don't know the branching scheme that the Linux kernel uses, but the idea is that for the oldest (most stable) branch, everything is a (sometimes backported) bugfix with security implications.
marbu 5y ago
Are you saying that you are able to read all incoming linux patches, and easily identify changes which fixes a security problem, so that you can come up with a POC by the time the security issue is announced? If the patch was flagged as a security problem from the beginning, it would give advantage to attackers, since they would know that the particular patch is worth investigating, while the defenders would have to wait for the patch to be finalized and tested anyway.
- staticassertion 5y ago
  You have it completely backwards.
- lmm 5y ago
  > Are you saying that you are able to read all incoming linux patches, and easily identify changes which fixes a security problem, so that you can come up with a POC by the time the security issue is announced? Their point is that a full-time attacker (and there's enough money in it to do it as a full-time job these days) can look for obfuscated commits and take the time to deobfuscate them, whereas a defender doesn't have that kind of time.
  - marbu 5y ago
    I agree, that is definitely possible. That said it requires lot of work, since there are lot of incoming patches. I wonder how many people would have to review every proposed patch, how to select subset of incoming patches for human review, and how much one have to pay a team doing all this, to get reasonable results and return of investment. My point was that if security patches are flagged as such from the start, it saves attackers lot of time (and money), as they will no longer have to go through (almost) every patch and evaluate whether it could be fixing a security problem. This means that such scenario will get a lot cheaper, while the defenders won't gain much from that, as one still needs to wait for the fix to be finalized and tested before deploying it in a production environment.
    - staticassertion 5y ago
bobthebuilders 5y ago
You have it the wrong way around. Tagging the release as security allows nation-state level attackers with large budgets to investigate the fixes, while normal people have to wait to patches. This gives nation-state level attackers with large budgets a heads-up, making it worse for everyone else. Furthermore, nation-state level attacks with large budgets are more focused on offense than defense.
- staticassertion 5y ago
  This comment is totally baseless. Anyone who does linux kernel exploit development knows how to crawl the commit log or syzkaller.
  - bobthebuilders 5y ago
    I'm sorry, but I'm not a nation-state. I wish I was though.

amluto 5y ago

Do you have actual evidence of that in a case like this? (This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objections won’t help.)

staticassertion 5y ago
Evidence of what, exactly? I can find you lots of evidence for hiding vulns, they don't even hide it - I'm sure Greg will admit to as much. Evidence of this being helpful to attackers and not defenders? IDK, talk to anyone who does Linux kernel exploit development. edit: There you go, Greg linked his policy, which explicitly notes this.
rfoo 5y ago
Not OP, but please do try to influence this policy if you can: 1. The commit message [1] does not mention any security implication. This is reasonable, because the patch is usually released to the public earlier and it makes sense to do some obfuscation, to deter patch-gappers. But note that this approach is not a controversy-free one. 2. But there is also no security announcement in stable release notes or any similar stuff. I don't know how to provide evidence of "something simply does not exist". 3. Check the timeline in the blog post. The bug being fixed in stable release (5.6.11 on 2022-02-23) marks the end of upstream's handling of this bug. Max then had to send the bug details to linux-distros list to kick off (another separate process) distro maintainers' response. If what you are maintaining is not a distro, good luck. Is this wrong-sounding enough? [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9d2231c5d74e13b2a0546fee6737ee4446017903 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- amluto 5y ago
  #1 is intentional, for better or for worse. It’s certainly well-intentioned too, although the intentions may be based on wrong assumptions. #2: upstream makes no general effort to identify security bugs as such. Obviously this one was known to be a security bug, but the general policy (see #1) is to avoid announcing it. #3: In any embargo situation, if you’re not on the distribution list, you don’t get notified. This is unavoidable. oss-security nominally handles everyone else, but it’s very spotty. Sometimes I wish there was a Linux kernel security advisory process, but this would need funding or a dedicated volunteer.
  - rfoo 5y ago
    TBH the thing annoyed me most in this story is the "Someone had to start the disclosure process on linux-distros again and if they didn't no one would know"-part. There are certainly silent bug fixes where the author intentionally (or not) does not post to linux-distros or any other maillists even after stable release. It would take an hour to dig a good example tho. (Okay, maybe 10 minutes if I'm going to read Brad Spengler's rants) I guess a Linux kernel security advisory process is needed to fix this, but yeah :(
    - amluto 5y ago
      For what it’s worth, linux-distros has its own opinions that are not necessarily compatible with those of the upstream kernel.
  - mjw1007 5y ago
    If only there was some kind of foundation with a revenue of $177 million last year which had an interest in Linux's success.
    - nisa 5y ago
      they are busy doing blockchain projects :)
  - sirdarckcat 5y ago
rocqua 5y ago
This is about the commit that fixed the bug, not the commit that introduced the bug. The accusation is not that linux developers intentionally introduced a vulnerability. Instead it is that linux developers hid that a commit fixed a vulnerability. Linux does this to prevent people from learning that the vulnerability exists.
- wtarreau 5y ago
  > Linux does this to prevent people from learning that the vulnerability exists No, not at all, just to leave time to users to deploy the fix before everyone jumps on exploits. This is important because every single backported patch is a candidate for an exploit already, and it's only a matter of time before any of them is exploited. Reason why embargoes have to stay short. It takes some time to figure whether a bug may have security impacts. It takes much less time once this is figured, to develop an exploit. By the way it could have really happened that the fix for data corruption would have been merged first, and only later the author figured there was a security impact. And the patch wouldn't have been any different. That's why leaving 1-2 weeks for the fix to flow via distros to users, and having the author post a complete article is by far the best solution for everyone.
  - roddux 5y ago
    Nobody is arguing that users having a 1-2 week patch window is a bad thing. However, this frankly seems incompatible with open-source projects. Silently patching issues does not work in practice; it frequently leads to missed fixes, misapplied patches and other incompatibility woes. The situation with backports and LTS releases showcases this well— the only truly well-supported kernel is latest. Everything else is a patchwork of best-effort fixes, not all of which may have been applied correctly. Brad Spengler of grsecurity fame talks frequently about this (primarily via Twitter): https://twitter.com/spendergrsec https://twitter.com/spendergrsec
    - wtarreau 5y ago
      Not really. As you say there's an extremely difficult balance with opensource and not exposing everyone at once. You can't get a fix deployed everywhere without it being public first or it ends up in a total unfixable mess. But if the fix is public and gives too many info (exploit procedure) then you put everyone in danger until the fix flows to users. Thus the only solution is to have a public fix describing the bug and not necessarily all the details, while distros prepare their update, and everyone discloses the trouble at the same time. Those who need early notification MUST ABSOLUTELY BE on linux-distros. There's no other way around. As soon as the patch is published, the risk is non-nul and a race is started between those who look for candidate fixes and those who have to distribute fixes to end users. This is not about silently patching or hiding bugs, quite the opposite, it's about making them public as quickly as possible so that the fix can be picked, but without the unneeded elements that help vandals damage shared systems before these systems have a chance to be updated. Then it is useful that the reporter communicates about their finding, this often helps improve general security by documenting how certain classes of bugs turn to security issues (Max did an awesome job here, probably the best such bug report in the last few years). And distros need to publish more details as well in their advisories, so details are not "hidden", they're just delayed during tha embargo. Those who are not notified AND who do not follow stable are simply irresponsible. But I don't think there are that many doing that nowadays, possibly just a few hero admins in small companies trying to impress their boss with their collection of carefully selected patches (that render their machine even more vulnerable and tend to make them vocal when such issues happen). In addition it's important to keep in mind that some bugs are discovered as being exploitable long after being fixed. That's why one MUST ABSOLUTELY NOT rely on the commit message alone to decide whether they are vulnerable or not, since it's quite common not to know upfront. I remember a year or two ago someone from Google's security team reported a bug on haproxy that could cause a crash in the HPACK decoder. That was extremely embarrassing as it could allow anyone to remotely crash haproxy. We had to release the fix indicating that the bug was critical and that the risk of crashing when facing bad data was real, without explaining how to crash it (since like a kernel it's a components many people tend to forget to upgrade). Then after the fix was merged, I was still discussing with the reporter and asked "do you think it could further be abused for RCE?". He said "let me check". A week later he came back saying "good news, I succeeded". No way to get that info in the commit message even if we wanted to, since that was too late. Yet the issue was important. Speaking of Brad, I personally think that grsec ought to be on linux-distros, but maybe they prefer not to appear as "tainted" by early notifications, or maybe they're having some fun finding other issues themselves. We even proposed Brad to be on the security list, because he has the skills to help a lot and improve the security there. He could have interesting writeups for some of the bugs, and it would probably change his perception of what happens there. Maybe one day he'll accept (still keeping hope :-)).

gregkh 5y ago

I've described how we (the kernel security team) handles this type of things many times, and even summarized it in the past here: http://www.kroah.com/log/blog/2018/02/05/linux-kernel-release-model/ http://www.kroah.com/log/blog/2018/02/05/linux-kernel-releas... Scroll down to the section entitled "Security" for the details. If you wish to disagree with how we handle all of this, wonderful, we will be glad to discuss it on the mailing lists. Just don't try to rehash all the same old arguments again, as that's not going to work at all. Also, this was fixed in a public kernel last week, what prevented you from updating your kernel already? Did you need more time to test the last release? Edit: It was fixed in a public release 12 days ago.

staticassertion 5y ago
I'm well aware of your policy. Yes, I disagree. Also, mailing lists suck and I'll continue to comment wherever I please about the matter. > Just don't try to rehash all the same old arguments again, as that's not going to work at all. No shit. People have been trying to explain all of this to you for decades lol I'm not stupid enough to think I'll succeed where they've failed.
- bombcar 5y ago
  What’s amusing is a few years back there WAS a security bug so critical that they DID handle it in a special and strange way.
  - gregkh 5y ago
    Yes, and because of that, we created a new process for those types of issues (i.e. broken hardware problems that need more coordination.) That process is documented at https://www.kernel.org/doc/html/latest/process/embargoed-hardware-issues.html https://www.kernel.org/doc/html/latest/process/embargoed-har... if you are curious. It's been working semi-well, and gives us a way to deal with longer embargo times (like months instead of weeks and days), but it does not integrate well into the linux-distro-like way of working just yet, which is an issue that hopefully will be resolved sometime in the future if the linux-distro members wish it to be.

bell-cot 5y ago

Attackers with the resources and patience to read and deeply analyze all the commits, over time... those guys were fairly likely to notice the bug back when it was introduced. Plain vs. obscure comments on the patch don't much matter to them. Low-resource and lower-skill attackers - "/* fix vuln. introduced in prior commit 123456789 */" could be quite useful to them.

staticassertion 5y ago
I don't think you understand how attackers work. Attackers don't just crawl code at random. Starting with known crashes or obfuscated commits is always faster.
- 3np 5y ago
  There are both kinds, and also those who do both.

hvidgaard 5y ago

What should they do instead? You have to rush to patch in any case. If the maintainers start to label commits with "security patch" the logical step is that it doesn't require immediate action when the label is not there. Never mind that the bug might actually be exploitable but undiscovered by white hats. If you do not want to rush to patch more than you have to, use a LTS kernel and know that updates matter and should be applied asap regardless of the reason for the patch.

staticassertion 5y ago
> What should they do instead? When someone submits a patch for a vulnerability label the commit with that information. > You have to rush to patch in any case. The difference is how much of a head start attackers have. Attackers are incentivized to read commits for obfuscated vulns - asking defenders to do that is just adding one more thing to our plates. That's a huge difference. > the logical step is that it doesn't require immediate action when the label is not there. So I can go about my patch cycle as normal. > Never mind that the bug might actually be exploitable but undiscovered by white hats. OK? So? First of all, it's usually really obvious when a bug might be exploitable, or at least it would be if we didn't have commits obfuscating the details. Second, I'm not suggesting that you only apply security labeled patches.
- roddux 5y ago
  Don't know why your other comment got downvoted. Silently patching bugs has left many LTS kernels vulnerable to old bugs, because they weren't tagged as security fixes. Also leads to other issues..: https://grsecurity.net/the_life_of_a_bad_security_fix https://grsecurity.net/the_life_of_a_bad_security_fix See also: https://twitter.com/spendergrsec https://twitter.com/spendergrsec
  - staticassertion 5y ago
    Not just downvoted. Flagged lol
- sirdarckcat 5y ago

camgunz 5y ago

Can you say what you're hoping to do? LK devs tag security fixes with "[SECURITY]" and then what? You would merge individual [SECURITY] commits into your tree? Currently the situation is that you can just follow development/stable trees right (e.g. [0])? Why would you only want the security patches (of which there look to be a lot just in the last couple weeks). Are you looking to not apply a patch because LK devs haven't marked it as a security patch? [0]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=v5.10.103 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...

staticassertion 5y ago
Assume I patch my Linux boxes once a month. I see a commit where an attacker has a trivial privesc. I read the commit, see if it's relevant to me, and potentially decide to do an out of cycle patch. As in, instead of updating next month I'll just update now.
- camgunz 5y ago
  Gotcha. Yeah it does seem like there's some space between the overpromising "I am a Linux Kernel Dev and I proclaim this patch is/is not a security patch" and the underpromising "I am a Linux Kernel Dev and have no knowledge of whether or not this is a security patch". It doesn't seem unreasonable to mark it somehow when you know. On the other hand, just on that page I linked, there's... a lot of issues in there I would consider patching for security reasons. I don't know how reasonable it is, given the existing kernel development model, to tag this stuff in the commit. The LTS branches pull in from a lot of other branches, so like, which ones do you follow? When Vijayanand Jitta patches a UAF bug in their tree, it might be hanging out on the internet for a while for hackers to see before it ever gets into a kernel tree you might consider merging from. I guess what I'm saying here is that it seems like a lot to ask that if I find a bug, I: - don't discuss it publicly in any way - perform independent research to determine whether there are security implications - if there are, ask everyone else to keep the fix secret until it lands in the release trees with a [SECURITY] tag - accept all the blame if I'm ever wrong, even once That too is a lot of overhead and responsibility. So I'm sympathetic to their argument of "honestly, you should just assume these are all security vulns". So maybe this is just a perspective thing? Like, there are a lot of commits, they can't all be security issues right? Well of course they can be! This is C after all. Like in that list, there's dozens of things I think should probably have a SECURITY tag. Over 14 days, let's just call that 2 patches a day. I'm not patching twice a day; it's hard for me to imagine anyone would, or would want to devote mental bandwidth to getting that down to a manageable rate ("I don't run that ethernet card", etc.) So for me, I actually kind of like the weekly batching? It feels pragmatic and a pretty good balancing of kernel dev/sysadmin needs. Can I envision a system that gave end-users more information? Yeah definitely, but not one that wouldn't ask LK devs to do a lot more work. Which I guess is a drawn out way of saying "feel free to write your own OS" or "consider OpenBSD" or "get involved in Rust in the kernel" or "try to move safer/microkernel designs forward" :).

rocqua 5y ago

What is your threat model / situation that you care about attackers who reverse engineer patches, but are not in the small circle of people who would be informed before hand. To me, it seems like the average corporate security team is not going to worry about these kinds of attackers. Security for state secrets might, but they seem likely to be clued in early by Linux developers. I'm probably missing something tho.

staticassertion 5y ago
> What is your threat model / situation that you care about attackers who reverse engineer patches, but are not in the small circle of people who would be informed before hand. Virtually every single Linux user. I think what you're missing is how commonplace and straightforward it is for attackers to review these commits and how uncommon it is for someone to be on the receiving end of an embargo. Most exploits are for N days, meaning that they're for vulnerabilities that have a patch out for them. Knowing that there's a patch is universally critical for all defenders. For context, my company will be posting about a kernel (then) 0day one of our security researchers discovered. You can read other Linux kernel exploitation work we've done here: https://www.graplsecurity.com/blog https://www.graplsecurity.com/blog
- rocqua 5y ago
  By threat model I mean, who are you worried about attacking you. I get that every linux user could be attacked. But why would someone with the relevant knowledge that could pull this off attack a given linux user? Why are you worried about it? (Not trying to be sarcastic, trying to get a sense of what threats you are worried about).
  - staticassertion 5y ago
    My point is that this is basically just how exploits work for Linux, so it's pretty universal unless your main concern is 0days. As for me personally, I run a company that uses Linux in production. We happen to explicitly do research into Linux kernel security (we'll be publishing tomorrow on a 0day we had reported) https://www.graplsecurity.com/blog https://www.graplsecurity.com/blog

titzer 5y ago

This is why stable branches are a thing. I don't know the branching scheme that the Linux kernel uses, but the idea is that for the oldest (most stable) branch, everything is a (sometimes backported) bugfix with security implications.

marbu 5y ago

Are you saying that you are able to read all incoming linux patches, and easily identify changes which fixes a security problem, so that you can come up with a POC by the time the security issue is announced? If the patch was flagged as a security problem from the beginning, it would give advantage to attackers, since they would know that the particular patch is worth investigating, while the defenders would have to wait for the patch to be finalized and tested anyway.

staticassertion 5y ago
You have it completely backwards.
lmm 5y ago
> Are you saying that you are able to read all incoming linux patches, and easily identify changes which fixes a security problem, so that you can come up with a POC by the time the security issue is announced? Their point is that a full-time attacker (and there's enough money in it to do it as a full-time job these days) can look for obfuscated commits and take the time to deobfuscate them, whereas a defender doesn't have that kind of time.
- marbu 5y ago
  I agree, that is definitely possible. That said it requires lot of work, since there are lot of incoming patches. I wonder how many people would have to review every proposed patch, how to select subset of incoming patches for human review, and how much one have to pay a team doing all this, to get reasonable results and return of investment. My point was that if security patches are flagged as such from the start, it saves attackers lot of time (and money), as they will no longer have to go through (almost) every patch and evaluate whether it could be fixing a security problem. This means that such scenario will get a lot cheaper, while the defenders won't gain much from that, as one still needs to wait for the fix to be finalized and tested before deploying it in a production environment.
  - staticassertion 5y ago

bobthebuilders 5y ago

You have it the wrong way around. Tagging the release as security allows nation-state level attackers with large budgets to investigate the fixes, while normal people have to wait to patches. This gives nation-state level attackers with large budgets a heads-up, making it worse for everyone else. Furthermore, nation-state level attacks with large budgets are more focused on offense than defense.

staticassertion 5y ago
This comment is totally baseless. Anyone who does linux kernel exploit development knows how to crawl the commit log or syzkaller.
- bobthebuilders 5y ago
  I'm sorry, but I'm not a nation-state. I wish I was though.