6 ms·

Do you have actual evidence of that in a case like this? (This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objectio

by amluto 5y ago

Do you have actual evidence of that in a case like this? (This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objections won’t help.)

staticassertion 5y ago
Evidence of what, exactly? I can find you lots of evidence for hiding vulns, they don't even hide it - I'm sure Greg will admit to as much. Evidence of this being helpful to attackers and not defenders? IDK, talk to anyone who does Linux kernel exploit development. edit: There you go, Greg linked his policy, which explicitly notes this.
rfoo 5y ago
Not OP, but please do try to influence this policy if you can: 1. The commit message [1] does not mention any security implication. This is reasonable, because the patch is usually released to the public earlier and it makes sense to do some obfuscation, to deter patch-gappers. But note that this approach is not a controversy-free one. 2. But there is also no security announcement in stable release notes or any similar stuff. I don't know how to provide evidence of "something simply does not exist". 3. Check the timeline in the blog post. The bug being fixed in stable release (5.6.11 on 2022-02-23) marks the end of upstream's handling of this bug. Max then had to send the bug details to linux-distros list to kick off (another separate process) distro maintainers' response. If what you are maintaining is not a distro, good luck. Is this wrong-sounding enough? [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9d2231c5d74e13b2a0546fee6737ee4446017903 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- amluto 5y ago
  #1 is intentional, for better or for worse. It’s certainly well-intentioned too, although the intentions may be based on wrong assumptions. #2: upstream makes no general effort to identify security bugs as such. Obviously this one was known to be a security bug, but the general policy (see #1) is to avoid announcing it. #3: In any embargo situation, if you’re not on the distribution list, you don’t get notified. This is unavoidable. oss-security nominally handles everyone else, but it’s very spotty. Sometimes I wish there was a Linux kernel security advisory process, but this would need funding or a dedicated volunteer.
  - rfoo 5y ago
    TBH the thing annoyed me most in this story is the "Someone had to start the disclosure process on linux-distros again and if they didn't no one would know"-part. There are certainly silent bug fixes where the author intentionally (or not) does not post to linux-distros or any other maillists even after stable release. It would take an hour to dig a good example tho. (Okay, maybe 10 minutes if I'm going to read Brad Spengler's rants) I guess a Linux kernel security advisory process is needed to fix this, but yeah :(
    - amluto 5y ago
      For what it’s worth, linux-distros has its own opinions that are not necessarily compatible with those of the upstream kernel.
  - mjw1007 5y ago
    If only there was some kind of foundation with a revenue of $177 million last year which had an interest in Linux's success.
    - nisa 5y ago
      they are busy doing blockchain projects :)
  - sirdarckcat 5y ago
    > Sometimes I wish there was a Linux kernel security advisory process, but this would need funding or a dedicated volunteer. This is already happening https://osv.dev/list?q=Kernel&affected_only=true&page=1&ecosystem=Linux https://osv.dev/list?q=Kernel&affected_only=true&page=1&ecos...
    - amluto 5y ago
      As far as I know, this doesn’t get information from upstream maintainers. For this to work well, I think we would want actual advisories generated around commit time, embargoed early notification, and a process for publication.
rocqua 5y ago
This is about the commit that fixed the bug, not the commit that introduced the bug. The accusation is not that linux developers intentionally introduced a vulnerability. Instead it is that linux developers hid that a commit fixed a vulnerability. Linux does this to prevent people from learning that the vulnerability exists.
- wtarreau 5y ago
  > Linux does this to prevent people from learning that the vulnerability exists No, not at all, just to leave time to users to deploy the fix before everyone jumps on exploits. This is important because every single backported patch is a candidate for an exploit already, and it's only a matter of time before any of them is exploited. Reason why embargoes have to stay short. It takes some time to figure whether a bug may have security impacts. It takes much less time once this is figured, to develop an exploit. By the way it could have really happened that the fix for data corruption would have been merged first, and only later the author figured there was a security impact. And the patch wouldn't have been any different. That's why leaving 1-2 weeks for the fix to flow via distros to users, and having the author post a complete article is by far the best solution for everyone.
  - roddux 5y ago
    Nobody is arguing that users having a 1-2 week patch window is a bad thing. However, this frankly seems incompatible with open-source projects. Silently patching issues does not work in practice; it frequently leads to missed fixes, misapplied patches and other incompatibility woes. The situation with backports and LTS releases showcases this well— the only truly well-supported kernel is latest. Everything else is a patchwork of best-effort fixes, not all of which may have been applied correctly. Brad Spengler of grsecurity fame talks frequently about this (primarily via Twitter): https://twitter.com/spendergrsec https://twitter.com/spendergrsec
    - wtarreau 5y ago
      Not really. As you say there's an extremely difficult balance with opensource and not exposing everyone at once. You can't get a fix deployed everywhere without it being public first or it ends up in a total unfixable mess. But if the fix is public and gives too many info (exploit procedure) then you put everyone in danger until the fix flows to users. Thus the only solution is to have a public fix describing the bug and not necessarily all the details, while distros prepare their update, and everyone discloses the trouble at the same time. Those who need early notification MUST ABSOLUTELY BE on linux-distros. There's no other way around. As soon as the patch is published, the risk is non-nul and a race is started between those who look for candidate fixes and those who have to distribute fixes to end users. This is not about silently patching or hiding bugs, quite the opposite, it's about making them public as quickly as possible so that the fix can be picked, but without the unneeded elements that help vandals damage shared systems before these systems have a chance to be updated. Then it is useful that the reporter communicates about their finding, this often helps improve general security by documenting how certain classes of bugs turn to security issues (Max did an awesome job here, probably the best such bug report in the last few years). And distros need to publish more details as well in their advisories, so details are not "hidden", they're just delayed during tha embargo. Those who are not notified AND who do not follow stable are simply irresponsible. But I don't think there are that many doing that nowadays, possibly just a few hero admins in small companies trying to impress their boss with their collection of carefully selected patches (that render their machine even more vulnerable and tend to make them vocal when such issues happen). In addition it's important to keep in mind that some bugs are discovered as being exploitable long after being fixed. That's why one MUST ABSOLUTELY NOT rely on the commit message alone to decide whether they are vulnerable or not, since it's quite common not to know upfront. I remember a year or two ago someone from Google's security team reported a bug on haproxy that could cause a crash in the HPACK decoder. That was extremely embarrassing as it could allow anyone to remotely crash haproxy. We had to release the fix indicating that the bug was critical and that the risk of crashing when facing bad data was real, without explaining how to crash it (since like a kernel it's a components many people tend to forget to upgrade). Then after the fix was merged, I was still discussing with the reporter and asked "do you think it could further be abused for RCE?". He said "let me check". A week later he came back saying "good news, I succeeded". No way to get that info in the commit message even if we wanted to, since that was too late. Yet the issue was important. Speaking of Brad, I personally think that grsec ought to be on linux-distros, but maybe they prefer not to appear as "tainted" by early notifications, or maybe they're having some fun finding other issues themselves. We even proposed Brad to be on the security list, because he has the skills to help a lot and improve the security there. He could have interesting writeups for some of the bugs, and it would probably change his perception of what happens there. Maybe one day he'll accept (still keeping hope :-)).

by amluto 5y ago

Do you have actual evidence of that in a case like this? (This is not a rhetorical question. I can possibly influence this policy, but unsubstantiated objections won’t help.)

staticassertion 5y ago

Evidence of what, exactly? I can find you lots of evidence for hiding vulns, they don't even hide it - I'm sure Greg will admit to as much. Evidence of this being helpful to attackers and not defenders? IDK, talk to anyone who does Linux kernel exploit development. edit: There you go, Greg linked his policy, which explicitly notes this.

rfoo 5y ago

Not OP, but please do try to influence this policy if you can: 1. The commit message [1] does not mention any security implication. This is reasonable, because the patch is usually released to the public earlier and it makes sense to do some obfuscation, to deter patch-gappers. But note that this approach is not a controversy-free one. 2. But there is also no security announcement in stable release notes or any similar stuff. I don't know how to provide evidence of "something simply does not exist". 3. Check the timeline in the blog post. The bug being fixed in stable release (5.6.11 on 2022-02-23) marks the end of upstream's handling of this bug. Max then had to send the bug details to linux-distros list to kick off (another separate process) distro maintainers' response. If what you are maintaining is not a distro, good luck. Is this wrong-sounding enough? [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9d2231c5d74e13b2a0546fee6737ee4446017903 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

amluto 5y ago
#1 is intentional, for better or for worse. It’s certainly well-intentioned too, although the intentions may be based on wrong assumptions. #2: upstream makes no general effort to identify security bugs as such. Obviously this one was known to be a security bug, but the general policy (see #1) is to avoid announcing it. #3: In any embargo situation, if you’re not on the distribution list, you don’t get notified. This is unavoidable. oss-security nominally handles everyone else, but it’s very spotty. Sometimes I wish there was a Linux kernel security advisory process, but this would need funding or a dedicated volunteer.
- rfoo 5y ago
  TBH the thing annoyed me most in this story is the "Someone had to start the disclosure process on linux-distros again and if they didn't no one would know"-part. There are certainly silent bug fixes where the author intentionally (or not) does not post to linux-distros or any other maillists even after stable release. It would take an hour to dig a good example tho. (Okay, maybe 10 minutes if I'm going to read Brad Spengler's rants) I guess a Linux kernel security advisory process is needed to fix this, but yeah :(
  - amluto 5y ago
    For what it’s worth, linux-distros has its own opinions that are not necessarily compatible with those of the upstream kernel.
- mjw1007 5y ago
  If only there was some kind of foundation with a revenue of $177 million last year which had an interest in Linux's success.
  - nisa 5y ago
    they are busy doing blockchain projects :)
- sirdarckcat 5y ago
  > Sometimes I wish there was a Linux kernel security advisory process, but this would need funding or a dedicated volunteer. This is already happening https://osv.dev/list?q=Kernel&affected_only=true&page=1&ecosystem=Linux https://osv.dev/list?q=Kernel&affected_only=true&page=1&ecos...
  - amluto 5y ago
    As far as I know, this doesn’t get information from upstream maintainers. For this to work well, I think we would want actual advisories generated around commit time, embargoed early notification, and a process for publication.

rocqua 5y ago

This is about the commit that fixed the bug, not the commit that introduced the bug. The accusation is not that linux developers intentionally introduced a vulnerability. Instead it is that linux developers hid that a commit fixed a vulnerability. Linux does this to prevent people from learning that the vulnerability exists.

wtarreau 5y ago
> Linux does this to prevent people from learning that the vulnerability exists No, not at all, just to leave time to users to deploy the fix before everyone jumps on exploits. This is important because every single backported patch is a candidate for an exploit already, and it's only a matter of time before any of them is exploited. Reason why embargoes have to stay short. It takes some time to figure whether a bug may have security impacts. It takes much less time once this is figured, to develop an exploit. By the way it could have really happened that the fix for data corruption would have been merged first, and only later the author figured there was a security impact. And the patch wouldn't have been any different. That's why leaving 1-2 weeks for the fix to flow via distros to users, and having the author post a complete article is by far the best solution for everyone.
- roddux 5y ago
  Nobody is arguing that users having a 1-2 week patch window is a bad thing. However, this frankly seems incompatible with open-source projects. Silently patching issues does not work in practice; it frequently leads to missed fixes, misapplied patches and other incompatibility woes. The situation with backports and LTS releases showcases this well— the only truly well-supported kernel is latest. Everything else is a patchwork of best-effort fixes, not all of which may have been applied correctly. Brad Spengler of grsecurity fame talks frequently about this (primarily via Twitter): https://twitter.com/spendergrsec https://twitter.com/spendergrsec
  - wtarreau 5y ago
    Not really. As you say there's an extremely difficult balance with opensource and not exposing everyone at once. You can't get a fix deployed everywhere without it being public first or it ends up in a total unfixable mess. But if the fix is public and gives too many info (exploit procedure) then you put everyone in danger until the fix flows to users. Thus the only solution is to have a public fix describing the bug and not necessarily all the details, while distros prepare their update, and everyone discloses the trouble at the same time. Those who need early notification MUST ABSOLUTELY BE on linux-distros. There's no other way around. As soon as the patch is published, the risk is non-nul and a race is started between those who look for candidate fixes and those who have to distribute fixes to end users. This is not about silently patching or hiding bugs, quite the opposite, it's about making them public as quickly as possible so that the fix can be picked, but without the unneeded elements that help vandals damage shared systems before these systems have a chance to be updated. Then it is useful that the reporter communicates about their finding, this often helps improve general security by documenting how certain classes of bugs turn to security issues (Max did an awesome job here, probably the best such bug report in the last few years). And distros need to publish more details as well in their advisories, so details are not "hidden", they're just delayed during tha embargo. Those who are not notified AND who do not follow stable are simply irresponsible. But I don't think there are that many doing that nowadays, possibly just a few hero admins in small companies trying to impress their boss with their collection of carefully selected patches (that render their machine even more vulnerable and tend to make them vocal when such issues happen). In addition it's important to keep in mind that some bugs are discovered as being exploitable long after being fixed. That's why one MUST ABSOLUTELY NOT rely on the commit message alone to decide whether they are vulnerable or not, since it's quite common not to know upfront. I remember a year or two ago someone from Google's security team reported a bug on haproxy that could cause a crash in the HPACK decoder. That was extremely embarrassing as it could allow anyone to remotely crash haproxy. We had to release the fix indicating that the bug was critical and that the risk of crashing when facing bad data was real, without explaining how to crash it (since like a kernel it's a components many people tend to forget to upgrade). Then after the fix was merged, I was still discussing with the reporter and asked "do you think it could further be abused for RCE?". He said "let me check". A week later he came back saying "good news, I succeeded". No way to get that info in the commit message even if we wanted to, since that was too late. Yet the issue was important. Speaking of Brad, I personally think that grsec ought to be on linux-distros, but maybe they prefer not to appear as "tainted" by early notifications, or maybe they're having some fun finding other issues themselves. We even proposed Brad to be on the security list, because he has the skills to help a lot and improve the security there. He could have interesting writeups for some of the bugs, and it would probably change his perception of what happens there. Maybe one day he'll accept (still keeping hope :-)).