7 ms·

I tangentially followed the debacle unfold for a while and this particular thread now has lead to heated debates on some IRC channels I'm on. While it is maybe

I tangentially followed the debacle unfold for a while and this particular thread now has lead to heated debates on some IRC channels I'm on. While it is maybe "scientifically interesting", intentionally introducing bugs into Linux that could potentially make it into production systems while work on this paper is going on, could IMO be described as utterly reckless at best. Two messages down in the same thread, it more or less culminates with the university e-mail suffix being banned from several kernel mailing lists and associated patches being removed[1], which might be an appropriate response to discourage others from similar stunts "for science". [1] https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah.com/ https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah...

gspr 6y ago
> While it is maybe "scientifically interesting", intentionally introducing bugs into Linux that could potentially make it into production systems while work on this paper is going on, could IMO be described as utterly reckless at best. I agree. I would say this is kind of a "human process" analog of your typical computer security research, and that this behavior is akin to black hats exploiting a vulnerability. Totally not OK as research, and totally reckless!
- gnfargbl 6y ago
  Yep. To take a physical-world analogy: Would it be okay to try and prove the vulnerability of a country's water supply by intentionally introducing a "harmless" chemical into the treatment works, without the consent of the works owners? Or would that be a go directly to jail sort of an experiment? I share the researchers' intellectual curiosity about whether this would work, but I don't see how a properly-informed ethics board could ever have passed it.
  - Avamander 6y ago
    > Would it be okay to try and prove the vulnerability of a country's water supply by intentionally introducing a "harmless" chemical into the treatment works, without the consent of the works owners? The question should also be due to who's neglect they gained access to the "water supply". If you also truly want to make this comparison.
    - corty 6y ago
      The question is also: "Will this research have benefits?" If the conclusion is "well, you can get access to the water supply and the only means to prevent it is to closely guard every brook, lake and river, needing half the population as guards". Well, then it is useless. And taking risks for useless research is unethical, no matter how minor those risks might be.
      - Avamander 6y ago
        > If the conclusion is "well, you can get access to the water supply and the only means to prevent it is to closely guard every brook, lake and river, needing half the population as guards". I don't think that was the conclusion.
        corty 6y ago
        And what was? I cannot find constructive criticism in the related paper or any of your comments.
  - md_ 6y ago
    https://www.theonion.com/reporters-expose-airport-security-lapses-by-blowing-up-1819594671 https://www.theonion.com/reporters-expose-airport-security-l...
  - deleted 6y ago
    [deleted]
  - rcxdude 6y ago
    The US navy did actually basically this with some pathogens in the 50s: https://en.wikipedia.org/wiki/Operation_Sea-Spray https://en.wikipedia.org/wiki/Operation_Sea-Spray ; the idea of 'ethical oversight' was not something a lot of scientists operated under in those days.
- chrisjc 6y ago
  Out of interest, is there any way to have some sort of automated way to test this weak link that is human trust? (I understand how absurd this question is) It's awfully scary to think about how vulnerabilities might be purposely introduced into this important code base (as well as many other) only to be exploited at a later date for an intended purpose. Edit: NM, see st_goliath response below https://news.ycombinator.com/item?id=26888538 https://news.ycombinator.com/item?id=26888538
konschubert 6y ago
Are there any measures being discussed that could make such attacks harder in future?
- st_goliath 6y ago
  Such as? Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs? The whole idea of the mailing list based submission process is that it allows others on the list to review your patch sets and point out obvious problems with your changes and discuss them, before the maintainer picks the patches up from the list (if they don't see any problem either). As I pointed out elsewhere, there are already test farms and static analysis tools in place. On some MLs you might occasionally see auto generated mails that your patch set does not compile under configuration such-and-such, or that the static analysis bot found an issue. This is already a thing. What happened here is basically a con in the patch review process. IRL con men can scam their marks, because most people assume, when they leave the house that the majority of the others outside aren't out there to get them. Except when they run into one where the assumption doesn't hold and end up parted from their money. For the paper, bypassing the review step worked in some instances of the many patches they submitted because a) humans aren't perfect, b) have a mindset that most of the time, most people submitting bug fixes do so in good faith. Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code? Yes, I know that this review process isn't perfect, that there are problems and I'm not trying to dismiss any concerns. But what technical measure would you propose that can effectively stop con men?
  - oefrha 6y ago
    > Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs? Yes, especially for critical projects? > Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code? I don’t jump to the conclusion that the random contributor is evil. I do however think about the potential impact of the submitted patch, security or not, and I do assume a random contributor can sneak in subtle bugs, usually not intentionally, but simply due to a lack of understanding.
md_ 6y ago
I'm confused. The cited paper contains this prominent section: Ensuring the safety of the experiment. In the experiment, we aim to demonstrate the practicality of stealthily introducing vulnerabilities through hypocrite commits. Our goal is not to introduce vulnerabilities to harm OSS. Therefore, we safely conduct the experiment to make sure that the introduced UAF bugs will not be merged into the actual Linux code. In addition to the minor patches that introduce UAF conditions, we also prepare the correct patches for fixing the minor issues. We send the minor patches to the Linux community through email to seek their feedback. Fortunately, there is a time window between the confirmation of a patch and the merging of the patch. Once a maintainer confirmed our patches, e.g., an email reply indicating “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch. Are you saying that despite this, these malicious commits made it to production? Taking the authors at their word, it seems like the biggest ethical consideration here is that of potentially wasting the time of commit reviewers—which isn't nothing by any stretch, but is a far cry from introducing bugs in production. Are the authors lying?
- DetroitThrow 6y ago
  >Are you saying that despite this, these malicious commits made it to production? Vulnerable commits reached stable trees as per the maintainers in the above email exchange, though the vulnerabilities may not have been released to users yet. The researchers themselves acknowledge the patches were accepted in the above email exchange, so it's hard to believe that they're being honest or are fully aware of their ethics violations/vulnerability introductions and that they would've prevented the patches from being released without gkh's intervention.
  - md_ 6y ago
    Ah, I must've missed that. I do see people saying patches have reached stable trees, but the researchers' own email is missing (I assume removed) from the archive. Where did you find it?
    - DetroitThrow 6y ago
alrs 6y ago
If they're public IRC channels, do you mind mentioning them here? I'm trying to find the remnant. :)
d0100 6y ago
I assume that having these go into production could make the authors "hackers" according to law, no? Haven't whitehat hackers doing unsolicited pen-testing been prosecuted in the past?

gspr 6y ago

> While it is maybe "scientifically interesting", intentionally introducing bugs into Linux that could potentially make it into production systems while work on this paper is going on, could IMO be described as utterly reckless at best. I agree. I would say this is kind of a "human process" analog of your typical computer security research, and that this behavior is akin to black hats exploiting a vulnerability. Totally not OK as research, and totally reckless!

gnfargbl 6y ago
Yep. To take a physical-world analogy: Would it be okay to try and prove the vulnerability of a country's water supply by intentionally introducing a "harmless" chemical into the treatment works, without the consent of the works owners? Or would that be a go directly to jail sort of an experiment? I share the researchers' intellectual curiosity about whether this would work, but I don't see how a properly-informed ethics board could ever have passed it.
- Avamander 6y ago
  > Would it be okay to try and prove the vulnerability of a country's water supply by intentionally introducing a "harmless" chemical into the treatment works, without the consent of the works owners? The question should also be due to who's neglect they gained access to the "water supply". If you also truly want to make this comparison.
  - corty 6y ago
    The question is also: "Will this research have benefits?" If the conclusion is "well, you can get access to the water supply and the only means to prevent it is to closely guard every brook, lake and river, needing half the population as guards". Well, then it is useless. And taking risks for useless research is unethical, no matter how minor those risks might be.
    - Avamander 6y ago
      > If the conclusion is "well, you can get access to the water supply and the only means to prevent it is to closely guard every brook, lake and river, needing half the population as guards". I don't think that was the conclusion.
      - corty 6y ago
        And what was? I cannot find constructive criticism in the related paper or any of your comments.
- md_ 6y ago
  https://www.theonion.com/reporters-expose-airport-security-lapses-by-blowing-up-1819594671 https://www.theonion.com/reporters-expose-airport-security-l...
- deleted 6y ago
  [deleted]
- rcxdude 6y ago
  The US navy did actually basically this with some pathogens in the 50s: https://en.wikipedia.org/wiki/Operation_Sea-Spray https://en.wikipedia.org/wiki/Operation_Sea-Spray ; the idea of 'ethical oversight' was not something a lot of scientists operated under in those days.
chrisjc 6y ago
Out of interest, is there any way to have some sort of automated way to test this weak link that is human trust? (I understand how absurd this question is) It's awfully scary to think about how vulnerabilities might be purposely introduced into this important code base (as well as many other) only to be exploited at a later date for an intended purpose. Edit: NM, see st_goliath response below https://news.ycombinator.com/item?id=26888538 https://news.ycombinator.com/item?id=26888538

konschubert 6y ago

Are there any measures being discussed that could make such attacks harder in future?

st_goliath 6y ago
Such as? Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs? The whole idea of the mailing list based submission process is that it allows others on the list to review your patch sets and point out obvious problems with your changes and discuss them, before the maintainer picks the patches up from the list (if they don't see any problem either). As I pointed out elsewhere, there are already test farms and static analysis tools in place. On some MLs you might occasionally see auto generated mails that your patch set does not compile under configuration such-and-such, or that the static analysis bot found an issue. This is already a thing. What happened here is basically a con in the patch review process. IRL con men can scam their marks, because most people assume, when they leave the house that the majority of the others outside aren't out there to get them. Except when they run into one where the assumption doesn't hold and end up parted from their money. For the paper, bypassing the review step worked in some instances of the many patches they submitted because a) humans aren't perfect, b) have a mindset that most of the time, most people submitting bug fixes do so in good faith. Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code? Yes, I know that this review process isn't perfect, that there are problems and I'm not trying to dismiss any concerns. But what technical measure would you propose that can effectively stop con men?
- oefrha 6y ago
  > Should we assume that every patch was submitted in bad faith and tries to sneakily introduce bugs? Yes, especially for critical projects? > Do you maintain a software project? On GitHub perhaps? What do you do if somebody opens a pull request and says "I tried such and such and then found that the program crashes here, this pull request fixes that"? When reviewing the changes, do you immediately, by default jump to the assumption that they are evil, lying and trying to sneak a subtle bug into your code? I don’t jump to the conclusion that the random contributor is evil. I do however think about the potential impact of the submitted patch, security or not, and I do assume a random contributor can sneak in subtle bugs, usually not intentionally, but simply due to a lack of understanding.

md_ 6y ago

I'm confused. The cited paper contains this prominent section: Ensuring the safety of the experiment. In the experiment, we aim to demonstrate the practicality of stealthily introducing vulnerabilities through hypocrite commits. Our goal is not to introduce vulnerabilities to harm OSS. Therefore, we safely conduct the experiment to make sure that the introduced UAF bugs will not be merged into the actual Linux code. In addition to the minor patches that introduce UAF conditions, we also prepare the correct patches for fixing the minor issues. We send the minor patches to the Linux community through email to seek their feedback. Fortunately, there is a time window between the confirmation of a patch and the merging of the patch. Once a maintainer confirmed our patches, e.g., an email reply indicating “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch. Are you saying that despite this, these malicious commits made it to production? Taking the authors at their word, it seems like the biggest ethical consideration here is that of potentially wasting the time of commit reviewers—which isn't nothing by any stretch, but is a far cry from introducing bugs in production. Are the authors lying?

DetroitThrow 6y ago
>Are you saying that despite this, these malicious commits made it to production? Vulnerable commits reached stable trees as per the maintainers in the above email exchange, though the vulnerabilities may not have been released to users yet. The researchers themselves acknowledge the patches were accepted in the above email exchange, so it's hard to believe that they're being honest or are fully aware of their ethics violations/vulnerability introductions and that they would've prevented the patches from being released without gkh's intervention.
- md_ 6y ago
  Ah, I must've missed that. I do see people saying patches have reached stable trees, but the researchers' own email is missing (I assume removed) from the archive. Where did you find it?
  - DetroitThrow 6y ago

alrs 6y ago

If they're public IRC channels, do you mind mentioning them here? I'm trying to find the remnant. :)

d0100 6y ago

I assume that having these go into production could make the authors "hackers" according to law, no? Haven't whitehat hackers doing unsolicited pen-testing been prosecuted in the past?