4 ms·

People who don't understand IRBs are vastly overstating the IRB issue. Yes, they should have sought it in advance, but on the other hand almost nobody in comput

People who don't understand IRBs are vastly overstating the IRB issue. Yes, they should have sought it in advance, but on the other hand almost nobody in computer science gets human subjects research training. IRBs issue waivers of consent fairly often, particularly if there's extremely low risk to participants and disclosing the research will significantly affect the results (both of which are true here). This particular study would very likely have been granted an apriori waiver of consent without much hassle at all. Frankly, the study seems to have been fairly responsibly designed. They fixed actual kernel bugs, but included some additional more subtle latent bugs, if the subtle bugs were not detected, they corrected and removed the latent bugs and educated maintainers who accepted them.

eCa 6y ago
> IRBs issue waivers of consent fairly often, particularly if there's extremely low risk to participants and disclosing the research will significantly affect the results (both of which are true here). Apparently, the unwilling test subjects seems to strongly disagree with your conclusion in particular (I would guess) when it comes to the first point.
- fluidcruft 6y ago
  Well I'm sure this will become an IRB case-study, but I really don't think anyone is going to care.
alpaca128 6y ago
> there's extremely low risk to participants How do you plan to determine the potential risk for all kernel users in case the patch finds its way into a release? No IRB who understands this would ever approve of such "research". > disclosing the research will significantly affect the results Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment.
- fluidcruft 6y ago
  An IRB did in fact approve the research as part of the peer review publication process. > Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment. The extent to which a bad actor can use patching of bugs to introduce subtle vulnerabilities is actually valuable knowledge. We only know this even happened because it was disclosed by the researchers. Make no mistake any person or organization who wants to do this will have zero qualms about doing it.
  - bluGill 6y ago
    This was already a known risk. The gains from quantifying how bad the risk are is far outweighed by the fact that they introduced a new risk.
    - deleted 6y ago
      [deleted]
  - teraflop 6y ago
    > An IRB did in fact approve the research as part of the peer review publication process. The IRB did not approve the research, it declared that it was "exempt". That's essentially a determination that the research poses absolutely no risk of harm to anyone and does not require a detailed review, which seems like a very questionable decision to me. One of the things that's worth investigating is whether the researchers were honest in describing their experiment to the IRB. Also, it's been alleged that the IRB exemption was only sought after the research was conducted and the paper was submitted for publication. If true, that seems to imply that they lied on their NSF grant application, because the principal investigator would have had to certify that either the research did not involve human subjects or that they had already applied for IRB approval/exemption.
    - fluidcruft 6y ago
      Why is it a questionable decision? I do not see the risk it poses. Literally anyone including organizations far more well funded and vested than UMN are likely submitting hypocrite patches. At its worst, this research is merely substituting one existing bug with a new bug. And yeah, approval was sought after the research was conducted. Which itself will result in corrective action (very likely: training in human subjects research).
clarkevans 6y ago
> This particular study would very likely have been granted an apriori waiver of consent without much hassle at all. We had significant IRB work to do for a grant that involved usability testing of an application, to determine if it was more effective than existing tools. The IRB committee wanted to know what kinds of observations about software usability would be recorded and what kind of analysis we would be doing. They wanted assurance testers would be anonymous. They wanted assurance all testers would know about the evaluation, and how the analysis would be used. They wanted assurance that evaluation of the software's usability would not morph into an evaluation of the testers themselves.
- fluidcruft 6y ago
  Ok? Why would that qualify for a waiver? Disclosing the research to participants poses no risk to the study's integrity.
yjftsjthsd-h 6y ago
> if the subtle bugs were not detected, they corrected and removed the latent bugs and educated maintainers who accepted them. Last I looked, this was claimed but unverified since they have yet to give any way to identify the patches in question.
- fluidcruft 6y ago
  Publicly disclosing the patches would actually harm the reputation of any maintainer who accepted them and expose them to harassment and potential employment issues, so I strongly suspect an IRB would not allow it. It's entirely possible to fix bugs and educate without disclosing that the bugs were deliberate or part of the research.
  - zymhan 6y ago
    You're skipping the part where the IRB would have to approve injecting vulnerabilities into a public software project, a process which involves obfuscating and lying to humans.
    - fluidcruft 6y ago
      Right, but according to what's said about the study design, they (1) fixed an existing bug (2) introduced a new bug, and if maintainers didn't notice the new bug, (3) followed up with a patch removing the new bug Overall there's benefit in that an existing bug is removed and no introduced bugs persist. I don't see why IRB would be likely to object to this.
zymhan 6y ago
> but on the other hand almost nobody in computer science gets human subjects research training. I took an undergrad CS class that specifically covered IRB processes and how to research human subjects. If this isn't covered at that institution, when students/researchers are performing studies involving online communities, then that is a major oversight of UMN.
- fluidcruft 6y ago
  I agree it's an oversight, but I'm not sure it's unexpected. Some fields just don't typically do human subjects research so IRBs don't come up. I can't see why cryptographers would be wasting their time learning about how to prevent repeats of Tuskegee for example. So it probably depends on how these people got into the security research field.
huitzitziltzin 6y ago
I do think the IRB got this one wrong, but I am not surprised that they got it wrong. From my experience with an IRB, they were most worried about researchers doing another Tuskegee experiment or a Stanford prison experiment. The question they were interested in was something like “are you going to torture/abuse/harm people directly?” The people on the board have likely never thought about the issue of introducing a (possibly important) vulnerability into software which is widely used. That also has the potential for harm, but doesn’t look obvious to the IRB in the way that deliberately infecting someone with syphilis would. Getting an exempt determination for this research would likely have been extremely easy at any IRB at any university in the country. Applications from CS departments are probably less than 1% of what they do (that figure is a guess so if anyone knows better please correct it!).
- fluidcruft 6y ago
  The research plan is said to include reversing the hypocrite patch. So basically you're replacing one existing bug (if the researchers could find it, anyone else could) with an obscure bug (that at least initially only the researchers are aware of) and then only if the obscure bug slips through the maintainers, then later the obscure bug is also purged. So even if the IRB understands what's going on, I think there's a reasonable beneficence case.
- Fomite 6y ago
  In my experience, what an IRB is worried about varies wildly by IRB. My graduate university was very careful, to the point of frustration (I had a four email back-and-forth with them about a project with no human subjects). The institution I did a postdoc with on the other hand was fairly lax and hands off. Current institution is somewhere in the middle.
  - huitzitziltzin 6y ago
    True in my experience. A four email back and forth about a project w/ no human subjects is a pain. I had several months of back and forth about simply using data on events which had happened 15 years earlier. (I made (what felt like to me like) appeals to metaphysics to argue that it was very unlikely my using this data could retroactively harm people in the study.) I would be surprised if there are IRBs which get a lot of action from CS departments, and more surprised if they have members who can make a qualified evaluation of the costs and benefits of this research. I'm also not convinced that it was a bad idea. This was extremely poorly executed, but illustrates an important (though maybe obvious) point: this method for introducing vulnerabilities would work. A better way to handle it, IMO, would have been to involve someone at the very top in the Linux organization, to say "hey we are going to do this to see if it works - are you ok with that?" and then as soon as it works once, or on some defined, very small number of times, stop immediately and tell everyone what you did.
Fomite 6y ago
"Yes, they should have sought it in advance, but on the other hand almost nobody in computer science gets human subjects research training." This is a major failing.

eCa 6y ago

> IRBs issue waivers of consent fairly often, particularly if there's extremely low risk to participants and disclosing the research will significantly affect the results (both of which are true here). Apparently, the unwilling test subjects seems to strongly disagree with your conclusion in particular (I would guess) when it comes to the first point.

fluidcruft 6y ago
Well I'm sure this will become an IRB case-study, but I really don't think anyone is going to care.

alpaca128 6y ago

> there's extremely low risk to participants How do you plan to determine the potential risk for all kernel users in case the patch finds its way into a release? No IRB who understands this would ever approve of such "research". > disclosing the research will significantly affect the results Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment.

fluidcruft 6y ago
An IRB did in fact approve the research as part of the peer review publication process. > Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment. The extent to which a bad actor can use patching of bugs to introduce subtle vulnerabilities is actually valuable knowledge. We only know this even happened because it was disclosed by the researchers. Make no mistake any person or organization who wants to do this will have zero qualms about doing it.
- bluGill 6y ago
  This was already a known risk. The gains from quantifying how bad the risk are is far outweighed by the fact that they introduced a new risk.
  - deleted 6y ago
    [deleted]
- teraflop 6y ago
  > An IRB did in fact approve the research as part of the peer review publication process. The IRB did not approve the research, it declared that it was "exempt". That's essentially a determination that the research poses absolutely no risk of harm to anyone and does not require a detailed review, which seems like a very questionable decision to me. One of the things that's worth investigating is whether the researchers were honest in describing their experiment to the IRB. Also, it's been alleged that the IRB exemption was only sought after the research was conducted and the paper was submitted for publication. If true, that seems to imply that they lied on their NSF grant application, because the principal investigator would have had to certify that either the research did not involve human subjects or that they had already applied for IRB approval/exemption.
  - fluidcruft 6y ago
    Why is it a questionable decision? I do not see the risk it poses. Literally anyone including organizations far more well funded and vested than UMN are likely submitting hypocrite patches. At its worst, this research is merely substituting one existing bug with a new bug. And yeah, approval was sought after the research was conducted. Which itself will result in corrective action (very likely: training in human subjects research).

clarkevans 6y ago

> This particular study would very likely have been granted an apriori waiver of consent without much hassle at all. We had significant IRB work to do for a grant that involved usability testing of an application, to determine if it was more effective than existing tools. The IRB committee wanted to know what kinds of observations about software usability would be recorded and what kind of analysis we would be doing. They wanted assurance testers would be anonymous. They wanted assurance all testers would know about the evaluation, and how the analysis would be used. They wanted assurance that evaluation of the software's usability would not morph into an evaluation of the testers themselves.

fluidcruft 6y ago
Ok? Why would that qualify for a waiver? Disclosing the research to participants poses no risk to the study's integrity.

yjftsjthsd-h 6y ago

> if the subtle bugs were not detected, they corrected and removed the latent bugs and educated maintainers who accepted them. Last I looked, this was claimed but unverified since they have yet to give any way to identify the patches in question.

fluidcruft 6y ago
Publicly disclosing the patches would actually harm the reputation of any maintainer who accepted them and expose them to harassment and potential employment issues, so I strongly suspect an IRB would not allow it. It's entirely possible to fix bugs and educate without disclosing that the bugs were deliberate or part of the research.
- zymhan 6y ago
  You're skipping the part where the IRB would have to approve injecting vulnerabilities into a public software project, a process which involves obfuscating and lying to humans.
  - fluidcruft 6y ago
    Right, but according to what's said about the study design, they (1) fixed an existing bug (2) introduced a new bug, and if maintainers didn't notice the new bug, (3) followed up with a patch removing the new bug Overall there's benefit in that an existing bug is removed and no introduced bugs persist. I don't see why IRB would be likely to object to this.

zymhan 6y ago

> but on the other hand almost nobody in computer science gets human subjects research training. I took an undergrad CS class that specifically covered IRB processes and how to research human subjects. If this isn't covered at that institution, when students/researchers are performing studies involving online communities, then that is a major oversight of UMN.

fluidcruft 6y ago
I agree it's an oversight, but I'm not sure it's unexpected. Some fields just don't typically do human subjects research so IRBs don't come up. I can't see why cryptographers would be wasting their time learning about how to prevent repeats of Tuskegee for example. So it probably depends on how these people got into the security research field.

huitzitziltzin 6y ago

I do think the IRB got this one wrong, but I am not surprised that they got it wrong. From my experience with an IRB, they were most worried about researchers doing another Tuskegee experiment or a Stanford prison experiment. The question they were interested in was something like “are you going to torture/abuse/harm people directly?” The people on the board have likely never thought about the issue of introducing a (possibly important) vulnerability into software which is widely used. That also has the potential for harm, but doesn’t look obvious to the IRB in the way that deliberately infecting someone with syphilis would. Getting an exempt determination for this research would likely have been extremely easy at any IRB at any university in the country. Applications from CS departments are probably less than 1% of what they do (that figure is a guess so if anyone knows better please correct it!).

fluidcruft 6y ago
The research plan is said to include reversing the hypocrite patch. So basically you're replacing one existing bug (if the researchers could find it, anyone else could) with an obscure bug (that at least initially only the researchers are aware of) and then only if the obscure bug slips through the maintainers, then later the obscure bug is also purged. So even if the IRB understands what's going on, I think there's a reasonable beneficence case.
Fomite 6y ago
In my experience, what an IRB is worried about varies wildly by IRB. My graduate university was very careful, to the point of frustration (I had a four email back-and-forth with them about a project with no human subjects). The institution I did a postdoc with on the other hand was fairly lax and hands off. Current institution is somewhere in the middle.
- huitzitziltzin 6y ago
  True in my experience. A four email back and forth about a project w/ no human subjects is a pain. I had several months of back and forth about simply using data on events which had happened 15 years earlier. (I made (what felt like to me like) appeals to metaphysics to argue that it was very unlikely my using this data could retroactively harm people in the study.) I would be surprised if there are IRBs which get a lot of action from CS departments, and more surprised if they have members who can make a qualified evaluation of the costs and benefits of this research. I'm also not convinced that it was a bad idea. This was extremely poorly executed, but illustrates an important (though maybe obvious) point: this method for introducing vulnerabilities would work. A better way to handle it, IMO, would have been to involve someone at the very top in the Linux organization, to say "hey we are going to do this to see if it works - are you ok with that?" and then as soon as it works once, or on some defined, very small number of times, stop immediately and tell everyone what you did.

Fomite 6y ago

"Yes, they should have sought it in advance, but on the other hand almost nobody in computer science gets human subjects research training." This is a major failing.