5 ms·

They obtained an "IRB-exempt letter" because their IRB found that this was not human research. It's quite likely that the IRB made this finding based on a misre

by thaeli 5y ago

They obtained an "IRB-exempt letter" because their IRB found that this was not human research. It's quite likely that the IRB made this finding based on a misrepresentation of the research during that initial stage; once they had an exemption letter the IRB wouldn't be looking any closer.

lbarrow 5y ago
My understanding is that it's pretty common for CS departments to get IRB exemption even when human participants are tangentially involved in studies.
- lmkg 5y ago
  I've seen from a distance one CS department struggle with IRB to get approval for using Amazon Mechanical Turk to label pictures for computer vision datasets. I believe the resolution was creating a specialized approval process for that family of tasks.
- waheoo 5y ago
  That sounds like a disconnect from reality.
  - WORLD_ENDS_SOON 5y ago
    I think it is because many labs in CS departments do very little research involving human subjects (e.g. a machine learning lab or a theory lab), so within those labs there isn't really an expectation that everything goes through IRB. Many CS graduate students likely never have to interact with IRB at all, so they probably don't even know when it is necessary to involve IRB. The rules for what requires IRB involvement are also somewhat open to interpretation. For example, surveys are often exempt depending on what the survey is asking about.
    - waheoo 5y ago
      Machine learning automatically being exempt is a huge red flag for me. There are immense repercussions for the world on every comp sci topic. It's just less direct, and often "digital" which seems separate but it's not.
- corty 5y ago
  It is also quite easy to pull the wool over an IRBs eyes. An IRB is usually staffed with a few people from the medicine, biology, psychology and maybe (for the good ethical looks) philosophy and theology departments. Usually they aren't really qualified to know what a computer scientist is talking about describing their research. And also, given that the stakes are higher e.g. in medicine, and the bar is lower in biology, one often gets a pass: "You don't want to poke anyone with needles, no LSD and no cages? Why are you asking us then?" Or something to that effect. The IRBs are just not used to such "harmless" things not being justified by the research objective.
  - avs733 5y ago
    see my other comment to the GP. pulling the wool suggests agency and intentionality that isn't necessarily present when you have disciplinary differences like you describe. Simple miscommunication, e.g., using totally normal field terminology that does not translate well, is different.
    - corty 5y ago
      It is your job as a researcher to make the committee fully understand all the implications of what you are doing. If you fail in that, you failed in your duties. The committee will also tell you this, as well as any ethical guideline. Given that level of responsibility, it isn't easy to ascribe this to negligence on the part of the researchers, intent is far more likely.
      - avs733 5y ago
        It is absolutely my job, but I don’t necessarily have actionable information that I created a misunderstanding. I submit unclear thing Thing is approved Thing must have been clear right?
        davidkuhta 5y ago
        'Not ignorance, but ignorance of ignorance is the death of knowledge.' - Alfred North Whitehead
devmor 5y ago
That's what it seemed like to me as well. Based on their research paper, they did not mention the individuals they interacted with at all. They also lied in the paper about their methodology - claiming that once their code was accepted, they told the maintainers it should not be included. In reality, several of their bad commits made it into the stable branch.
- rocqua 5y ago
  > In reality, several of their bad commits made it into the stable branch. Is it known whether these commits were indeed bad? It is certainly worth removing them just in case, but is there any confirmation?
  - Koliakis 5y ago
    https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7ccb399@roeck-us.net/ https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7c...
    - nightpool 5y ago
      This is a completely separate incident, a year apart from the paper under discussion.
      - Koliakis 5y ago
        Then just go through the linked mailing list in the OP. It's in the quoted parts. Honestly, the people around here.
  - devmor 5y ago
    I don't think we know if they contain bugs, but from what I gathered reading the mailing list, we do know that they added nothing of value.
- bogwog 5y ago
  I don’t think that’s what’s happening here. The research paper you’re talking about was already published, and supposedly only consisted of 3 patches, not the 200 or so being reverted here. So it’s possible that this situation has nothing to do with that research, and is just another unethical thing that coincidentally comes from the same university. Or it really is a new study by the same people. Either way, I think we should get the facts straight before the wrong people are attacked.
avs733 5y ago
Not necessarily. And the conflation of IRB-exemption and not human subjects research is not exactly correct.[0] Each institution, and each IRB is made up of people and a set of policies. One does not have to meaningfully misrepresent things to IRBs for them to be misunderstood. Further, exempt from IRB review and 'not human subjects research' are not actually the same thing. I've run into this problem personally - IRB declines to review the research plan because it does not meet their definition of human subjects research, however the journal will not accept the article without IRB review. Catch-22. Further, research that involves deception is also considered a perfectly valid form of research in certain fields (e.g., Psychology). The IRB may not have responded simply because they see the complaint as invalid. Their mandate is protecting human beings from harm, not random individuals who email them from annoyance. They don't have in their framework protecting the linux kernel from harm any more than they have protecting a jet engine from harm (Sorry if that sounds callous). Someone not liking a study is not research misconduct and if the IRB determined within their processes that it isn't even human subjects research, there isn't a lot they can do here. I suspect that this is just one of those disconnects that happens when people talk across disciplines. no misrepresentation was needed, all that was needed was for someone reviewing this, who's background is medicine and not CS, to not understand the organizational and human processes behind submitting a software 'patch'. The follow up behavior...not great...but the start of this could be a serious of individually rational actions that combine into something problematic because they were not holistically evaluated in context. [0] https://oprs.usc.edu/irb-review/types-of-irb-review/ https://oprs.usc.edu/irb-review/types-of-irb-review/
- nightpool 5y ago
  My understanding in this case is not that the IRB declined to review the study plan, but that (quoting the study authors) "The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained)." (more information here: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....) Do you think that the IRB was correct to make the determination they did? It does sound like a bit of a grey area
  - 5y ago

by thaeli 5y ago

lbarrow 5y ago

My understanding is that it's pretty common for CS departments to get IRB exemption even when human participants are tangentially involved in studies.

lmkg 5y ago
I've seen from a distance one CS department struggle with IRB to get approval for using Amazon Mechanical Turk to label pictures for computer vision datasets. I believe the resolution was creating a specialized approval process for that family of tasks.
waheoo 5y ago
That sounds like a disconnect from reality.
- WORLD_ENDS_SOON 5y ago
  I think it is because many labs in CS departments do very little research involving human subjects (e.g. a machine learning lab or a theory lab), so within those labs there isn't really an expectation that everything goes through IRB. Many CS graduate students likely never have to interact with IRB at all, so they probably don't even know when it is necessary to involve IRB. The rules for what requires IRB involvement are also somewhat open to interpretation. For example, surveys are often exempt depending on what the survey is asking about.
  - waheoo 5y ago
    Machine learning automatically being exempt is a huge red flag for me. There are immense repercussions for the world on every comp sci topic. It's just less direct, and often "digital" which seems separate but it's not.
corty 5y ago
It is also quite easy to pull the wool over an IRBs eyes. An IRB is usually staffed with a few people from the medicine, biology, psychology and maybe (for the good ethical looks) philosophy and theology departments. Usually they aren't really qualified to know what a computer scientist is talking about describing their research. And also, given that the stakes are higher e.g. in medicine, and the bar is lower in biology, one often gets a pass: "You don't want to poke anyone with needles, no LSD and no cages? Why are you asking us then?" Or something to that effect. The IRBs are just not used to such "harmless" things not being justified by the research objective.
- avs733 5y ago
  see my other comment to the GP. pulling the wool suggests agency and intentionality that isn't necessarily present when you have disciplinary differences like you describe. Simple miscommunication, e.g., using totally normal field terminology that does not translate well, is different.
  - corty 5y ago
    It is your job as a researcher to make the committee fully understand all the implications of what you are doing. If you fail in that, you failed in your duties. The committee will also tell you this, as well as any ethical guideline. Given that level of responsibility, it isn't easy to ascribe this to negligence on the part of the researchers, intent is far more likely.
    - avs733 5y ago
      It is absolutely my job, but I don’t necessarily have actionable information that I created a misunderstanding. I submit unclear thing Thing is approved Thing must have been clear right?
      - davidkuhta 5y ago
        'Not ignorance, but ignorance of ignorance is the death of knowledge.' - Alfred North Whitehead

devmor 5y ago

That's what it seemed like to me as well. Based on their research paper, they did not mention the individuals they interacted with at all. They also lied in the paper about their methodology - claiming that once their code was accepted, they told the maintainers it should not be included. In reality, several of their bad commits made it into the stable branch.

rocqua 5y ago
> In reality, several of their bad commits made it into the stable branch. Is it known whether these commits were indeed bad? It is certainly worth removing them just in case, but is there any confirmation?
- Koliakis 5y ago
  https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7ccb399@roeck-us.net/ https://lore.kernel.org/lkml/78ac6ee8-8e7c-bd4c-a3a7-5a90c7c...
  - nightpool 5y ago
    This is a completely separate incident, a year apart from the paper under discussion.
    - Koliakis 5y ago
      Then just go through the linked mailing list in the OP. It's in the quoted parts. Honestly, the people around here.
- devmor 5y ago
  I don't think we know if they contain bugs, but from what I gathered reading the mailing list, we do know that they added nothing of value.
bogwog 5y ago
I don’t think that’s what’s happening here. The research paper you’re talking about was already published, and supposedly only consisted of 3 patches, not the 200 or so being reverted here. So it’s possible that this situation has nothing to do with that research, and is just another unethical thing that coincidentally comes from the same university. Or it really is a new study by the same people. Either way, I think we should get the facts straight before the wrong people are attacked.

avs733 5y ago

Not necessarily. And the conflation of IRB-exemption and not human subjects research is not exactly correct.[0] Each institution, and each IRB is made up of people and a set of policies. One does not have to meaningfully misrepresent things to IRBs for them to be misunderstood. Further, exempt from IRB review and 'not human subjects research' are not actually the same thing. I've run into this problem personally - IRB declines to review the research plan because it does not meet their definition of human subjects research, however the journal will not accept the article without IRB review. Catch-22. Further, research that involves deception is also considered a perfectly valid form of research in certain fields (e.g., Psychology). The IRB may not have responded simply because they see the complaint as invalid. Their mandate is protecting human beings from harm, not random individuals who email them from annoyance. They don't have in their framework protecting the linux kernel from harm any more than they have protecting a jet engine from harm (Sorry if that sounds callous). Someone not liking a study is not research misconduct and if the IRB determined within their processes that it isn't even human subjects research, there isn't a lot they can do here. I suspect that this is just one of those disconnects that happens when people talk across disciplines. no misrepresentation was needed, all that was needed was for someone reviewing this, who's background is medicine and not CS, to not understand the organizational and human processes behind submitting a software 'patch'. The follow up behavior...not great...but the start of this could be a serious of individually rational actions that combine into something problematic because they were not holistically evaluated in context. [0] https://oprs.usc.edu/irb-review/types-of-irb-review/ https://oprs.usc.edu/irb-review/types-of-irb-review/

nightpool 5y ago
My understanding in this case is not that the IRB declined to review the study plan, but that (quoting the study authors) "The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained)." (more information here: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....) Do you think that the IRB was correct to make the determination they did? It does sound like a bit of a grey area
- 5y ago