18 ms·

I totally agree this is a very balanced review of the affair. The fact that "kernel maintainers ... are overworked and do not have the time to properly review

I totally agree this is a very balanced review of the affair. The fact that "kernel maintainers ... are overworked and do not have the time to properly review every patch that passes through their hands" is not a situation created by the group of researchers. If the Linux kernel is so crucial to the real world industry, and the kernel team is so distressed at their work, this is a problem need to be seriously addressed. Proactively punish the whole UMN does nothing to address the real issue here. If the team is so distressed at this tremendously important work and they constantly feel they are on the edge of collapsing, may be you really really really should ask for outside help and even demand more contributions from the big techs. Nobody should overwork, especially those work on something so crucial to the whole industry. Edit: I just read a line from the Rust Foundation page [1]: "Good software is built by happy, well-supported people." 1. https://foundation.rust-lang.org/ https://foundation.rust-lang.org/

josefx 6y ago
> is not a situation created by the group of researchers. As far as I got from various threads on the issue the university has been involved in at least two major time wasters: * Breaking the trust model and forcing the maintainers to revert all already accepted patches on suspicion alone * Running a primitive static code analysis tool that got overeager with correcting impossible bugs and not marking the generated patches as such. > Proactively punish the whole UMN does nothing to address the real issue here. Proactive? The universities review board signed off on that mess, there is nothing proactive about banning a group of bad actors after the fact.
- SiempreViernes 6y ago
  Proactive probable refers to also banning the 6700 staff, along with the 46000 students, that are not part of the research project from ever contributing.
  - monocasa 6y ago
    That staff and student body is governed by the same ethics board that allowed this to happen.
    - cycomanic 6y ago
      Facebook experimented on people (some might have even been kernel programmers), but Facebook programmers are still contributing to the kernel. Let's not even talk about all the other organisations who have done some pretty unethical things, I mean lots of corporations who have or still are violating the GPL (on Kernel code even) are allowed to continue to contribute to the kernel.
      - monocasa 6y ago
        They didn't experiment on the kernel process itself. And it would be counterproductive to set up the incentives so that people who aren't contributing in the first place can't.
      - pabs3 5y ago
        Do you have any examples of companies that are both still GPL violating and still contributing to Linux mainline?
    - seoaeu 5y ago
      Why is that a problem? Most kernel contributors aren't governed by any ethics board at all!
      - josefx 5y ago
        They probably work for some company and can you imagine the shit storm if it ever came out that Microsofts board of directors signed off on a Microsoft dev. actively submitting vulnerabilities into the Linux kernel?
        seoaeu 5y ago
        An IRB and a board of directors aren't remotely similar. UMN actually does have a board of trustees but they've had literally nothing to do with this incident. The senior administration (and the CS&E department prior to the mailing list blowup) never even knew that this research was happening.
  - zopa 6y ago
    Realistically, if a UMN student submits a patch from their personal, non-.edu email address, will anyone notice or care?
- staticassertion 6y ago
  > * Running a primitive static code analysis tool that got overeager with correcting impossible bugs and not marking the generated patches as such. This is not a major time waster. It happens constantly - students everywhere are doing this, and the patches are accepted constantly. Further, while Greg classified the entirety of the analyzers findings as being useless, he was incorrect.[0] > The universities review board signed off on that mess, there is nothing proactive about banning a group of bad actors after the fact. In terms of the researchers submitting malicious patches it was done without edu addresses, which means that banning them solves nothing. [0] https://lore.kernel.org/lkml/b43fc2b0-b3cf-15ab-7d3c-25c1f2a3e658@canonical.com/ https://lore.kernel.org/lkml/b43fc2b0-b3cf-15ab-7d3c-25c1f2a... In fact, most of the fixes fit into the bucket of "potentially useful".
  - spacechild1 5y ago
    > students everywhere are doing this Which somehow makes it ok? The Linux team told UMN not to run experiments on the kernel anymore, and then a student comes along and does it again. Given his association with Kangjie Lu, he should have known better.
- seoaeu 5y ago
  > Breaking the trust model and forcing the maintainers to revert all already accepted patches on suspicion alone The kernel developers chose to do that out of a combination of spite and excessive caution. The actual malicious patches didn't even come from UMN email addresses. > Proactive? The universities review board signed off on that mess, there is nothing proactive about banning a group of bad actors after the fact. Why does everyone keep repeating this? The IRB declared that it wasn't human subject research. Tons of other IRBs would have done the same because they all use a standardized definition that hasn't been updated for interactions with online communities.
  - josefx 5y ago
    > Tons of other IRBs would have done the same because they all use a standardized definition that hasn't been updated for interactions with online communities. I can imagine a paper from the 1940s showing how they checked the security precautions of baby food manufacturers by having a factory worker under their control randomly lace the produced baby food with arsenic to see if the manipulated batches got caught by an "abstract process" before they could enter general distribution. Yeah, nothing that went wrong in this case can be excused by "but the internets!!!".
    - seoaeu 5y ago
      In another part of the thread, a commenter linked the response from the department which pointed out that the definition of human subject research and even IRBs themselves are governed by federal statute. A bunch did go wrong in this case, but the actions of the IRB weren't one of them. They were acting exactly as directed by the law. The fault lies with the researchers who conducted the experiment, and perhaps broadly with the academic community for not establishing guidelines for ethical oversight of non-human subject research.
      - josefx 5y ago
        The definition of human subject is literally "a human a researcher collects information on", as far as I can find the law doesn't even try to over complicate that point. Claiming that researching the Linux review process does not involve human subjects seems to be willfully ignorant.
deleted 6y ago
[deleted]
ajross 6y ago
> Proactively punish the whole UMN does nothing to address the real issue here. Sorry, but academics using the presumption of good faith that their status brings to do black/gray hat security work using an open source project of critical social importance as a dupe is a very real problem on its own. And it absolutely requires a separate solution. Other academic fields treat research ethics as critically important. Something like would simply end the career of someone in medicine, for example. Maybe computer science should too. That the public kernel maintainers need more support and resources is a problem also. But the solution is different. We should be willing to discuss both and not use the latter as a way to make excuses for the nutjobs at Minnesota.
- temp8964 6y ago
  Publishing sting is not the UMN team's intervention. The Sokal affair [1] is probably the most famous one. There are many others too [2]. In the Grievance studies affair [3], three researchers submitted 20 fake articles. They must have wasted tons of works on review teams. 1. https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair 2. https://en.wikipedia.org/wiki/List_of_scholarly_publishing_stings https://en.wikipedia.org/wiki/List_of_scholarly_publishing_s... 3. https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair
  - ajross 6y ago
    There's a big difference between a publishing a fake article and trojaning the kernel.
    - whimsicalism 6y ago
      Not in terms of the reviewers time.
      - deleted 6y ago
        [deleted]
      - 6y ago
deleted 6y ago
[deleted]
dkersten 6y ago
Maybe it doesn't fix the underlying issues of overworked maintainers, but what the "hypocrite commits" team did was definitely wrong. I mean, a hacker breaking into a bank for research without explicit permission (i.e. contracts signed etc) would results in prosecution. You don't pull a stunt like that.
- wnevets 6y ago
  According the researchers they never actually committed any "hypocrite commits". If that is true, in your example it would accusing someone of a crime for discussing how to break into a bank for research
  - dkersten 6y ago
    From the article: In response, the UMN researchers posted an open letter apologizing to the community, followed a few days later by a summary of the work they did [PDF] as part of the "hypocrite commits" project. Five patches were submitted overall from two sock-puppet accounts, but one of those was an ordinary bug fix that was sent from the wrong account by mistake. Of the remaining four, one of them was an attempt to insert a bug that was, itself, buggy, so the patch was actually valid; the other three (1, 2, 3) contained real bugs. None of those three were accepted by maintainers, though the reasons for rejection were not always the bugs in question. So three submissions were malicious and buggy.
    - wnevets 6y ago
      There appears to be a disconnect. Their FAQ says otherwise. > We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in the email did not even become a Git commit in any Linux branch. None of the Linux users would be affected. The following shows the specific procedure of the experiment. [1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.... [1]
hinkley 6y ago
Burnout is a very, very real problem for volunteer organizations in general. There's also a middle ground where you are doing the most you can sustain. If you run a volunteer group where everyone thinks "I could do more" all the time then there are a lot of missed opportunities. However, when everyone thinks, "I can't do any more", then any additional insult on top of that becomes a much bigger deal, because it is now 'too much'. For the group's sanity you probably want some fraction to feel underutilized at all times. When 'too much' happens, they get an opportunity to do more, and that extra effort is immediately appreciated. But even then, sooner or later you'll have two or three dramas that overlap in time and energy and people will get grumpy. Has the kernel team been dealing with some other bullshit this last year? I mean, besides participating in the epidemic with the rest of us, which counts as at least 2 dramas. I think right now you still have to cut people a little slack for getting upset about things you think they shouldn't.
iforgotpassword 6y ago
I wish I could just start reviewing random patches that get submitted/accepted into linux, but it's a daunting task. Although I'm good at C, lacking good knowledge of the according subsystem in the kernel makes it impossible to find anything but very trivial errors, which usually get caught by static analyzers afaict. I'm afraid of introducing noise that costs maintainers even more time, by commenting that something might be wrong because "I think you cannot sleep here" or "I think this needs to be in the critical section" when I'm not 100% sure. The sad thing is that the Linux kernel isn't a small simple project, which manifests in many ways. I once debugged and fixed an issue in i915. Finding out where exactly the patch should go, which repo and branch it should be based on, how to format the commit message, etc took at least twice as long as finding and fixing the issue. You end up in some doc listing instructions for a dozen mail clients to configure them so they don't fuck up your patch when you send it. I guess that's why there is "git send-email"; at some point it was easier to add email sending capabilities to freaking git than to deal with email clients, because the whole process is so archaic. I wonder how many people get discouraged by this whole process and just give up. I'd like to think it's just filtering out low effort contributions, but I'm not really sure. Long story short, I think the approachability of the Linux kernel is pretty low and I believe there are a bunch of capable people who'd happily get involved in some way, but are discouraged by the high bar of entry. It's a far stretch from github pull requests and issues usability-wise.
finnthehuman 6y ago
>the real issue here One phrase that always drives up the wall is "the real issue here." There can be more than one issue. For anything complicated, there are going to be at least two issues at hand. Banning a University to get the attention of leadership to corral their research department is a valid response to one of the real issues here.
- temp8964 6y ago
  I don't see the university here is a real issue. Sting research is always uncommon, naturally it won't happen in many years (or ever). It's not like if you don't ban UMN, someone else from their CS department will do another sting on the Linux kernel... What is far far more important is the overwork problem of the kernel team. And malicious attack from bad actors is way more important threat than a improper research study.
CrankyBear 6y ago
It's a known problem, but getting kernel maintainers up to speed is Not an easy job.
slim 6y ago
Why are you presuming that the problem of trust would be solved by having more paied developers involved ? Usually a problem of trust is solved by having less people involved
splithalf 6y ago
How about the jokers at UMN make an effort to “help out”? Sheesh. If the publicly funded universities can’t lend a hand or do their fair share, what hopes have we that big tech will come in and save open source. Do the universities have money problems?
matheusmoreira 6y ago
> Proactively punish the whole UMN does nothing to address the real issue here. The real issue here is human experimentation without consent or even information. Why do they think they can run experiments on free software developers without even contacting them about it?
twirlock 6y ago
The Linux kernel knows more about how to collaboratively guarantee secure, stable code than you or the cs department at UMN does, by a long long shot. And guess what: banning bad actors regardless of whatever moralizing horseshit they're hiding behind is a big part of that.
ThinkBeat 6y ago
I am not sure if you by more contributions from the tech world means donate more money, or donate more patches / fixes etc. Big tech adds a constant stream of features / extensions to the kernel for some fairly obscure factors that are important if you are a big enterprise or cloud company but matters very little to regular people and regular companies. If you mean bigger monetary investment I agree. With the exception that with large donations come strings attached.

josefx 6y ago

> is not a situation created by the group of researchers. As far as I got from various threads on the issue the university has been involved in at least two major time wasters: * Breaking the trust model and forcing the maintainers to revert all already accepted patches on suspicion alone * Running a primitive static code analysis tool that got overeager with correcting impossible bugs and not marking the generated patches as such. > Proactively punish the whole UMN does nothing to address the real issue here. Proactive? The universities review board signed off on that mess, there is nothing proactive about banning a group of bad actors after the fact.

SiempreViernes 6y ago
Proactive probable refers to also banning the 6700 staff, along with the 46000 students, that are not part of the research project from ever contributing.
- monocasa 6y ago
  That staff and student body is governed by the same ethics board that allowed this to happen.
  - cycomanic 6y ago
    Facebook experimented on people (some might have even been kernel programmers), but Facebook programmers are still contributing to the kernel. Let's not even talk about all the other organisations who have done some pretty unethical things, I mean lots of corporations who have or still are violating the GPL (on Kernel code even) are allowed to continue to contribute to the kernel.
    - monocasa 6y ago
      They didn't experiment on the kernel process itself. And it would be counterproductive to set up the incentives so that people who aren't contributing in the first place can't.
    - pabs3 5y ago
      Do you have any examples of companies that are both still GPL violating and still contributing to Linux mainline?
  - seoaeu 5y ago
    Why is that a problem? Most kernel contributors aren't governed by any ethics board at all!
    - josefx 5y ago
      They probably work for some company and can you imagine the shit storm if it ever came out that Microsofts board of directors signed off on a Microsoft dev. actively submitting vulnerabilities into the Linux kernel?
      - seoaeu 5y ago
        An IRB and a board of directors aren't remotely similar. UMN actually does have a board of trustees but they've had literally nothing to do with this incident. The senior administration (and the CS&E department prior to the mailing list blowup) never even knew that this research was happening.
- zopa 6y ago
  Realistically, if a UMN student submits a patch from their personal, non-.edu email address, will anyone notice or care?
staticassertion 6y ago
> * Running a primitive static code analysis tool that got overeager with correcting impossible bugs and not marking the generated patches as such. This is not a major time waster. It happens constantly - students everywhere are doing this, and the patches are accepted constantly. Further, while Greg classified the entirety of the analyzers findings as being useless, he was incorrect.[0] > The universities review board signed off on that mess, there is nothing proactive about banning a group of bad actors after the fact. In terms of the researchers submitting malicious patches it was done without edu addresses, which means that banning them solves nothing. [0] https://lore.kernel.org/lkml/b43fc2b0-b3cf-15ab-7d3c-25c1f2a3e658@canonical.com/ https://lore.kernel.org/lkml/b43fc2b0-b3cf-15ab-7d3c-25c1f2a... In fact, most of the fixes fit into the bucket of "potentially useful".
- spacechild1 5y ago
  > students everywhere are doing this Which somehow makes it ok? The Linux team told UMN not to run experiments on the kernel anymore, and then a student comes along and does it again. Given his association with Kangjie Lu, he should have known better.
seoaeu 5y ago
> Breaking the trust model and forcing the maintainers to revert all already accepted patches on suspicion alone The kernel developers chose to do that out of a combination of spite and excessive caution. The actual malicious patches didn't even come from UMN email addresses. > Proactive? The universities review board signed off on that mess, there is nothing proactive about banning a group of bad actors after the fact. Why does everyone keep repeating this? The IRB declared that it wasn't human subject research. Tons of other IRBs would have done the same because they all use a standardized definition that hasn't been updated for interactions with online communities.
- josefx 5y ago
  > Tons of other IRBs would have done the same because they all use a standardized definition that hasn't been updated for interactions with online communities. I can imagine a paper from the 1940s showing how they checked the security precautions of baby food manufacturers by having a factory worker under their control randomly lace the produced baby food with arsenic to see if the manipulated batches got caught by an "abstract process" before they could enter general distribution. Yeah, nothing that went wrong in this case can be excused by "but the internets!!!".
  - seoaeu 5y ago
    In another part of the thread, a commenter linked the response from the department which pointed out that the definition of human subject research and even IRBs themselves are governed by federal statute. A bunch did go wrong in this case, but the actions of the IRB weren't one of them. They were acting exactly as directed by the law. The fault lies with the researchers who conducted the experiment, and perhaps broadly with the academic community for not establishing guidelines for ethical oversight of non-human subject research.
    - josefx 5y ago
      The definition of human subject is literally "a human a researcher collects information on", as far as I can find the law doesn't even try to over complicate that point. Claiming that researching the Linux review process does not involve human subjects seems to be willfully ignorant.

deleted 6y ago

[deleted]

ajross 6y ago

> Proactively punish the whole UMN does nothing to address the real issue here. Sorry, but academics using the presumption of good faith that their status brings to do black/gray hat security work using an open source project of critical social importance as a dupe is a very real problem on its own. And it absolutely requires a separate solution. Other academic fields treat research ethics as critically important. Something like would simply end the career of someone in medicine, for example. Maybe computer science should too. That the public kernel maintainers need more support and resources is a problem also. But the solution is different. We should be willing to discuss both and not use the latter as a way to make excuses for the nutjobs at Minnesota.

temp8964 6y ago
Publishing sting is not the UMN team's intervention. The Sokal affair [1] is probably the most famous one. There are many others too [2]. In the Grievance studies affair [3], three researchers submitted 20 fake articles. They must have wasted tons of works on review teams. 1. https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair 2. https://en.wikipedia.org/wiki/List_of_scholarly_publishing_stings https://en.wikipedia.org/wiki/List_of_scholarly_publishing_s... 3. https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair
- ajross 6y ago
  There's a big difference between a publishing a fake article and trojaning the kernel.
  - whimsicalism 6y ago
    Not in terms of the reviewers time.
    - deleted 6y ago
      [deleted]
    - 6y ago

deleted 6y ago

[deleted]

dkersten 6y ago

Maybe it doesn't fix the underlying issues of overworked maintainers, but what the "hypocrite commits" team did was definitely wrong. I mean, a hacker breaking into a bank for research without explicit permission (i.e. contracts signed etc) would results in prosecution. You don't pull a stunt like that.

wnevets 6y ago
According the researchers they never actually committed any "hypocrite commits". If that is true, in your example it would accusing someone of a crime for discussing how to break into a bank for research
- dkersten 6y ago
  From the article: In response, the UMN researchers posted an open letter apologizing to the community, followed a few days later by a summary of the work they did [PDF] as part of the "hypocrite commits" project. Five patches were submitted overall from two sock-puppet accounts, but one of those was an ordinary bug fix that was sent from the wrong account by mistake. Of the remaining four, one of them was an attempt to insert a bug that was, itself, buggy, so the patch was actually valid; the other three (1, 2, 3) contained real bugs. None of those three were accepted by maintainers, though the reasons for rejection were not always the bugs in question. So three submissions were malicious and buggy.
  - wnevets 6y ago
    There appears to be a disconnect. Their FAQ says otherwise. > We did not introduce or intend to introduce any bug or vulnerability in the Linux kernel. All the bug-introducing patches stayed only in the email exchanges, without being adopted or merged into any Linux branch, which was explicitly confirmed by maintainers. Therefore, the bug-introducing patches in the email did not even become a Git commit in any Linux branch. None of the Linux users would be affected. The following shows the specific procedure of the experiment. [1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.... [1]

hinkley 6y ago

Burnout is a very, very real problem for volunteer organizations in general. There's also a middle ground where you are doing the most you can sustain. If you run a volunteer group where everyone thinks "I could do more" all the time then there are a lot of missed opportunities. However, when everyone thinks, "I can't do any more", then any additional insult on top of that becomes a much bigger deal, because it is now 'too much'. For the group's sanity you probably want some fraction to feel underutilized at all times. When 'too much' happens, they get an opportunity to do more, and that extra effort is immediately appreciated. But even then, sooner or later you'll have two or three dramas that overlap in time and energy and people will get grumpy. Has the kernel team been dealing with some other bullshit this last year? I mean, besides participating in the epidemic with the rest of us, which counts as at least 2 dramas. I think right now you still have to cut people a little slack for getting upset about things you think they shouldn't.

iforgotpassword 6y ago

I wish I could just start reviewing random patches that get submitted/accepted into linux, but it's a daunting task. Although I'm good at C, lacking good knowledge of the according subsystem in the kernel makes it impossible to find anything but very trivial errors, which usually get caught by static analyzers afaict. I'm afraid of introducing noise that costs maintainers even more time, by commenting that something might be wrong because "I think you cannot sleep here" or "I think this needs to be in the critical section" when I'm not 100% sure. The sad thing is that the Linux kernel isn't a small simple project, which manifests in many ways. I once debugged and fixed an issue in i915. Finding out where exactly the patch should go, which repo and branch it should be based on, how to format the commit message, etc took at least twice as long as finding and fixing the issue. You end up in some doc listing instructions for a dozen mail clients to configure them so they don't fuck up your patch when you send it. I guess that's why there is "git send-email"; at some point it was easier to add email sending capabilities to freaking git than to deal with email clients, because the whole process is so archaic. I wonder how many people get discouraged by this whole process and just give up. I'd like to think it's just filtering out low effort contributions, but I'm not really sure. Long story short, I think the approachability of the Linux kernel is pretty low and I believe there are a bunch of capable people who'd happily get involved in some way, but are discouraged by the high bar of entry. It's a far stretch from github pull requests and issues usability-wise.

finnthehuman 6y ago

>the real issue here One phrase that always drives up the wall is "the real issue here." There can be more than one issue. For anything complicated, there are going to be at least two issues at hand. Banning a University to get the attention of leadership to corral their research department is a valid response to one of the real issues here.

temp8964 6y ago
I don't see the university here is a real issue. Sting research is always uncommon, naturally it won't happen in many years (or ever). It's not like if you don't ban UMN, someone else from their CS department will do another sting on the Linux kernel... What is far far more important is the overwork problem of the kernel team. And malicious attack from bad actors is way more important threat than a improper research study.

CrankyBear 6y ago

It's a known problem, but getting kernel maintainers up to speed is Not an easy job.

slim 6y ago

Why are you presuming that the problem of trust would be solved by having more paied developers involved ? Usually a problem of trust is solved by having less people involved

splithalf 6y ago

How about the jokers at UMN make an effort to “help out”? Sheesh. If the publicly funded universities can’t lend a hand or do their fair share, what hopes have we that big tech will come in and save open source. Do the universities have money problems?

matheusmoreira 6y ago

> Proactively punish the whole UMN does nothing to address the real issue here. The real issue here is human experimentation without consent or even information. Why do they think they can run experiments on free software developers without even contacting them about it?

twirlock 6y ago

The Linux kernel knows more about how to collaboratively guarantee secure, stable code than you or the cs department at UMN does, by a long long shot. And guess what: banning bad actors regardless of whatever moralizing horseshit they're hiding behind is a big part of that.

ThinkBeat 6y ago

I am not sure if you by more contributions from the tech world means donate more money, or donate more patches / fixes etc. Big tech adds a constant stream of features / extensions to the kernel for some fairly obscure factors that are important if you are a big enterprise or cloud company but matters very little to regular people and regular companies. If you mean bigger monetary investment I agree. With the exception that with large donations come strings attached.