21 ms·

Open letter from researchers involved in the “hypocrite commit” debacle

• fastglass 6y ago
  'yo, my bad'
• haswell 6y ago
  I'd like to give them the benefit of the doubt, but this is written like an apology they know they must write. It does not come across as apologetic. It comes across as rationalization veiled as an apology, and it doesn't sit well with me. I hope I'm just being overly sensitive here.
  • justinjlynn 6y ago
    Indeed. This letter is an attempt to justify and rationalise their actions. Essentially, it amounts to saying "we're sorry you were offended and felt hurt by our legitimate work but we had no choice but to lie to you and unethically experiment on you without your consent or we wouldn't have been able to do it". Their statement is not an actual apology, even if it is phrased in the language of apology, and it is an excellent example of what not to write if you are seeking forgiveness. The core issue here is that the system under which they were working, and the researchers themselves, did not consider this work to amount to unethical human experimentation - even though the work directly involved infiltrating and exploiting humans and human social systems. There is no mention of this nor of any substantial desire to understand or address this systemic failing and, until there is an unconditional acceptance of responsibility for the harm caused and a real actionable change for the better with accountability, I don't see how we can consider this matter to have been apologised for.
    • rajup 6y ago
      Just curious, and I'm genuinely asking as someone who thought the letter seemed well-intentioned: what should they have put in the apology letter?
      • spectralblu 6y ago
        > we did that because we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches. This bit just reads like "sorry you're upset". I would have expected something along the lines of "sorry, we should have asked the maintainers and we understand why it was wrong not to", instead of "sorry, we didn't ask the maintainers, but this is why we didn't".
        • wang_li 6y ago
          "It's just a prank dude!" Legitimately these folks have some kind of personality defect. They observed something that everyone already knew and then decided to act on it and pretend they were doing something interesting and new rather than just being twats. They should be treated exactly as should be based on their actions, regardless of their claims of being researchers. This would be a perfect cover for being on the take from a government agency. They should be investigated to ensure they aren't actual malicious actors. It takes almost no imagination to think of hundreds of "security vulnerabilities" in the world, yet the vast majority of people realize that it's dumb to act on those observations.
      • R0b0t1 6y ago
        They should have addressed why the kernel maintainers felt the way they do. As it is now they are trying to rationalize their actions instead of showing they have made an attempt to understand why their research was questionable.
      • justinjlynn 6y ago
        > We are sorry for the harm we've caused by our unethical experimentation involving members of the Linux community. In accordance with community standards and a desire to avoid benefiting from our ethical failure, we have requested the immediate retraction of the papers and other published work resulting from this unethical experimentation. We are also working together with our institutional leadership to address the systemic and individual failings which permitted this unethical research to take place. As a result of this, we will publicly prepare a report regarding the actions taken and potential systemic changes which will ensure these actions do not reoccur and further harm is not caused. Out of an abundance of caution and because experience has shown our judgement, as is, to be questionable, no further research similar in nature to the research in question will be undertaken by this institution or these researchers until proper community consultation has taken place regarding the report and proposed changes to systemic review are implemented. Once again, we take full responsibility for any harm we have caused and we will work to ensure that such harm never occurs again. We hope that, at some point, we may regain your trust and confidence and, if you'll permit us, to work alongside you. [signed personally, by every involved researcher, professor, IRB member, and all of the relevant university leadership including the board] Something like that would do nicely, in my opinion. Then they'd have to absolutely follow through on it. Ideally, they'd also ask involved members of the community so harmed, among other experts, to independently review and watch the process to ensure no further harm is being done, progress is made, and public accountability is kept.
    • blondin 6y ago
      maybe i am being very naive here but that letter read like a sincere apology to me as well.
      • hn8788 6y ago
        I think if it were a sincere apology, they would have apologized for what they did, not just that there were undesirable side effects of their research. The letter is a typical "I'm sorry you got mad" apology that people make when they're forced to, but don't think they did anything wrong.
    • astrange 6y ago
      People love to analyze apologies after the fact, but it seems totally unfair to me. Once someone has said an apology you can take the text of it and turn it into anything you want and say it proves they were lying.
      • deagle50 6y ago
        Yep, it's called accountability. Would you prefer people acted without regard for others knowing that magic words can be spoken after the damage is done?
        • astrange 6y ago
          I would prefer you not go around saying someone is "apologizing wrong" because you've found the magic formula to turn all apology text into "I'm sorry you feel bad". There's no point in making a statement like that the original person can't respond to, anyway.
          • deagle50 6y ago
            I get what you're saying and I don't want that that either. But I also don't think a simple apology that doesn't attempt to justify one's actions can be so easily flipped.
      • shkkmo 6y ago
        An good apology needs to demonstrate an understanding of why what was done was wrong and ideally includes concrete steps that will prevent anything similar from happening again. You seem to be implying that one apology is as good as another and that is absolutely not true. To me is is absolutely unfair to expect that people will treat a zero effort "sorry your feelings were hurt" apology the same as a deeply heartfelt apology by someone who feels deep remorse. By it's nature, an apology should be an act of making amends to a person your have wronged. If the words of your apology don't do that, you have failed, even if you had good intentions. If the author of the apology isn't a native speaker of english, this seems like a document of sufficient significance to merit asking a friend or colleague for a proofread. (Which you should do in a case like this even if English is your first language.)
    • bluenose69 6y ago
      Agreed. This is like a psychologist apologizing for traumatizing children by testing a hypothesis that telling them scary stories before bedtime would give them bad dreams. "Oh, but it's important to learn what scares kids, for the greater good" just doesn't cut it. Software is complex. Wasting maintainer's time is harmful to the maintainers and to those who use the software. If the goal is to explain what sort of exploits to be on the look-out for, just write the paper without doing the exploits. As for the letter, their goal is to recover their reputations, not to apologize. I suppose a lawyer helped them in spots, but there is still a certain truth that shines through: they want to continue on doing this sort of thing, because they are oh-so-clever and their work is oh-so-valuable. Nice try, but the academic community may be better off without these folks doing this sort of research and influencing students to follow their methods.
  • johnklos 6y ago
    I agree, but let's give them a little extra benefit of the doubt. I thought as I was reading it that it seemed stilted and forced, then I wondered if the author(s) don't speak / write English as a primary language. I'll be interested to see how they react to feedback / responses.
    • jimsug 6y ago
      > We just want you to know that we would never intentionally hurt the Linux kernel community and never introduce security vulnerabilities. Our work was conducted with the best of intentions and is all about finding and fixing security vulnerabilities. Either this is a bald-faced lie or they didn't think through the implications of their research, neither of which bode well for their trustworthiness in future. However, if it's the latter, then they possibly deserve additional leeway. On the other hand, I wonder whether this letter would have eventuated without the ban - if not, then they've likely been forced to write it, in which case any remorse is either selfish (i.e. they've been professionally reprimanded and feel bad about getting caught, or making such an egregious error in judgement), or hollow. > I agree, but let's give them a little extra benefit of the doubt. I thought as I was reading it that it seemed stilted and forced, then I wondered if the author(s) don't speak / write English as a primary language. https://www-users.cs.umn.edu/~kjlu/ https://www-users.cs.umn.edu/~kjlu/ It looks like you might be right about this, but I think that reading it as stilted and forced is still accurate.
    • lolinder 6y ago
      The wording is definitely stilted and forced, but the content is also wrong for an apology. They spend a lot of time justifying their actions, and close with "this has been painful for us too". This may also be cultural, but it's overly self-centered for an apology.
  • CharlesW 6y ago
    I was just about to post: "This is a great apology." Context is everything, of course. I think you're right that it's tainted by the fact that they absolutely did not have a choice, and coming from people who've deceived the same tribes they're now trying to apologize to.
    • Covzire 6y ago
      And what of the merits? If the previous 190 patches were indeed legitimate it strikes me as overly vengeful to pull them in a "punish the son for the sins of the father and the father for the sins of the son" kind of way.
      • jimsug 6y ago
        Agree that it could be seen as excessive, but this was not just in response to these commits, but also to the university's failure to act once they were informed of them. They university, and indeed, the researchers, only acted once the ban was imposed - i.e. they were getting away with it without reputational damage, and only once it was stained did they actually do anything.
      • andreareina 6y ago
        Or it's not in vengeance, but caution after the researchers have demonstrated that they put their needs above the community's. When the presumption of innocence is lost, it's appropriate to revert the patches until they are known to be good. ETA: GKH on reverting the patches[1] > This patchset has the "easy" reverts, there are 68 remaining ones that need to be manually reviewed. Some of them are not able to be reverted as they already have been reverted, or fixed up with follow-on patches as they were determined to be invalid. Proof that these submissions were almost universally wrong. [1] https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh@linuxfoundation.org/ https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh...
        • x3__ 6y ago
          counterpoint: The LF Technical Advisory Board is taking a look at the history of UMN's contributions and their associated research projects. At present, it seems the vast majority of patches have been in good faith, but we're continuing to review the work. Several public conversations have already started around our expectations of contributors. https://lwn.net/Articles/854064/ https://lwn.net/Articles/854064/ What the authors did is wrong, so did GKH actions. Let's be honest in calling BS out.
  • csours 6y ago
    As of now, it's just words on a page. Let's see how their actions back up those words.
  • sneak 6y ago
    They didn’t have to write it. The first two sentences seem sincere. This reads as a real apology.
    • tartoran 6y ago
      Exculpatory apology
    • pseudalopex 6y ago
      They didn't have to write it if they wanted the whole university to stay banned.
  • tmotwu 6y ago
    Yeah, it's not a great apology. I would say as far as justifications go, I think it's a certain level of depth expected from academics, you shouldn't overthink it. As far as giving benefit of the doubt, it's likely not done out of malice in the first place. Just a combination of poor reasoning and doesn't exactly clear up if they even considered alternatives to control their variables. Unfortunately, it seems like they were already given the benefit of the doubt from their first offense, but did not take any lessons away from that, so it would be understandable if the maintainers kept their ban.
    • sombragris 6y ago
      I don't think they had malice, but the breach of elemental ethics is just appalling. They show no remorse for being trusted and abusing that trust and good faith. They show no remorse for using human beings as involuntary guinea pigs. In sum, they show not remorse for doing wrong.
      • tmotwu 6y ago
        For me, it's not learning the lesson the first time around. I completely agree their experiment is unethical. However, it's not actually clear cut to most researchers the ethical bounds of their work, especially for study papers that's never really been explored before. Ethics in of itself is largely a active subtopic for many areas in CS, not only security research. AI is one area where qualifying potential harm to human beings remains largely controversial. Ask any ML scientist, and they'll tell you that determining the ethics of a project is not their responsibility.
        • sombragris 6y ago
          Great insight. But as for the ML scientists you mention, ethics _is_ their responsibility. We all are ethical beings.
        • shkkmo 6y ago
          I could not disagree more. The ethics around research that involves deception have been pretty well established and are are several good comments here explaining them. Every scientist is personally respsible for the ethics of the research they conducts. Full Stop, no caveats allowed. If your research is in an area where the ethics are controversial or grey, that means your need to spend MORE time considering the ethics of your research, not that you get a free pass from being responsible. If a any scientist espouses the opinion that determining the ethics of their projects is not their responsibility, they should be permanently barred from recieving grant money.
  • slothtrop 6y ago
    > It does not come across as apologetic. The words are there. It's not up to us to decide if they're "genuine". No one's a mindreader. The tendency to view apologies as fake seems more often to reflect how harshly we view the one making it.
    • haswell 6y ago
      Their opening statement has the most classic of all non-apology tactics: "We're sorry for any harm we caused". Maybe this wasn't their intent, but this is one of the oldest "say the words without having to mean anything" tactics in the book. It's like telling your partner "I'm sorry if I hurt you". No, you hurt them. Apologize for hurting them.
      • cgriswald 6y ago
        An apology should be for actions taken, not for how the other party felt about it. Acknowledging those feelings is important, but taking credit for those feelings takes agency away from the other person.
        • ddingus 6y ago
          That loss is accountability in action. Agency returns as trust does.
        • foldr 6y ago
          I don’t think that really makes sense. Often in a personal relationship the “action taken” will be innocuous considered in itself. But you may have done it knowing full well that it would hurt the other person’s feelings. In a case where the person’s feelings are totally irrational and you had a strong independent justification for taking the action, it does get more complicated. In general, though, I think it’s a huge mistake to assume that you should never apologize for hurting someone’s feelings.
          • cgriswald 6y ago
            I don't think you can take the action outside of the context in which it is performed. If you take action knowing another person is likely to feel hurt, are you... not sorry you took that action? Regardless, you can't genuinely apologize for things that aren't in your control. (You can and should empathize and sympathize with their feelings about it.) It may be a convenient fiction at times, but there are many downsides to this type of boundary blurring. I think you do the recipient a disservice by implying they have no agency—especially if they believe you. You also face the danger that your apology seems insincere or you misjudged their feelings, which could actually make matters worse.
  • roenxi 6y ago
    The kernel community isn't a spurned lover or anything; the apology just has to admit fault, promise not to do that again and be truthful about it. There is no need for the researchers to be in any specific state of mind or even apologetic as long as they don't try to submit bad faith patches again. Really the only reason there is even a need for a public apology is to signal how much pressure they are under so other people think twice before doing the same silly thing. Otherwise they could just apologise to the maintainers directly.
  • jf 6y ago
    I don't think you're being overly sensitive. To me, it's always a distinct sign that an apology isn't sincere when it hedges apologizing for "any harm" that they might have caused. In my mind, a sincere apology clearly acknowledges harm that has been caused and apologizes for that.
  • Igelau 6y ago
    Social media insanity has greatly reduced our capacity to accept public apologies at face value. There is always a seething mob that is ready to question the sincerity of the apology once there are no more demands left to be made.
    • ddingus 6y ago
      Not really. Well constructed apologies are hard to seethe over. This one could be better. Many, myself included, believe it should have been.
  • tlb 6y ago
    I agree. There are weasel-words throughout, starting with apologizing for "any harm". When you've been told what the harm was, don't apologize for "any harm" you might have caused. You should start an apology by demonstrating that you have listened and understood exactly what harm you caused. Listing the harm you caused is also an important part of the public record, so that anyone else considering a similar scheme in the future can see what consequences were agreed to have happened the last time.
  • sullyj3 6y ago
    There's no apology they could have written that would not result in some people saying it was insincere.
  • spfzero 6y ago
    You're not. It does not come off as apologetic at all. It is clearly someone writing as if they wanted to go back in time and act like they legitimately were caring, thoughtful people who made a mistake. In fact, they thought this was worth trying and got caught.
• 29athrowaway 6y ago
  I was reading this thread... https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh@linuxfoundation.org/ https://lore.kernel.org/lkml/20210421130105.1226686-1-gregkh... The Linux kernel team had to revert changes in over 200 files. They will also have to go back and review hundreds of commits to make sure they do not contain any malicious code. The technical effort in addition to all the time spent dealing with the backlash of this is a waste of time for a lot of people doing important work that benefits everyone. So, what else can be said... Play stupid games, win stupid prizes. To Mr. Lu, Wu, and Pakki: enjoy your ban and worldwide press coverage.
  • CRConrad 6y ago
    > The Linux kernel team had to revert changes in over 200 files. No, they didn't have to. That seems like quite a bit of an over-reaction by mr Kroah-Hartman.
• shaggyfrog 6y ago
  This is a textbook non-apology and is just further reason to keep this researcher (and whoever else was involved) banned from ever contributing again.
• xbar 6y ago
  I find it false. There were months of opportunity for contrition. They seemed a little too proud of their paper for Oakland'21, and this note comes much too late, for the message to be sincere.
• ncal 6y ago
  Only "sorry" because you got yourselves banned.
• sweettea 6y ago
  To summarize: -they apologize for the three patches in 2020 -they claim asking permission would have defeated their research -they claim the other 200 were legitimate patch attempts (a sample of which were, from my personal reading, from 'innocuous but useless' to 'slightly harmful'). Hard to believe, but plausible.
  • dane-pgp 6y ago
    > they claim asking permission would have defeated their research But surely this proves malice aforethought. If you interact with someone under false pretences to deliberately mislead them, then you are effectively lying to them. In fact, if you are doing so in order to receive something of value from them (such as their time reviewing your code) then it could even be seen as fraud. Did they tell their IRB that their research involved deception? Before declaring the project exempt from oversight, the IRB should have required that the researchers answer a question like "Does your research involve interacting with people who are not aware of your project and who would act differently if they were aware?" Until the IRB process at the university contains at least this level of protection against such ethical failings (not just in Computer Science research but across all areas of study) I think it is right for the kernel team to refuse to interact with members of that institution.
    • schwede 6y ago
      I would like see their research proposal to answer some of these questions. That probably won’t happen though.
• asaph 6y ago
  Is this post actually research for an upcoming paper on apologizing to the open source community?
• sombragris 6y ago
  I really appreciate the apology and as they stated, its unconditional nature. Good. However, I find something very problematic. This quote shows it: "We have learned some important lessons about research with the open source community from this incident." This is something I don't like. This is not something about "research with the open source community". If anything, they should have learned something about treating human beings as persons and not as involuntary guinea pigs. They should have learned something about not breaking anyone's (not only any open source community) good faith and trust, and respecting that. They behaved like jerks and they still cannot see that.
  • at_a_remove 6y ago
    Going further, "with" seems like the wrong preposition. "On" seems more accurate.
    • neltnerb 6y ago
      It would make sense if they were talking about mice, in which case "on" is still more accurate but "with" is probably correctly assumed to be equivalent. Research with humans involves them being on the author list...
    • ENOTTY 6y ago
      The professor is not a native English speaker. I’d cut him some slack in minor issues with his word choice.
  • dylan604 6y ago
    >They behaved like jerks and they still cannot see that. They just come off as disconnected academics. Samething with the experiments with algorithms from Facebook. They are so wrapped up in what they are doing, they no longer see the "users" as human beings. There are some times this is almost required to survive like being an ER doctor. If you lose that many people, it could be debilitating unless you were able to disconnect. That's far and away much different than these robots.
  • true_religion 6y ago
• january_fanatic 6y ago
  Looking forward to the follow-up paper, "Open Source Insecurity: Concealing Vulnerabilities via Hypocrite Apologies"
  • jraph 6y ago
    The apology reads sincere to me even if it could be better, but this is for me why it might be very difficult for the Linux community to recover trust in these guys. This, and the previous mail that I found pretty insulting too and seemingly written in bad faith but I'd give the benefit of the doubt, one can react badly to a difficult situation.
    • neatze 6y ago
      I am skeptical if they are sincere, because they did not retracted IEEE paper.
• neatze 6y ago
  > As many observers have pointed out to us, we made a mistake by not finding a way to consult with the community and obtain permission before running this study; we did that because we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches. This shows one thing to me, clearly, they still don't understand how such studies are conducted. not authorized penetration testing is illegal, to a very large degree. For example, as far I understand, and as many pointed out; you get permission from members for an study without specifying time or any details of a study, then conduct study in 6 month.
• voldacar 6y ago
  Huge "I'm sorry I got caught" vibe. And it still refers to this malicious annoying behavior as "work" or "research", lmao
  • tmp538394722 6y ago
    I mean, there’s all kinds of terrible unethical research. What makes you think this isn’t research?
    • x3__ 6y ago
      Their paper got accepted to S&P 2021. That is the top conference in computer security in case one is not aware. Not justifying what they did but this apparently is still research.
      • mewse 6y ago
        This paint on canvas got sold and therefore it is Art.
        • x3__ 6y ago
          What is that supposed to mean? Conference publications are for research. I pulled this from the website of Oakland 2021 Since 1980 in Oakland, the IEEE Symposium on Security and Privacy has been the premier forum for computer security research, presenting the latest developments and bringing together researchers and practitioners. We solicit previously unpublished papers offering novel research contributions in any aspect of security or privacy. Papers may present advances in the theory, design, implementation, analysis, verification, or empirical evaluation and measurement of secure systems. https://www.ieee-security.org/TC/SP2021/cfpapers.html https://www.ieee-security.org/TC/SP2021/cfpapers.html
          • mewse 6y ago
            You’re defining whether something is “research” not based upon what it is or what it contains, but just based upon whether or not it’s been published. I was pointing out that this is like defining whether or not something is “art” not based upon the work itself, but instead based solely upon whether somebody bought it. Incidentally, my recipe for apple pie was printed in a newspaper, and so therefore my recipe is “news”.
• RcouF1uZ4gsC 6y ago
  Does the response of the Linux community seem like the response of some corporations when security vulnerabilities are disclosed? In each case, a vulnerability was disclosed. With Linux being used everywhere you can be sure intelligence services etc are likely sending in bad patches. If the Linux kernel requires only patchers with good intentions, and doesn’t have other means of catching this stuff, we are screwed.
  • dane-pgp 6y ago
    > the response of some corporations when security vulnerabilities are disclosed There's a big ethical difference between trying to exploit a piece of commercially produced software, and trying to exploit the time and actions of humans who are producing software which is given away for free.
    • shash 6y ago
      Also, this wasn’t a vulnerability found in existing code that was disclosed. This was an attempt to introduce several of them in the form of innocent looking commits. I don’t think the free vs commercial aspect is the main issue here; lots of kernel devs are paid for their work after all...
      • dane-pgp 6y ago
        > Also, this wasn’t a vulnerability found in existing code that was disclosed. You're right that the OP's analogy breaks down if the researchers were unsuccessful in getting their malicious patches accepted, because then there is arguably no vulnerability to report, and OP said "when security vulnerabilities are disclosed". Steelmanning that analogy, though, what the researchers were doing was "probing for possible vulnerabilities", which some vendors also complain about, especially if the target is an online service rather than software running locally on the researcher's machine. In that case, the main flaw in the analogy is still the difference between exploiting software and exploiting humans, so I probably should have focused on that. Nevertheless, there is a small ethical difference in some circumstances between software that is bought and software which is freely downloadable (regardless of the licences involved), since if you paid for something which is defective then you might deserve a refund.
• mikl 6y ago
  A BS non-apology is not enough. These assholes should at the very least be reprimanded by the University of Minnesota for unethical research practises (psychological experiments on humans without their consent) and bringing the whole institution in disrepute.
  • whatupmiked 6y ago
    The reputational damage to the department is staggering.
• SquibblesRedux 6y ago
  Whether it is appropriate or not, the Linux kernel is used in many mission critical and essential services. One could very convincingly argue that the open source Linux kernel is a vital part of mission critical infrastructure found all around the world. An apology, likely written under duress, for an inappropriate research method or for consuming the time of maintainers (either volunteer or paid) does not address the most egregious transgression: One does not conduct security research or experimentation on mission critical infrastructure without the informed consent of the persons maintaining or responsible for said infrastructure. The stark difference between white (or even gray) hat and black hack security operations is whether those operations are done with the informed consent of the owners of the system(s) being manipulated. To knowingly and deliberately attempt to manipulate critical infrastructure without the informed consent of the persons responsible for that infrastructure is, in my eyes, clearly unethical black hat hacking. It does not matter how many resources were expended nor does it matter if there was no personal injury or destruction of property. A quiet, relatively uneventful outcome of a black hat operation is nothing more than a dose of good luck. There are many other circumstances in which this sort of clandestine operation would be obviously unacceptable. Consider if a researcher wished to surreptitiously tamper with vaccine supply chains, automotive or aircraft supply chains, fuel distribution supply chains, chemical manufacturing supply chains, etc. Even if the researcher claimed to have the best of intentions, and had a written plan to carefully back-out or otherwise reverse (or prevent from going to production) the intentionally flawed manipulations or modifications of critical supply chain components, I think very few people would consider the research ethical. There are an enormous number of issues, defects, and mishaps that occur even when a group of people tries their best to implement and maintain critical systems. When a clandestine operator decides to interfere, using whatever justification, then reliability of the system can be expected to suffer. I do not believe the written apology addresses the underlying problem, which seems to be that the University of Minnesota does not exercise sufficient governance over the researchers in its employ.
  • shash 6y ago
    To be fair, you wouldn’t expect these individual researchers to address the governance question. That should come from the University or the department in question.
• neatze 6y ago
  We are sorry, but we are going ahead and present our paper on IEEE ? Since when it is a ok to acquire data through unethical means and then publish it in a journal ?
• createdapril24 6y ago
  There is a major error made by the research group. It starts and ends here: "we did that because we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches." I am a Red Teamer and work with companies to understand how their detective/preventative/recovery controls and processes are working. Here's how you resolve this: You work with maintainers to get their coordination on the research. You work out a mechanism to prevent submitted patches from being merged (e.g. maintainers are notified before bad patches accepted by code review processes are merged). You do not tell them when the patches are coming. You do not tell them which identities are going to be used for the patches (e.g. from which email addresses). You do not tell them which area of code will be targeted. You set rules and time bounds for the study. You wait some amount of time before submitting such patches (weeks to months). Realistically this is all that's needed. If hypersensitive, set this up earlier and let it bake longer. At this point, you submit patches from a variety of addresses (probably not associated with your university - it is easy to create many such identities). You also can coordinate with other researchers, universities, and companies to submit patches under identities as needed. You also study submitting from yandex, gmail, .cn and other email addresses (because isn't that interesting to know?). The premise that there's some ultimatum between working with the community and performing the research is on its face incorrect. This is either ignorance or laziness on behalf of the researchers. Clearly, they hadn't taken the time to work with the community to work out an approach that could be mutually acceptable.
  • drewzero1 6y ago
    This is how it should've been done. Like a war game (at least,as depicted in movies like Periscope Down), the organizers of the study and the project maintainers need to agree on (and be aware of!) bounds of the study, without necessarily knowing all of the details.
  • ComodoHacker 6y ago
    >I am a Red Teamer and work with companies OK, how do you test SocEng vectors? Do you obtain consent and coordinate with every employee that might be targeted to receive your e-mail? >You also study submitting from yandex, gmail, .cn and other email addresses No, the point was submitting from a known and respectable entity, which might affect the level of scrutiny. They weren't testing a whole patching process, but a specific human component of it.
• h2odragon 6y ago
  Wouldn't it be better if the kernel had a governance committee and such so that innocent mistakes like this could be smoothed over and the personalities over-reacting on the basis of nothing but years of expertise gained at efforts that benefit the public at large could be dismissed because of their personality problems? Maybe we can find or make up something unrelated but unsavory about these characters objecting to those pissing in the font of Linux... That's the normal way these things are handled now, right? (scorching sarcasm intended)
• StefanKarpinski 6y ago
  The thing I’m still missing is a detailed explanation of what the heck was going on with the recent bogus commit that triggered the banning. Supposedly the “hypocrite commit” research was all done in 2020 and is now in the past. So what was going on with this latest bad commit? The student who submitted it claimed it was generated by a static analysis tool, which kernel maintainers have plausibly called bullshit on. Was that student lying? If so, what were they actually doing? If not, it seems absolutely necessary at this point to prove that they were telling the truth by publishing how that commit was produced, including the tool’s source code and how it was invoked. Even if the campaign to intentionally introduce security flaws was over in 2020, the more recent issue is that the same research group submitted an apparently intentionally incorrect patch and then seems to have lied about its provenance and why it was submitted. Until that’s cleared up any kind of apology feels premature and impossible to evaluate. How can anyone decide if an apology is sincere without understanding what was done?
  • shash 6y ago
    The latest patch adds a null check around a call to gss_release_msg. The commit message says “ The patch adds a check to avoid a potential double free.” According to other people in the conversation, this is already taken care of by reference counting (https://lore.kernel.org/linux-nfs/20210407153458.GA28924@fieldses.org/#t https://lore.kernel.org/linux-nfs/20210407153458.GA28924@fie... ) and the patch apparently does nothing. The commit doesn’t reference any specific tool they’re using, or any bug they faced. Looks innocuous, but I guess past behaviour from this group left enough of a bad taste for Greg KH to be suspicious.
    • StefanKarpinski 6y ago
      A later message claims (outraged at being accused of submitting intentionally broken code to the kernel, despite having previously done exactly that) that the patch was generated by a static analysis tool. Ok, what tool? How did you run it? The message where he claims this has since been deleted (by who? Edit: probably never sent to the list, see below) but here is a message from Greg KH which quotes it: https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah.com/#r https://lore.kernel.org/linux-nfs/YH%2FfM%2FTsbmcZzwnX@kroah... You can see that Greg KH is skeptical that the patch was generated by a tool. It’s also unclear to me if the patch is actually harmless or not. It would be good to get some definitive clarity on that but the lkml discussion of it seems inconclusive. I would add that the student’s tone in this message feels really familiar as an open source maintainer: this is the tone of someone doing something they know is bad being called out on it and trying to deflect with outrage. The last time I got that tone, it was someone who had created multiple sock puppet accounts to try to discredit my project and force moderators to reverse or apologize for a (completely justified and mild) moderation action. So I can’t help but feel that something fishy is going on here and the UMN research group still isn’t being honest about it.
• dc396 6y ago
  Supply chain attacks are the security buzzthreat of day, at least since Solarwinds. Mucking about with the Linux kernel would be a really juicy target for nation-state actors. If you wanted to study this risk, how would you go about doing it? I haven't done kernel work since BSD4.3 so my opinion isn't particularly interesting. With that said, I would agree that the researchers were naive and their approach has caused collateral damage. However, I'd also argue that the (apparent) topic of research is quite valid. There are a lot of extremely well funded adversaries who are quite talented at obfuscation. "Be careful out there."
  • _aleph2c_ 6y ago
    Agreed, and as such the Linux community should counter signal to gain deterrence. If you do this kind of thing and you get caught, a simple apology is not enough: you get banned. It's a kind of a game theoretic approach.
  • shkkmo 6y ago
    > If you wanted to study this risk, how would you go about doing it? The main factor in doing this ethically is obtaining consent from your research subjects. I believe that some Red Teams test precisely these sorts of security mechanisms in commercial software projects and there are solid comments else discussing procedures they use.
  • detaro 6y ago
    The question always is, could you have learned the same thing with less "invasive" methods? And I think there's a good argument here that that's the case: I.e. the researchers first looked at bugs in the kernel and how their history, finding that such scenarios happened by accident. They could likely have gotten bugs past reviewers they had told that they were trying something like this. lots of options, so going for the most extreme option to "find" something that's not exactly a new or widely disagreed idea is quite unnecessary.
• waheoo 6y ago
  Leave them banned.
• gadf 6y ago
  Contrarian thought: Linux Kernel Community, forever above board, pure, pristine and virtuous, is perhaps eager to cast the first stone against these researcher / activists in order to clean their own clothes by dirtying others, or deflect attention from their own failings? Quick search through the Lore for "problematic" phrases: - site:lkml.org intext:moronic ~ 46 results - site:lkml.org intext:idiotic ~ 278 results - site:lkml.org intext:fucking (or f*cking) ~ 350 results In future, all communities, and all participants, should only be headed by the most ethical, upstanding, righteous, virtuous, pure, pristine, moral, kind, fair, balanced, unbiased, and wise humans, and these traits must be established through rigorous vetting and investigations. Any violations should require extensive and genuine apologies, the genuineness of which must be assessed through committees convened for that purpose, and apologies must be examined from every possible angle to determine if they do really feel genuine contrition or are just faking it. In all circumstances the omniscience of the LKML in knowing the true feelings of apologizers is final, perfect and can never be questioned. /S
  • neatze 6y ago
    But you have a choice to interact with community or not to interact with an community, difference is that researcher's of “hypocrite commit” did not give such choice to community members, as of today there is overwhelming indication that this researcher's violated basic and fundamental principles of scientific ethical research.
    • gadf 6y ago
      But if I choose to interact with the community, does my choice to present myself, justify any and all abusive actions they may volley at me? Oof coarse not. The point would ordinarily be a good one, but I think you've misused it. The Linux kernel community, by your logic, chose to interact with and be open to patches which were not properly vetted. This framing seems secondary to a seemingly more important point, which is, nobody is fucking perfect. In LKLM or in UMinn. The light now shines on UMinn, but to me, the feigned horror of LKML at UMinn's transgressions, and the lack of commensurate outrage at the many abusive, toxic and hostile transgressions of the LKML community, presupposes a terrifying "normalization" of abusive behavior within the LKLM community. By sooch notion, it's safer to interact with the UMinn transgressors, than it is with LKML, who here fail to even once question their own guilt, introspect upon their own created violations of trust and ethics, but heartily condemn the egregious breaches of trust of others. I'd rather hang with the contrite, or at least hangdog, scallywag, than the quicktongued accusers blind to their own abuses. I'd feel safer there anyhoow.
• booleandilemma 6y ago
  we believe we have much to contribute in the future Please, you've contributed enough already, really.
  • spfzero 6y ago
    Yep, we've had a taste of your contribution. Hope there's no more of it.
  • denton-scratch 6y ago
    Yeah. Please find another project, that doesn't involve deliberately breaking other people's work, and screwing up the kernel that a billion people rely on. If that means changing jobs, most of us have been there. Looking for a job sucks, but if you screw up that badly, then you do have to consider stacking shelves in a supermarket; you're not fit to work on a public project of such importance.
• generalizations 6y ago
  > We sincerely apologize for any harm our research group did to the Linux kernel community. This doesn't admit to harm done, and sounds like an apology that doesn't admit guilt.
• juliansimioni 6y ago
  This apology fails from the 5th word: "We sincerely apologize for any harm..." While there are other requirements, a sincere apology cannot in any way entertain doubt about the fact that there WAS harm. Truly acknowledging the harm done is foundational to a real apology, and most of us (myself included) end up sneaking in weasel words or phrases like this. Psychologically, its nice for the apologizer, since it allows one to think "i'm being good by apologizing, but maybe I didn't do anything bad after all?". But from the apologizee standpoint, these phrases are often devastating and can make it clear that the apologizer has no real recognition or care of what happened. Personally I've worked pretty hard to try to remove these sorts of phrases from my apologies. It's not easy. It makes you feel much more vulnerable and you really have to let whatever you did sit with you in a very uncomfortable way. But it's worth it.
  • audiometry 6y ago
    Oh yeah, the "mistakes were made" passive-voice non-apology apoplogy. zzzz
  • jack9 6y ago
    > a sincere apology cannot in any way entertain doubt about the fact that there WAS harm Someone will always be apologizing wrong for some. Some interpretation of what harm there was, is not necessarily the same as my interpretation. There are too many ways to construe what harm there was or may have been according to others to satisfy everyone addressed. This is an efficient wording that doesn't explicitly satisfy your (and many people's) specific issues out of "the community", which illustrates the point.
    • MattGaiser 6y ago
      "If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." I understand doubting the sincerity of the authors, but this argument hinges here on them saying "any" instead of "all" and they are often used interchangeably in casual conversation.
      • 6y ago
• MattGaiser 6y ago
  > we did that because we knew we could not ask the maintainers of Linux for permission, or they would be on the lookout for the hypocrite patches. For you security people out there, how do red teams handle this issue?
  • shash 6y ago
    A red teamer answered above: https://news.ycombinator.com/item?id=26929797 https://news.ycombinator.com/item?id=26929797 Basically, warn in that you’re going to do it at some point in the future. Then do it in a time-frame of maybe 6 months or so from another ID. Maybe get some of the higher-ups to sign off (say Linus or Greg in this case) beforehand. At the very least, engage with the community...
• pabs3 6y ago
  Coincidentally, Software Freedom Conservancy recently published an article about how to apologise in a sincere way: https://sfconservancy.org/blog/2021/apr/20/how-to-apologize/ https://sfconservancy.org/blog/2021/apr/20/how-to-apologize/
• dang 6y ago
  Past related threads: “They introduce kernel bugs on purpose” - https://news.ycombinator.com/item?id=26887670 https://news.ycombinator.com/item?id=26887670 - April 2021 (1902 comments) UMN CS&E Statement on Linux Kernel Research - https://news.ycombinator.com/item?id=26895510 https://news.ycombinator.com/item?id=26895510 - April 2021 (313 comments) Others?
• pabs3 6y ago
  Hmm, I wonder why in the section about rebuilding relationships, the Linux Foundation appears before the Linux community? LF doesn't have much to do with Linux development AFAIK.
• sigstoat 6y ago
  to the folks picking this apart word-by-word: all evidence suggests that two of the signatures are natives of china, and the third of india. it is reasonable to credit them with not being native english speakers.
  • lasagnaphil 6y ago
    You can write a good apology even with simple English. And since the researchers clearly have the ability to write papers in major English conferences this doesn’t seem to be a problem. It’s just that the apology doesn't do enough in what harm they have caused and what they are going to do in the future to prevent this harm.
    • sigstoat 6y ago
      > It’s just that the apology doesn't do enough in what harm they have caused and what they are going to do in the future to prevent this harm. and that's fair! (also completely unrelated to what i said.) and has nothing to do with individual word choices that they're being picked apart on, which would be subtle to even non-college educated english speakers. > And since the researchers clearly have the ability to write papers in major English conferences this doesn’t seem to be a problem. you're reading different conference papers than i, native-level english is not generally a requirement.
  • davidgerard 6y ago
    there's no way this didn't go through multiple layers of review
    • sigstoat 6y ago
      sure there is. this is less important to the university than it is to HN. probably it's been mentioned to the dean of engineering, but i doubt anyone above that knows or would care without half an hour of explanation. and even the letter was heavily reviewed, the moral thing to do is to give a professor and two grad students leeway in interpretation of their english. (i'll grant that their past leeway in interpretation of their actions, but those aren't the same.) that doesn't mean it can't be a bad apology, it doesn't mean you have to accept or like it. just, come on, find more substantial issues than their adjectival choices.
• HarryHirsch 6y ago
  We are a research group whose members devote their careers to improving the Linux kernel. We have been working on finding and patching vulnerabilities in Linux for the past five years. The past observations with the patching process had motivated us to also study and address issues with the patching process itself. Can someone please explain why someone who is working with others on a common goal for years suddenly conducts a breaching experiment? What were they thinking? (Also, there must be experience within Linux about malicious commits, three-letter agencies from all over the world must have tried before. Why not try archeology?)
• TA-blahhh 6y ago
  get fuckeddddd umn
• nuclearnice1 6y ago
  I agree with this post > Unless the researchers are lying (which I've not seen a clear indication of), the 190 patches you have selected here are nothing more than collateral damage while you are completely missing the supposed patch submission addresses from which the malicious patches were sent! This all really sounds like a knee-jerk reaction to thier posting. I have to say, I think it's the wrong reaction to have. https://lore.kernel.org/lkml/18edc472a95f1d4efe3ef40cc9b8d2611d4ab990.camel@redhat.com/ https://lore.kernel.org/lkml/18edc472a95f1d4efe3ef40cc9b8d26... Greg Kroah-Hartman’s poorly thought out bulk reversion is the bigger waste of time here.
  • genmud 6y ago
    If someone, or in this case a set of affiliated someones, have proven to do malicious or ethically questionable things, trust has to be earned. The risk of letting things stay without extensive verification is not worth it and I definitely don't consider this a kneejerk reaction given the potential perception risks.
    • nuclearnice1 6y ago
      Hypocrite patches came from Gmail. Banned umn mail. Because that’s going to reduce risk?
      • tmp538394722 6y ago
        It’s not so much that they banned umn email, as it is they banned the university of Minnesota. This particular act is about avoiding the risk of researchers wasting kernel developers time. As evinced by the response, it seems really unlikely that other institutions would think it’s a good idea to perform experiments on the kernel devs in the future.
        • x3__ 6y ago
          You do know that one can contribute without having to use their institution email address ?
• ineptech 6y ago
  Not pictured: any attempt to argue that experimenting on the team that reviews kernel patches is a legitimate thing to do. They discuss it as if they were researching something naturally occuring, like the animals living in a tidepool. I can imagine research on a "process" made up of humans as having some value. For example, testing EMTs on whether they correctly diagnose injuries, or testing TAs on whether they catch exam cheating. But I would expect the researchers to ask someone (in those cases, maybe the manager of the EMTs or the TAs, in this case I suppose Linus) whether this research is wanted, and for guidance on how to go about it. For example they could submit the questionable patches during slow times. Does anyone know if that happened? I assume if it had, they would've mentioned it. With that kind of permission, I wouldn't have a problem with this research and I don't think most people here would either. Without, this is the academic equivalent of those "just a prank, bro" youtube videos.
• ciconia 6y ago
  "To err is human, to forgive divine." So much incredulity and negativity in the comments here. The indignation is totally counter productive. The apology is fine, folks. Just accept it and move on.
• deleted 6y ago
  [deleted]
• ransElectronics 6y ago
  Do not entertain these people's pony show. Do not give them attention. The only way for UMN to be redeemed is for them to expel all the researchers responsible and to fire all the staff tangentially responsible for letting it happen.
  • tmp538394722 6y ago
    Really? I unequivocally think it was unethical research, and thus a mistake. But what’s the appropriate proportional response? I honestly have a hard time thinking firing a bunch of people over this would be a good outcome for anyone. What’s the end goal? I feel like already there’s been enough action to deter this kind of sneaky research in the future. But at some point you’re just going to scare away the ethical researchers too.
    • shkkmo 6y ago
      Non-academics get fired for doing unethical things entirely unrelated to their jobs all the time, just to avoid bad publicity. Determining how to conduct your research ethically is a core part of being a researcher. Basic research into pen testing and security research would have revealed how unethical this study was. I see firing as completely justifiable given that was a failure in a core part of thier job, that not only caused a public relations mess, but has directly negatively affected other members of the research institute because they have been banned from Kernel contribution. What further incompetence do you think is required to justify firing this academic?
      • tmotwu 6y ago
        > Non-academics get fired for doing unethical things entirely unrelated to their jobs all the time, just to avoid bad publicity. Yes, only if it's unrelated to their jobs. People get hired, not fired, to do unethical research in industry labs. Ethics is breached all the time in industry - rarely even considered. Google tried to make amends, but decided caring too much about ethics was a roadblock to their goals. Tesla markets their "FSD" at the risk of other drivers in the road. The only difference here is that companies have a bigger, better legal and PR team. Comparing the moral compass between academia and industry is absolutely ridiculous.
• protomyth 6y ago
  First, they released their apology on a Saturday? The Friday / Saturday release is generally a bit shady to start with. The single mention of the IRB and lack of apology for doing human behavior research without permission of the test subjects is still problematic. I suppose being involved with an industry that does A/B testing on users might be an excuse. Also, "any harm" should be "the harm" in a sincere apology.
• ajarmst 6y ago
  There are some relatively minor issues with this apology that appear to already have ample discussion here, and I'll not repeat it. I want something more: I want to hear from the sponsoring faculty, research ethics board, and editors of the journal that published the article. There appear to be some systemic issues in addition to the investigators' ill-considered project. How was it that this research, which is clearly unethical, ended up being published? I've sat on an IRB: this study would not even have been a close call. Did sponsoring faculty even send it for IRB approval? Did they disclose that they were misleading their subjects? If so, did the IRB approve it? Did it make any recommendations? When submitted for publication, did they state they had IRB approval? Did they disclose that they were misleading experimental subjects? Did any reviewers express ethical concerns? Were the ethics of this study discussed by editors? Because there's either some serious systemic flaws in the review and publication process for this paper or the investigators engaged in serious misconduct. If the latter an apology (while absolutely required) is far from sufficient to address the issue. If the former, then there are several other apologies due, along with confirmation that the process will be reviewed and corrected.
  • neatze 6y ago
    What are your thoughts about this article[0], my reading or article is that; author fails in similar way (to some extent) as researchers and there's IRB in regards to ethics of such research. [0] https://dave-dittrich.medium.com/security-research-ethics-review-cdcabf1bbabf https://dave-dittrich.medium.com/security-research-ethics-re...
    • ajarmst 6y ago
      It's an insightful review into some of the issues and regulations around this. I'm not American, so our local rules are different, and I resigned from my institution's IRB several years ago, and these issues have become more fraught in this period. However, the way I was trained to look at these issues were around the concepts of harm, both actual and potential. In this particular case, as Dittrich notes, the questions are around the ethics of using deception in research. Deception does have a role in legitimate research. Arguably, double-blind experiments, the sine qua non of medical research have a fundamental component of deception. They also represent the most common way the ethical dilemma is resolved: subjects are told that they may be deceived, and they have an opportunity to give informed consent. That could have been done in this case: have project leads inform people working on the project that, in the interest of evaluating the patching process, patches that are incorrect or which introduce vulnerabilities may be submitted by researchers. That, of course, would require some senior members of the organization be aware the study was going on. That last is the way penetration testing and red team investigations of security resolve the ethical question---and distinguish themselves from mere vandals and criminals. Other ways to resolve it include collecting data without deception: instead of introducing flawed or malicious patches themselves, researchers identify such patches that have historically been submitted and then review the processes that led to their acceptance or rejection. This is more difficult, but might arguably produce better results. In the case that there are few or no such cases on record, then I would question the value of doing the study at all: deceiving people to study a phenomena that doesn't appear to occur at an appreciable rate is difficult to justify. There's a simple heuristic: if you're studying a group of human beings that you are not a member of, and for which no members are consciously participating, you must be extremely careful. The general rule in anthropological and sociological research is that you do not lie to your subjects. There are cases where the value of the research is sufficient, and for which no other options are available, to break that rule. But they are rare and the utility must be clearly shown and carefully reviewed by a qualified third party. This experiment doesn't come close. There will certainly be those those willing to argue that this isn't human experimentation and thus does not require ethical review. If your experiment depends on misleading human beings---directly or by omission---then it requires an ethical review. It is unethical to waste people's time to no purpose. In the case of an open source project where volunteers are donating their time, it is particularly egregious: they were squandering volunteers' time. In effect they were destroying part of the contribution people made to a project they care about. That requires a very clear justification, which this particular project absolutely does not provide. I recall a conversation with other IRB members shortly after the Sokal Hoax became known. Our general consensus was that it was hilarious but absolutely unethical if considered as an experiment.
• klipklop 6y ago
  I wonder if the real reason for the patches was to help out certain government agencies with known vulnerabilities. Wild speculation, etc. Whatever the reasons, this open letter is a joke.
• neilv 6y ago
  The overwhelming majority of the HN comments seem critical of a party who... wrote and shipped code intended specifically to exploit the trust/naivete of others, to the others' detriment, for the offending party's career gain. And which party then -- to add insult to injury -- didn't seem to appreciate the offense, when called on it. IMHO, to examine this incident very seriously is appropriate and beneficial. If we ease up on the casting stones, and look at the problem a bit more generally, some of the lessons learned might hit close to home.
• marsdepinski 6y ago
  They should be expelled. Their department chair should be sending the apology.
• Black101 6y ago
  Will they review incoming patches more carefully going forward?
  • Black101 6y ago
    I'll take the down-vote as a no.... to tell you the truth, I expected more from Linux (self-preservation)
• seriousquestion 6y ago
  > any harm > it was hurtful A rather vague and strange way to put it. When did the Linux community adopt the language of safteyism?
• stonogo 6y ago
  Any apology that doesn't involve retracting the paper is hollow.
• mint2 6y ago
  The tone is not really apologetic with the passive, detached, and past tense phrasing. It’s like they are avoiding owning up fully that it was their methods, their choices, and actions were unethical. > we are very sorry that the method used in the “hypocrite commits” paper was inappropriate.
• ondrek 6y ago
  I asked Kangjie Lu to explain the original complain. Here is his reply from Wednesday: —- Thanks for sharing. The statements in the link are wrong. I will make some clarifications. 1. We have never intentionally introduced any bugs in Linux. 2. The project that investigated the issues with OSS patching process was done in November 2020. We did get an IRB letter for the research. The purpose of the research is to improve the security of the OSS patching process. 3. One of my students is working on a different project which has nothing to do with the project mentioned in 2. The student aims to fix problems in Linux instead of introducing problems. If you are interested in the project mentioned in 2, please find more details here: https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.pdf https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc.... I hope these clarify the misunderstandings. Kangjie Lu Assistant Professor Department of Computer Science and Engineering University of Minnesota https://www-users.cs.umn.edu/~kjlu https://www-users.cs.umn.edu/~kjlu —-
• azinman2 6y ago
  People here don’t seem to be convinced. There’s a good amount of defense in this letter, so I get it, and it could have been framed better, but ultimately it seems like they learned a valuable lesson and have value to add, so why not let people learn from mistakes and move on? They’ve already been publicly shamed…
  • sideshowb 6y ago
    Reading this thread I am saddened to see hn apparently conform to the law of maximum offence. We see the least charitable explanation for everything said in the apology because outrage gets upvotes.
    • rcxdude 6y ago
      I agree. This is what I would expect from twitter. I would have hoped HN could manage to be better.
    • davidgerard 6y ago
      there's "assume good faith", and then there's "assume good faith of someone with a documented record of bad faith." The present case is the second. Assuming good faith in this apology is not sensible in the face of the existing evidence. Their goal seems to have been to generate a paper, and they haven't withdrawn that paper.
• a0-prw 6y ago
  Maybe the apology letter is also research. Best not to give these idiots the benefit of the doubt now.
• savant_penguin 6y ago
  It's easier to ask forgiveness than to get permission
• usgroup 6y ago
  Wasn’t it the case that in their last exchange with the maintainer (that email that caused all the stir) they were accused of submitting patches that did nothing at all and wasting time in doing so? In this letter they claim these patches were real fixes. Having got their University banned and overturned the effort it took to write 190 previous patches from others which have now been reverted, it seems likely to me that they are under internal pressure. In all it makes me doubt the sincerity of any of this especially given the tone in the last email with the maintainer. A lot of sudden learning seems to have taken place between then and now.
• x3__ 6y ago
  Unpopular opinion: People here seem to be overreacting when Linus Torvalds thinks this is not a big deal. https://itwire.com/open-source/torvalds-says-submitting-known-buggy-patches-is-a-breach-of-trust.html https://itwire.com/open-source/torvalds-says-submitting-know...
  • shkkmo 6y ago
    Linus doesn't say it isn't a big deal, he says the technical impact of the incident is not high. The reason this is a big deal is the violations of ethics and the breach of trust, not the technical damge to the source code, all of which Linus acknowledges.
• chmike 6y ago
  That might be an hypocrite open letter
• junon 6y ago
  > We just want you to know that we would never intentionally hurt the Linux kernel community and never introduce security vulnerabilities. Our work was conducted with the best of intentions and is all about Uhh.. but that was the exact intent of the paper. And they did it successfully. So.. mission failed successfully? What a bizarre attempt to save-face. This is academic misconduct and they're trying to save their asses. finding and fixing security vulnerabilities.
  • kmm 6y ago
    Obviously the intent was never to introduce security vulnerabilities, no matter how naive and badly thought-out their methodology was. The intent was to show it could be done. None of the proposed vulnerabilities ever got in the kernel, whenever they were at risk of being accepted, the maintainer was warned and the process was aborted.
    • junon 6y ago
      Not from what I read. They mentioned there were multiple commits on stable branch.
      • CRConrad 6y ago
        Yes, but that's among all those previous commits from anyone with a University of Minnesota address that mr Kroah-Hartman ripped out; some going back over a decade ago, others five or six years. AFAICT those were from totally unrelated people, many of whom may not even be there any more, and who had nothing to do with this project. If there are bugs in those, then I think we must conclude that those are "genuine" inadvertent bugs. Unless we were to assume there have been similar research projects there before... Which were never publicised anywhere at all and remain secret to this day? That wouldn't be a research project but genuine black-hat activity. But AFAICS there's no more logical reason to suspect former UMinn contributors of that than any other former contributors.
• throw14082020 6y ago
  Another “I’m sorry you feel that way” letter.
• doe88 6y ago
  Cynical take, is this mail genuine/sincere or is also part of some kind of new experiment? How can we trust them not to running some kind of new trolling?
• rurban 6y ago
  So now it gets interesting when the linux maintainers themselves cannot seperate good from bad patches. It's a 190 vs 3 bad, against ~50 vs 143 bad. Let's hope the maintainers can correctly identify the good ones, now with the given hints (August 2020 only). Or if they still keep staying in witchhunt mode. Word against word. Some of the latest patches do look wrong, but I'm not an expert. Analysis will need a few weeks.
• withinboredom 6y ago
  This feels like a genuine apology that has been edited to hell by superiors to CYA.
• lrvick 6y ago
  If we don't somehow make it socially acceptable to let any security researchers conduct the required social engineering to test supply chain attack susceptibility of OSS maintainers without tipping them off in advance, dangerous state actors -will- continue to do it and -not- tell anyone when they are successful. Punishing these researchers this harshly about bad manners is creating a chilling effect that will scare researchers away from evaluating potentially one of the single biggest vulnerabilities in our industry. To be honest if anyone went around anonymously trying to merge security exploits all across well used open source supply chains and always told everyone when they were successful immediately after and helped everyone become more vigilant in code review, I would call them a public servant even if they were almost universally hated for it. I don't think most people realize just how easy supply chain attacks are and how widely they are being exploited by very dangerous organizations. If something does not change fast to dramatically increase the level of scrutiny we give to code contributions, it is going to get much worse. I have gone very far in pentests. Planting malicious USB cables, modifying keyboard firmware, straight up taking unlocked laptops and walking off, sniping recently expired domains to do XSS attacks, obtaining password reset links for the email accounts of maintainers of highly depended on third party dependencies. Even with consent from high levels at orgs it still upsets unknowing people that are tricked at lower levels in the org. I don't apologize for this because it is my job. I have seen real and successful social engineering attacks by state actors up close and you would way prefer a security researcher with bad manners to break you of your your survivors bias over being hit by the real thing. The reality is big companies can afford to pay people like me. Open source projects by random solo maintainers that the security of almost everyone on the internet relies on... can't. We should be very thankful for people that risk public rebuke to do research like this in open source at minimal cost to the receiving organization.
  • squaresmile 6y ago
    What's your suggestions on how oss maintainers can fend off state actor attacks? Always need two people to sign off on a patch? I strongly agree that supply chain attack is a huge deal and worse attacks will come to light eventually but what should be done at the level of oss maintainers? Like you said, I feel like Open source projects by random solo maintainers that the security of almost everyone on the internet relies on... can't do a lot of things.
• pdimitar 6y ago
  Let's have a bet: The commission / board that approved this experiment will never suffer any consequences. And IMO they are just as guilty. They didn't do their job well.
• noxer 6y ago
  Admission of wrongdoings wont get you uncancelled in 2021.
• egberts1 6y ago
  Definitely would not hire these three guys to do vulnerability assessment of ANY network. Your work is predicated on: - consent by selected few - buy-in on all major parties - documentation trail, ongoing I’m sure there’s more but ...
• richardfey 6y ago
  The only good thing of this whole ordeal is that patches scrutiny and security awareness will improve for some time; but I despise the experiment and anyways believe that it will not lead to some organic improvement of the review process.
• projectileboy 6y ago
  This was not an apology. What’s worse is that it is clear that they still fundamentally do not understand that what they did was wrong.
• dwarfsandstuff 6y ago
  This "apology" sounds like social engineering to me. Like they only sent it so they can continue their "research" about how easily they can manipulate the humans reviewing patches for the Linux kernel.
• davidgerard 6y ago
  Greg K-H replied: > Thank you for your response. > As you know, the Linux Foundation and the Linux Foundation's Technical Advisory Board submitted a letter on Friday to your University outlining the specific actions which need to happen in order for your group, and your University, to be able to work to regain the trust of the Linux kernel community. > Until those actions are taken, we do not have anything further to discuss about this issue. Was this letter publicly released?
• kerng 6y ago
  I think the apology comes across as insincere because they don't just stop with apologizing but go on attempting to justify their actions and already look beyond the issue at hand. First, you just apologize for what you did and that's that. Ball is in the other side of the court. I haven't seen the message from the Linux Foundation to the university- assuming that's private and I didnt miss it somewhere?
• adultSwim 6y ago
  This was clearly human research. It needed to go through IRB review and gain consent of the participants. Organization failures allowed this to happen and need to be addressed.