4 ms·

Out of curiosity, what would be an actually good way to poke at the pipeline like this? Just ask if they'd OK a patch w/o actually submitting it? A survey?

roca 6y ago
Ask Linus to approve it.
- throwawaybbq1 6y ago
  No .. Linus can approve it on himself. Linus cannot approve such a thing on behalf of other maintainers.
  - mafuy 6y ago
    Agree. Since these researchers did not even ask him, they did not fulfill even the most basic requirement. If, and only if, he approves, then we can talk about who else needs to be in the know, etc.
  - roca 6y ago
    That's fair, but asking for and getting Linus' approval would have at least put them in a much stronger position. They didn't even do that. (And I doubt Linus would have even given his approval, in which case they wouldn't be in this mess.)
throwawaybbq1 6y ago
This is a good question. You would recruit actual maintainers, [edit: or whoever is your intended subject pool] (who would provide consent, perhaps be compensated for their time). You could then give them a series of patches to approve (some being bug free and others having vulnerabilities). [edit: specifying the population of a study is pretty important. Getting random students from the University to approve your security patch doesn't make sense. Picking students who successfully completed a computer security course and got a high grade is better than that but again, may not generalize to the real world. One of the most impressive ways I have seen this being done by grad students was a user study by John Ousterhout and others on Paxos vs. Raft. IIRC, they wanted to claim that Raft was more understandable or led to fewer bugs. Their study design was excellent. See here for an example: https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=DiegoOngaro https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=Diego... ]
- dataflow 6y ago
  If an actual maintainer (i.e. an "insider") approves your bug, then you're not testing the same thing (i.e. the impact an outsider can have), are you?
  - throwawaybbq1 6y ago
    I meant the same set of subjects they wanted to focus on.
    - dataflow 6y ago
      How is this supposed to work? Do you trust everyone equally? If I mailed you something (you being the "subject" in this case), would you trust it just as much as if someone in your family gave it to you?
- robertlagrant 6y ago
  > This took 1 min of thinking btw. QFT.
- ret2plt 6y ago
  This wouldn't really be representative. If people know they are being tested, they will be much more careful and cautious than when they are doing "business as usual".
saagarjha 6y ago
Probably ask the maintainers to consent and add some blinding so that the patches look otherwise legitimate.
ajuc 6y ago
Ask about this upfront, get consent, wait rand()*365 days and do the same thing they did. Inform people immediately after it got accepted.

roca 6y ago

Ask Linus to approve it.

throwawaybbq1 6y ago
No .. Linus can approve it on himself. Linus cannot approve such a thing on behalf of other maintainers.
- mafuy 6y ago
  Agree. Since these researchers did not even ask him, they did not fulfill even the most basic requirement. If, and only if, he approves, then we can talk about who else needs to be in the know, etc.
- roca 6y ago
  That's fair, but asking for and getting Linus' approval would have at least put them in a much stronger position. They didn't even do that. (And I doubt Linus would have even given his approval, in which case they wouldn't be in this mess.)

throwawaybbq1 6y ago

This is a good question. You would recruit actual maintainers, [edit: or whoever is your intended subject pool] (who would provide consent, perhaps be compensated for their time). You could then give them a series of patches to approve (some being bug free and others having vulnerabilities). [edit: specifying the population of a study is pretty important. Getting random students from the University to approve your security patch doesn't make sense. Picking students who successfully completed a computer security course and got a high grade is better than that but again, may not generalize to the real world. One of the most impressive ways I have seen this being done by grad students was a user study by John Ousterhout and others on Paxos vs. Raft. IIRC, they wanted to claim that Raft was more understandable or led to fewer bugs. Their study design was excellent. See here for an example: https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=DiegoOngaro https://www.youtube.com/watch?v=YbZ3zDzDnrw&ab_channel=Diego... ]

dataflow 6y ago
If an actual maintainer (i.e. an "insider") approves your bug, then you're not testing the same thing (i.e. the impact an outsider can have), are you?
- throwawaybbq1 6y ago
  I meant the same set of subjects they wanted to focus on.
  - dataflow 6y ago
    How is this supposed to work? Do you trust everyone equally? If I mailed you something (you being the "subject" in this case), would you trust it just as much as if someone in your family gave it to you?
robertlagrant 6y ago
> This took 1 min of thinking btw. QFT.
ret2plt 6y ago
This wouldn't really be representative. If people know they are being tested, they will be much more careful and cautious than when they are doing "business as usual".

saagarjha 6y ago

Probably ask the maintainers to consent and add some blinding so that the patches look otherwise legitimate.

ajuc 6y ago

Ask about this upfront, get consent, wait rand()*365 days and do the same thing they did. Inform people immediately after it got accepted.