2 ms·

One of my friends leads artifact evaluation for USENIX Sec. It's an emerging but I think valuable addition to the peer-review process in CS. There's been a grow

One of my friends leads artifact evaluation for USENIX Sec. It's an emerging but I think valuable addition to the peer-review process in CS. There's been a growing movement, which I very much like, for improved reproducibility. Here's an example from NeurIPS: https://towardsdatascience.com/reproducible-machine-learning-cf1841606805 https://towardsdatascience.com/reproducible-machine-learning... ^ As you know, many areas of science have a reproducibility crisis including psychology and medicine. I hope this will go a long way towards addressing some weaknesses in peer review in CS. For myself, I try to be thorough when refereeing papers. Anecdotally I know some more busy academics farm out their reviews to their grad students. I had a largely positive experience in grad school also thanks to my excellent advisor. Although I can say that my ~$25k/yr stipend made it difficult to live on without the additional scholarship money that I earned, and even then... I agree with you that there absolutely needs to be more transparency in many aspects including complaints about advisors, and investigations into dishonest publishing behaviour like in this case.

wegs 6y ago
I think we're long past that point. We need to realign incentive structures and see wholesale change. You'll never see quality peer reviews on an honor system, with no incentives to do quality. The basic concept is broken. It's much better to let people publish as they do research, and have post-hoc reviews, revisions, and discussions. And for transparency, I'm not going to trust any research out of MIT research until MIT openly gets rid of NDAs and non-disparage agreements. Just no. Yuck. And that includes the chains of corporate walls MIT has built up to hide stuff. I won't trust MIT research until I can file a public records request, and get records. I should be able to discover conflicts-of-interest, financial arrangements, and all the other messes MIT gets itself into. I won't trust MIT research until faculty meetings are subject to open meeting laws, and the public can sit in on them and understand how decisions get made. I won't trust MIT research until data, source code, and papers are out in the sun for public review by anyone. I won't trust MIT research until I see bad actors getting disciplined and these things publicly discussed. Until then, unless I know something at MIT isn't corrupt, I'll just assume it might be. I won't trust Stanford either. Too much stuff is simply fabricated to tell a good story. Heck, I won't trust a lot of the elite academy, since I assume it's just as bad. Things get a little bit better one tier down, but the rot is starting to seep in. MIT CSAIL just shut down its main mailing list -- thousands of people -- since people started discussing institutional corruption. Footnote: I had no problem living on my similar stipend. I paid for one bedroom in a four-bedroom apartment, ate cheap food, and didn't spend much otherwise. My university

wegs 6y ago

I think we're long past that point. We need to realign incentive structures and see wholesale change. You'll never see quality peer reviews on an honor system, with no incentives to do quality. The basic concept is broken. It's much better to let people publish as they do research, and have post-hoc reviews, revisions, and discussions. And for transparency, I'm not going to trust any research out of MIT research until MIT openly gets rid of NDAs and non-disparage agreements. Just no. Yuck. And that includes the chains of corporate walls MIT has built up to hide stuff. I won't trust MIT research until I can file a public records request, and get records. I should be able to discover conflicts-of-interest, financial arrangements, and all the other messes MIT gets itself into. I won't trust MIT research until faculty meetings are subject to open meeting laws, and the public can sit in on them and understand how decisions get made. I won't trust MIT research until data, source code, and papers are out in the sun for public review by anyone. I won't trust MIT research until I see bad actors getting disciplined and these things publicly discussed. Until then, unless I know something at MIT isn't corrupt, I'll just assume it might be. I won't trust Stanford either. Too much stuff is simply fabricated to tell a good story. Heck, I won't trust a lot of the elite academy, since I assume it's just as bad. Things get a little bit better one tier down, but the rot is starting to seep in. MIT CSAIL just shut down its main mailing list -- thousands of people -- since people started discussing institutional corruption. Footnote: I had no problem living on my similar stipend. I paid for one bedroom in a four-bedroom apartment, ate cheap food, and didn't spend much otherwise. My university