3 ms·

It's like police. Good ones do good, and bad ones do harm. You'd expect the good ones to be pro-reform, but there apparently isn't enough pro-reform in academi

by wegs 6y ago

It's like police. Good ones do good, and bad ones do harm. You'd expect the good ones to be pro-reform, but there apparently isn't enough pro-reform in academia to have reform. With that, less and less good will happen, and more and more harm.

SoylentOrange 6y ago
There’s a lot of room for improvement: - grad student stipends are negligently low, making it difficult for students to complete their PhDs. - many journals in CS are not open access despite the authors and reviewers not being compensated for their time. I think pre-print servers and the ability to publish work on your own website has really improved the state of CS research over the past 5 years. In the past, the official policies of many top conferences stated that a paper would not appear in the proceedings if it had been published to your website or a pre-print server. This policy has now been dropped (or is not being enforced) by almost all top CS conferences. - it can be difficult to resolve a conflict with your advisor especially if they have tenure. Tenured professors enjoy significant faculty protection (though maybe not to the same degree as police). Often the most productive avenue for a student in this case is to quietly switch advisors. This can lead to entrenchment of toxic faculty. At my alma mater I can think of a professor who was notorious for having none of their students graduate - they all either left academia or switched advisors It’s certainly an imperfect system and I’m sure folks can point out other flaws in the comments. There is a lot of room for reform in these and other areas. For example the lack of transparency in this case makes the entire academic community look bad for no good reason. However I do think that folks on HN (on average) have an overly negative view of academia, some of which may not be based on an accurate understanding of its good parts. For example peer review, despite its many flaws, turns out to be very valuable. During the COVID-19 pandemic, a lot of spurious science has been published in pre-print form and on Medium, only to then be rebutted and rejected by scientific venues with a robust peer-review process.
- wegs 6y ago
  Well, I had a great time in my Ph.D program. My advisor was the very model of caring about students and integrity. I did good research which I was proud of (but which had very little impact since at the time I didn't know how to sell / position it well) I see the upsides of a functioning academia. My cynicism doesn't come from a lack of understanding of the academy or the good parts of the academy. To the contrary; it's based on: 1) Seeing the upper levels of the academy (which are often incredibly corrupt, although this is well-hidden from students). 2) Seeing enough cases handled to know that cover-ups are much more common than resolutions. 3) Seeing many students get abused, and seeing the impact fake research has on the real world. The lack of transparency makes academia look bad for perfectly good reason: there are huge amounts of money flowing into pockets corruptly at elite schools, entire academic careers build on academic fraud (which is /not/ the same as legal fraud), and there's enough rot in the system that it's hard to reform. The higher you go, the more rot there is. Appearances to the contrary, students are university customers, and faculty are employees. The power legally rests with the administration and the board. The rot persists since the people benefiting set the rules. If I'm the president of a school, and a reform would put me in jail, or even make me lose my million dollar salary, why would I listen to student protests? Faculty protests carry a little more weight, but not enough, and enough faculty are corrupt themselves that you can't build a united front for reform. Regarding peer review, the system is entirely bankrupt. There are no incentives to do a good job. Anecdotally, most reviewers skim papers; no one reads them, and certainly no one checks the math or replicates the results. Less anecdotally, when people have looked at data, it's no different from a die roll (NIPS did a proper evaluation of peer review). Regarding grad student stipends, the basic deal is good on paper: freedom to pursue intellectual interests, quality mentorship, and a quality education with basic living expenses covered (on the taxpayer dollar). This seems like exactly the right thing. It breaks when universities break their side of the bargain, with the taxpayer dollars and charitable donations go to faculty clubs, yachts, and vacation homes, internal power dynamics forcing graduate students to be treated as cheap labor rather than as students, and competition forcing widespread academic fraud.
  - SoylentOrange 6y ago
    One of my friends leads artifact evaluation for USENIX Sec. It's an emerging but I think valuable addition to the peer-review process in CS. There's been a growing movement, which I very much like, for improved reproducibility. Here's an example from NeurIPS: https://towardsdatascience.com/reproducible-machine-learning-cf1841606805 https://towardsdatascience.com/reproducible-machine-learning... ^ As you know, many areas of science have a reproducibility crisis including psychology and medicine. I hope this will go a long way towards addressing some weaknesses in peer review in CS. For myself, I try to be thorough when refereeing papers. Anecdotally I know some more busy academics farm out their reviews to their grad students. I had a largely positive experience in grad school also thanks to my excellent advisor. Although I can say that my ~$25k/yr stipend made it difficult to live on without the additional scholarship money that I earned, and even then... I agree with you that there absolutely needs to be more transparency in many aspects including complaints about advisors, and investigations into dishonest publishing behaviour like in this case.
    - wegs 6y ago
      I think we're long past that point. We need to realign incentive structures and see wholesale change. You'll never see quality peer reviews on an honor system, with no incentives to do quality. The basic concept is broken. It's much better to let people publish as they do research, and have post-hoc reviews, revisions, and discussions. And for transparency, I'm not going to trust any research out of MIT research until MIT openly gets rid of NDAs and non-disparage agreements. Just no. Yuck. And that includes the chains of corporate walls MIT has built up to hide stuff. I won't trust MIT research until I can file a public records request, and get records. I should be able to discover conflicts-of-interest, financial arrangements, and all the other messes MIT gets itself into. I won't trust MIT research until faculty meetings are subject to open meeting laws, and the public can sit in on them and understand how decisions get made. I won't trust MIT research until data, source code, and papers are out in the sun for public review by anyone. I won't trust MIT research until I see bad actors getting disciplined and these things publicly discussed. Until then, unless I know something at MIT isn't corrupt, I'll just assume it might be. I won't trust Stanford either. Too much stuff is simply fabricated to tell a good story. Heck, I won't trust a lot of the elite academy, since I assume it's just as bad. Things get a little bit better one tier down, but the rot is starting to seep in. MIT CSAIL just shut down its main mailing list -- thousands of people -- since people started discussing institutional corruption. Footnote: I had no problem living on my similar stipend. I paid for one bedroom in a four-bedroom apartment, ate cheap food, and didn't spend much otherwise. My university

by wegs 6y ago

SoylentOrange 6y ago

There’s a lot of room for improvement: - grad student stipends are negligently low, making it difficult for students to complete their PhDs. - many journals in CS are not open access despite the authors and reviewers not being compensated for their time. I think pre-print servers and the ability to publish work on your own website has really improved the state of CS research over the past 5 years. In the past, the official policies of many top conferences stated that a paper would not appear in the proceedings if it had been published to your website or a pre-print server. This policy has now been dropped (or is not being enforced) by almost all top CS conferences. - it can be difficult to resolve a conflict with your advisor especially if they have tenure. Tenured professors enjoy significant faculty protection (though maybe not to the same degree as police). Often the most productive avenue for a student in this case is to quietly switch advisors. This can lead to entrenchment of toxic faculty. At my alma mater I can think of a professor who was notorious for having none of their students graduate - they all either left academia or switched advisors It’s certainly an imperfect system and I’m sure folks can point out other flaws in the comments. There is a lot of room for reform in these and other areas. For example the lack of transparency in this case makes the entire academic community look bad for no good reason. However I do think that folks on HN (on average) have an overly negative view of academia, some of which may not be based on an accurate understanding of its good parts. For example peer review, despite its many flaws, turns out to be very valuable. During the COVID-19 pandemic, a lot of spurious science has been published in pre-print form and on Medium, only to then be rebutted and rejected by scientific venues with a robust peer-review process.

wegs 6y ago
Well, I had a great time in my Ph.D program. My advisor was the very model of caring about students and integrity. I did good research which I was proud of (but which had very little impact since at the time I didn't know how to sell / position it well) I see the upsides of a functioning academia. My cynicism doesn't come from a lack of understanding of the academy or the good parts of the academy. To the contrary; it's based on: 1) Seeing the upper levels of the academy (which are often incredibly corrupt, although this is well-hidden from students). 2) Seeing enough cases handled to know that cover-ups are much more common than resolutions. 3) Seeing many students get abused, and seeing the impact fake research has on the real world. The lack of transparency makes academia look bad for perfectly good reason: there are huge amounts of money flowing into pockets corruptly at elite schools, entire academic careers build on academic fraud (which is /not/ the same as legal fraud), and there's enough rot in the system that it's hard to reform. The higher you go, the more rot there is. Appearances to the contrary, students are university customers, and faculty are employees. The power legally rests with the administration and the board. The rot persists since the people benefiting set the rules. If I'm the president of a school, and a reform would put me in jail, or even make me lose my million dollar salary, why would I listen to student protests? Faculty protests carry a little more weight, but not enough, and enough faculty are corrupt themselves that you can't build a united front for reform. Regarding peer review, the system is entirely bankrupt. There are no incentives to do a good job. Anecdotally, most reviewers skim papers; no one reads them, and certainly no one checks the math or replicates the results. Less anecdotally, when people have looked at data, it's no different from a die roll (NIPS did a proper evaluation of peer review). Regarding grad student stipends, the basic deal is good on paper: freedom to pursue intellectual interests, quality mentorship, and a quality education with basic living expenses covered (on the taxpayer dollar). This seems like exactly the right thing. It breaks when universities break their side of the bargain, with the taxpayer dollars and charitable donations go to faculty clubs, yachts, and vacation homes, internal power dynamics forcing graduate students to be treated as cheap labor rather than as students, and competition forcing widespread academic fraud.
- SoylentOrange 6y ago
  One of my friends leads artifact evaluation for USENIX Sec. It's an emerging but I think valuable addition to the peer-review process in CS. There's been a growing movement, which I very much like, for improved reproducibility. Here's an example from NeurIPS: https://towardsdatascience.com/reproducible-machine-learning-cf1841606805 https://towardsdatascience.com/reproducible-machine-learning... ^ As you know, many areas of science have a reproducibility crisis including psychology and medicine. I hope this will go a long way towards addressing some weaknesses in peer review in CS. For myself, I try to be thorough when refereeing papers. Anecdotally I know some more busy academics farm out their reviews to their grad students. I had a largely positive experience in grad school also thanks to my excellent advisor. Although I can say that my ~$25k/yr stipend made it difficult to live on without the additional scholarship money that I earned, and even then... I agree with you that there absolutely needs to be more transparency in many aspects including complaints about advisors, and investigations into dishonest publishing behaviour like in this case.
  - wegs 6y ago
    I think we're long past that point. We need to realign incentive structures and see wholesale change. You'll never see quality peer reviews on an honor system, with no incentives to do quality. The basic concept is broken. It's much better to let people publish as they do research, and have post-hoc reviews, revisions, and discussions. And for transparency, I'm not going to trust any research out of MIT research until MIT openly gets rid of NDAs and non-disparage agreements. Just no. Yuck. And that includes the chains of corporate walls MIT has built up to hide stuff. I won't trust MIT research until I can file a public records request, and get records. I should be able to discover conflicts-of-interest, financial arrangements, and all the other messes MIT gets itself into. I won't trust MIT research until faculty meetings are subject to open meeting laws, and the public can sit in on them and understand how decisions get made. I won't trust MIT research until data, source code, and papers are out in the sun for public review by anyone. I won't trust MIT research until I see bad actors getting disciplined and these things publicly discussed. Until then, unless I know something at MIT isn't corrupt, I'll just assume it might be. I won't trust Stanford either. Too much stuff is simply fabricated to tell a good story. Heck, I won't trust a lot of the elite academy, since I assume it's just as bad. Things get a little bit better one tier down, but the rot is starting to seep in. MIT CSAIL just shut down its main mailing list -- thousands of people -- since people started discussing institutional corruption. Footnote: I had no problem living on my similar stipend. I paid for one bedroom in a four-bedroom apartment, ate cheap food, and didn't spend much otherwise. My university