3 ms·

I don't think he's trying to do himself any favors. If anything he's trying to do others a favor by expressing a very common problem within academia.

by Kranar 4y ago

matthewdgreen 4y ago
Academia is full of mechanisms to negatively judge your research work. In fact judgement is pretty much the only constant in the field: every single piece of work you do will be under constant scrutiny, will be torn apart both before and after publication, and your entire (research) value will be harshly judged by outsiders on a more or less constant basis. The one positive of this environment that when you see someone who has succeeded in publishing a substantial amount, it's safe to presume that their work is not entirely without merit. That's not a 100% guarantee, but it's a good heuristic. A post like this implies that the author's personal judgement is more insightful than a process that I know to be enormously thorough and harsh, and yet it provides only vague assertions and no details on methodology. At the same time it's hard to ignore that the post is written by someone who is (understandably) a bit bitter and probably isn't being entirely objective in their evaluation. In general it's not my impression that this sort of post produces reliable feedback that outsiders should rely on, and (unless it's very carefully anonymized, which it isn't) it also leaves bitterness in its wake.
- Kranar 4y ago
  >Academia is full of mechanisms to negatively judge your research work. Fascinating, as I did not find this at all to be the case in general. I imagine in some cases it could be true, but for the most part I found most research in academia to not be of much interest to many people, one way or another. Hardly anyone takes much time to either validate research or to critique it. I'm glad to hear that you managed to find something to research that caught the attention of many people/mechanisms, even if only to judge it negatively. It must mean you were doing something important, which I think is the exception and not at all the norm.
  - matthewdgreen 4y ago
    I suspect from your comment that you're not familiar with the actual experience of R1 academic research. It is a Darwinian process that involves repeatedly submitting papers to highly competitive conferences for peer-review (a typical top-tier conference in computer security averages <20% acceptance rate.) You must do this many times just to get an Assistant Professorship. Then, assuming you publish N papers during grad school, you need to continue publishing maybe N/2 additional papers (or more) per year while also submitting successful grant proposals to organizations like NSF. These proposals are even more competitive (and worse: these organizations also consider your publication history and reputation, not just the merits of the proposal.) If you do this unabated for seven years or so then you might get tenure, which means you'll get to keep jumping those hurdles for another 25 years or so. Of course you can argue that all this academic competition selects for the wrong thing (that's subjective) but the competition and testing is never-ending. There is a reason that a lot of people leave for industry, simply because it's so much less stressful.
- btrettel 4y ago
  > The one positive of this environment that when you see someone who has succeeded in publishing a substantial amount, it's safe to presume that their work is not entirely without merit. That's not a 100% guarantee, but it's a good heuristic. A post like this implies that the author's personal judgement is more insightful than a process that I know to be enormously thorough and harsh [...] This has not been my experience. In my own field, I've found that quite a few well-cited peer-reviewed papers have severe flaws that are not difficult to find. I can't call the process "enormously thorough and harsh". I view current academic processes as more of a facade than anything else. Sure, the process probably filters out absolute garbage, but I don't think most people need peer review to recognize the worst quality research.
  - matthewdgreen 4y ago
    Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. The argument about peer review is not that it’s perfect on any given paper, but that someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker. Everyone makes a few mistakes and papers, but in the aggregate it is a shockingly hard gauntlet to run, whatever your impressions may be from the sidelines.
    - btrettel 4y ago
      > Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. My point wasn't a complaint about a few papers, rather, important papers that presumably have had a lot of eyeballs go over them in addition to being peer reviewed. If those papers have serious flaws, then I think that does say something about the standards in academia broadly. One reason why I don't work in academia is that too many academics resist the idea that there is a quality problem. Also: I didn't say that my field was broken or anything like that. It's not optimal, but for the most part it's okay. That has little to do with peer review. > someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker Sure, it's evidence that the work is good, but it's not particularly strong evidence in my view. > whatever your impressions may be from the sidelines. If you're implying that I haven't published in a peer-reviewed venue myself, you're mistaken. I've never had a problem publishing. I think that's because I agree more with Gauss: "few, but ripe". Unfortunately "publish or perish" reins supreme in academia, which probably explains why so many people have trouble as they seem to try to produce "least publishable units". Borderline papers like that of course will have trouble.

I don't think he's trying to do himself any favors. If anything he's trying to do others a favor by expressing a very common problem within academia.

by Kranar 4y ago

I don't think he's trying to do himself any favors. If anything he's trying to do others a favor by expressing a very common problem within academia.

matthewdgreen 4y ago

Academia is full of mechanisms to negatively judge your research work. In fact judgement is pretty much the only constant in the field: every single piece of work you do will be under constant scrutiny, will be torn apart both before and after publication, and your entire (research) value will be harshly judged by outsiders on a more or less constant basis. The one positive of this environment that when you see someone who has succeeded in publishing a substantial amount, it's safe to presume that their work is not entirely without merit. That's not a 100% guarantee, but it's a good heuristic. A post like this implies that the author's personal judgement is more insightful than a process that I know to be enormously thorough and harsh, and yet it provides only vague assertions and no details on methodology. At the same time it's hard to ignore that the post is written by someone who is (understandably) a bit bitter and probably isn't being entirely objective in their evaluation. In general it's not my impression that this sort of post produces reliable feedback that outsiders should rely on, and (unless it's very carefully anonymized, which it isn't) it also leaves bitterness in its wake.

Kranar 4y ago
>Academia is full of mechanisms to negatively judge your research work. Fascinating, as I did not find this at all to be the case in general. I imagine in some cases it could be true, but for the most part I found most research in academia to not be of much interest to many people, one way or another. Hardly anyone takes much time to either validate research or to critique it. I'm glad to hear that you managed to find something to research that caught the attention of many people/mechanisms, even if only to judge it negatively. It must mean you were doing something important, which I think is the exception and not at all the norm.
- matthewdgreen 4y ago
  I suspect from your comment that you're not familiar with the actual experience of R1 academic research. It is a Darwinian process that involves repeatedly submitting papers to highly competitive conferences for peer-review (a typical top-tier conference in computer security averages <20% acceptance rate.) You must do this many times just to get an Assistant Professorship. Then, assuming you publish N papers during grad school, you need to continue publishing maybe N/2 additional papers (or more) per year while also submitting successful grant proposals to organizations like NSF. These proposals are even more competitive (and worse: these organizations also consider your publication history and reputation, not just the merits of the proposal.) If you do this unabated for seven years or so then you might get tenure, which means you'll get to keep jumping those hurdles for another 25 years or so. Of course you can argue that all this academic competition selects for the wrong thing (that's subjective) but the competition and testing is never-ending. There is a reason that a lot of people leave for industry, simply because it's so much less stressful.
btrettel 4y ago
> The one positive of this environment that when you see someone who has succeeded in publishing a substantial amount, it's safe to presume that their work is not entirely without merit. That's not a 100% guarantee, but it's a good heuristic. A post like this implies that the author's personal judgement is more insightful than a process that I know to be enormously thorough and harsh [...] This has not been my experience. In my own field, I've found that quite a few well-cited peer-reviewed papers have severe flaws that are not difficult to find. I can't call the process "enormously thorough and harsh". I view current academic processes as more of a facade than anything else. Sure, the process probably filters out absolute garbage, but I don't think most people need peer review to recognize the worst quality research.
- matthewdgreen 4y ago
  Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. The argument about peer review is not that it’s perfect on any given paper, but that someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker. Everyone makes a few mistakes and papers, but in the aggregate it is a shockingly hard gauntlet to run, whatever your impressions may be from the sidelines.
  - btrettel 4y ago
    > Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. My point wasn't a complaint about a few papers, rather, important papers that presumably have had a lot of eyeballs go over them in addition to being peer reviewed. If those papers have serious flaws, then I think that does say something about the standards in academia broadly. One reason why I don't work in academia is that too many academics resist the idea that there is a quality problem. Also: I didn't say that my field was broken or anything like that. It's not optimal, but for the most part it's okay. That has little to do with peer review. > someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker Sure, it's evidence that the work is good, but it's not particularly strong evidence in my view. > whatever your impressions may be from the sidelines. If you're implying that I haven't published in a peer-reviewed venue myself, you're mistaken. I've never had a problem publishing. I think that's because I agree more with Gauss: "few, but ripe". Unfortunately "publish or perish" reins supreme in academia, which probably explains why so many people have trouble as they seem to try to produce "least publishable units". Borderline papers like that of course will have trouble.