4 ms·

> The one positive of this environment that when you see someone who has succeeded in publishing a substantial amount, it's safe to presume that their work is n

> The one positive of this environment that when you see someone who has succeeded in publishing a substantial amount, it's safe to presume that their work is not entirely without merit. That's not a 100% guarantee, but it's a good heuristic. A post like this implies that the author's personal judgement is more insightful than a process that I know to be enormously thorough and harsh [...] This has not been my experience. In my own field, I've found that quite a few well-cited peer-reviewed papers have severe flaws that are not difficult to find. I can't call the process "enormously thorough and harsh". I view current academic processes as more of a facade than anything else. Sure, the process probably filters out absolute garbage, but I don't think most people need peer review to recognize the worst quality research.

matthewdgreen 4y ago
Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. The argument about peer review is not that it’s perfect on any given paper, but that someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker. Everyone makes a few mistakes and papers, but in the aggregate it is a shockingly hard gauntlet to run, whatever your impressions may be from the sidelines.
- btrettel 4y ago
  > Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. My point wasn't a complaint about a few papers, rather, important papers that presumably have had a lot of eyeballs go over them in addition to being peer reviewed. If those papers have serious flaws, then I think that does say something about the standards in academia broadly. One reason why I don't work in academia is that too many academics resist the idea that there is a quality problem. Also: I didn't say that my field was broken or anything like that. It's not optimal, but for the most part it's okay. That has little to do with peer review. > someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker Sure, it's evidence that the work is good, but it's not particularly strong evidence in my view. > whatever your impressions may be from the sidelines. If you're implying that I haven't published in a peer-reviewed venue myself, you're mistaken. I've never had a problem publishing. I think that's because I agree more with Gauss: "few, but ripe". Unfortunately "publish or perish" reins supreme in academia, which probably explains why so many people have trouble as they seem to try to produce "least publishable units". Borderline papers like that of course will have trouble.
  - matthewdgreen 4y ago
    I think tarring the entire field of scientific research by saying "I claim I found serious flaws in a few unnamed papers" is pretty meaningless. Provide specifics!
    - btrettel 4y ago
      Based on conversations I've had with tenured academics like yourself, providing specifics won't convince you. You could simply say that you aren't qualified in my field so you can't evaluate what I've said, or deny that I've identified enough problems. Those are reasonable counters, but they won't be convincing to me either. The most effective thing I can do is encourage you to find problems in published papers yourself. Another factor in my reluctance is that explaining each case would take a significant amount of time. (Brandolini's law, basically.) I can point you to places where I already explained problems I've found, for example: https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21DAC https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21... That paper has over 300 forward citations over about 25 years. It was published first in a conference, and later in the top specialized journal in my sub-field. I went through a large fraction of the forward citations and could not find anyone who noticed the major error I point out. And it's not difficult to spot if one does basic spot checks.
      - matthewdgreen 4y ago
        Let me understand this: you found an error in a paper published in 1998, over 24 years ago. Based on this finding you were able to publish a critique and partial solution in a more recent conference or journal, which seems like an absolutely first-class scientific outcome. (I can only assume that very few people were relying on this stupid model, since the flaw wasn’t noticed for decades.) And yet despite this reasonably good outcome correcting an ancient and apparently little-studied result, you feel that this error indicts the entire field of scientific research… because people have cited the paper?? (Most likely, given the number of citations, without relying on the model and simply listing it in “previous work” sections.) You are also unhappy that you can’t get a response from the first author [initially I read their Google scholar as indicating they’d left the field, but they haven’t and that was my mistake reading Google Scholar in Korean.] Meanwhile you have not contacted the third (typically senior and advisory) author, who appears to have publications as of 2022? Yet even if you do the basic legwork here and contact all the authors, I’m not sure quite what result you want from this? I presume you’d like an acknowledgement that an old paper has errors? You’ve already received that through the acceptance of your own paper. Would an admission by the third author help? Do you demand a retraction of that ancient paper and every single paper that cited it since? And how is this the devastating critique of all science that you seem to think it is?

matthewdgreen 4y ago

Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. The argument about peer review is not that it’s perfect on any given paper, but that someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker. Everyone makes a few mistakes and papers, but in the aggregate it is a shockingly hard gauntlet to run, whatever your impressions may be from the sidelines.

btrettel 4y ago
> Complaining about flaws in a few papers is like saying “I found a piece of software with a bug” and then claiming software engineering is a broken field based on those examples. My point wasn't a complaint about a few papers, rather, important papers that presumably have had a lot of eyeballs go over them in addition to being peer reviewed. If those papers have serious flaws, then I think that does say something about the standards in academia broadly. One reason why I don't work in academia is that too many academics resist the idea that there is a quality problem. Also: I didn't say that my field was broken or anything like that. It's not optimal, but for the most part it's okay. That has little to do with peer review. > someone who has repeatedly survived a publication competition with a ~20% success rate and then got other serious researchers to cite and build their work is probably not a total piker Sure, it's evidence that the work is good, but it's not particularly strong evidence in my view. > whatever your impressions may be from the sidelines. If you're implying that I haven't published in a peer-reviewed venue myself, you're mistaken. I've never had a problem publishing. I think that's because I agree more with Gauss: "few, but ripe". Unfortunately "publish or perish" reins supreme in academia, which probably explains why so many people have trouble as they seem to try to produce "least publishable units". Borderline papers like that of course will have trouble.
- matthewdgreen 4y ago
  I think tarring the entire field of scientific research by saying "I claim I found serious flaws in a few unnamed papers" is pretty meaningless. Provide specifics!
  - btrettel 4y ago
    Based on conversations I've had with tenured academics like yourself, providing specifics won't convince you. You could simply say that you aren't qualified in my field so you can't evaluate what I've said, or deny that I've identified enough problems. Those are reasonable counters, but they won't be convincing to me either. The most effective thing I can do is encourage you to find problems in published papers yourself. Another factor in my reluctance is that explaining each case would take a significant amount of time. (Brandolini's law, basically.) I can point you to places where I already explained problems I've found, for example: https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21DAC https://pubpeer.com/publications/95455FA4147A9CBD5EAA5185D21... That paper has over 300 forward citations over about 25 years. It was published first in a conference, and later in the top specialized journal in my sub-field. I went through a large fraction of the forward citations and could not find anyone who noticed the major error I point out. And it's not difficult to spot if one does basic spot checks.
    - matthewdgreen 4y ago
      Let me understand this: you found an error in a paper published in 1998, over 24 years ago. Based on this finding you were able to publish a critique and partial solution in a more recent conference or journal, which seems like an absolutely first-class scientific outcome. (I can only assume that very few people were relying on this stupid model, since the flaw wasn’t noticed for decades.) And yet despite this reasonably good outcome correcting an ancient and apparently little-studied result, you feel that this error indicts the entire field of scientific research… because people have cited the paper?? (Most likely, given the number of citations, without relying on the model and simply listing it in “previous work” sections.) You are also unhappy that you can’t get a response from the first author [initially I read their Google scholar as indicating they’d left the field, but they haven’t and that was my mistake reading Google Scholar in Korean.] Meanwhile you have not contacted the third (typically senior and advisory) author, who appears to have publications as of 2022? Yet even if you do the basic legwork here and contact all the authors, I’m not sure quite what result you want from this? I presume you’d like an acknowledgement that an old paper has errors? You’ve already received that through the acceptance of your own paper. Would an admission by the third author help? Do you demand a retraction of that ancient paper and every single paper that cited it since? And how is this the devastating critique of all science that you seem to think it is?