3 ms·

The effort is admirable, but I don't see this fixing the problem. An important part of the evaluation of academics is the reputation of the journals they publis

by Vinnl 7y ago

The effort is admirable, but I don't see this fixing the problem. An important part of the evaluation of academics is the reputation of the journals they publish in, and that reputation is primarily driven by the "impact" of their publications, in the worst cases measured by the Impact Factor [1]. Negative results usually just don't have what is understood as impact by evaluators. This means that there is little incentive to actually publish negative results here, and that those results will likely always be disproportionately outnumbered by selective results - unless the evaluation system changes. In other words: I don't think scarcity of publication venues is the main reason few negative results get published. (Disclosure: I'm involved with another project that hopes to alleviate the problem. [2]) [1] https://medium.com/flockademic/the-ridiculous-number-that-can-make-or-break-academic-careers-704e00ae070a https://medium.com/flockademic/the-ridiculous-number-that-ca... [2] https://medium.com/flockademic/why-replication-studies-are-not-rewarded-and-how-to-fix-that-523c2387820e https://medium.com/flockademic/why-replication-studies-are-n...

the_duke 7y ago
This is nice and all, but how is a "like button" going to help this situation in any way? The root causes don't have to do with informal peer recognition, but a public, measurable kind of recognition, expressed in publications and citations. Citations actually already are a implicit voting system for quality work. ( in theory...) Encouraging journals to publish more "boring" research, and if need be, dedicated journals, can have an actual impact. === Edit: what would be more interesting imo is similar but more extensive: a decentralized, public peer review system.
- Vinnl 7y ago
  (For those reading along: the_duke is talking about the project I'm involved with, https://plaudit.pub https://plaudit.pub .) I think you're right about the root cause; that's why Plaudit's endorsement data is free and open data, publicly available through CrossRef. Citations indeed are a similar implicit voting system, but they have the "problem" that they only accumulate over a larger time scale: an article first has to undergo peer review, then get published, then get read, then get used, and then people using it also need to get their results reviewed and published. We're talking several years here. That's why evaluators often use the journal name/Impact Factor as a proxy for "expected number of citations". With Plaudit, research can start accumulating endorsements from the moment the preprint is published, so it could serve a similar role. (And since Plaudit endorsements judge impact and robustness separately, articles and the journals that publish them can also get recognition for "boring" research - which is not possible using the Impact Factor. === Luckily, there are also plenty of initiatives for alternative public peer review systems. Additionally, there are also tons of "blockchain for science" projects (caveat: most appear to consist of a single whitepaper), although it's not clear to me what science-specific problem the decentralisation solves. If you'd like some pointers to some specific such projects, let me know.
  - the_duke 7y ago
    The goal is laudable. My main point of criticism is that it is essentially a like button that requires almost no effort. I can see the end resulting being researchers just trading "likes" without much consideration. It could at least require a public text that justifies your approval of the paper, with a certain minimum length (100 characters plus). This would be something like a "peer review light".
    - Vinnl 7y ago
      Yes, this is the main potential problem. The hope is that this is combated somewhat by endorsements being completely transparent, traceable and open data, instead of the intransparant process of peer review - which has its own problem of nepotism.
- kd0amg 7y ago
  Citations actually already are a implicit voting system for quality work. ( in theory...) I'm not sure this is even true in theory. The decision to cite something looks to me to be more about relevance than quality. Citing a paper says, "they investigated a related question," rather than "their investigation was especially rigorous/thorough." Encouraging journals to publish more "boring" research, and if need be, dedicated journals, can have an actual impact. Only if tenure committees and funding agencies consider the "boring" research sufficiently important.
dsr_ 7y ago
I think it would be reasonable for a paper to cite the prior art showing negative results when: - the paper is a meta-analysis, or - the paper shows a contradiction of prior negative results, or - the paper's hypothesis explains the prior negative results Normalization of publication of negative results is also useful as a goal of its own; consider that a hundred-person high energy physics experiment probably has room for two hundred negative-result papers along with the two winners, as long as they properly pre-register their hypotheses.
- Vinnl 7y ago
  That is fair, and I think you're right. That said, I think those cases are uncommon enough that it's still practically a guarantee that negative results will not be racking up citations, and hence there still being very little incentive to publish those negative results. There are some exceptions to this, of course, but I think those would get published now anyway. In other words: I don't think limited publication venues is the reason few negative results get published. [1] But absolutely, it should be normalised. I also do think preregistration would be an effective tool in making that happen, if funders start requiring it - which is a challenge in its own right. [1] I think this summarises my views best, so I'm adding this to my original comment as well.
thanatropism 7y ago
Part of the problem is actually in the process of being solved: the social role that Holy Immaculate Science had been acquiring. I know this sounds a bit reactionary in times of antivaxers and even flat-earthers, but we cannot afford to trust "Science" in the way we had been doing up to the mid 2010s. How much social policy was being enacted in the name of p-hacked psychology and social science? How much of our baseline ideologies? Even the first announcement of results from the LHC was botched. Now, you may say "great, this is science, it's supposed to be conjectures and refutations". But then we can't trust it as guarantee of ground truth. Scientific papers can't be used in service of internet debates, etc.
- umvi 7y ago
  > How much social policy was being enacted in the name of p-hacked psychology and social science? How much of our baseline ideologies? I'm interested in how often some interest group pays for p-hacked results so that they can use it to strengthen their position in some way? In my opinion, it's very hard to have good science whenever there is external money in the picture. The temptation/pressure to slightly tweak the variables or scope of the study to achieve a desired outcome for the benefactor is just too high. The worst part is that this allows people to strengthen their biases by cherry picking "scientific studies" that seem to agree with their position even if intuitively something doesn't seem right. And science is the highest authority you can appeal to, so there is no way to refute it short of funding your own, better study that arrives at the opposite conclusion.
- Vinnl 7y ago
  In what way is more scepticism towards science solving the lack of publication of negative results?
- bumby 7y ago
  Your "p-hacked" comment reminded me of this: https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/ It does a pretty good job of illustrating why investigating the rigor of the methodology is as important as the statistical results.

by Vinnl 7y ago

the_duke 7y ago

This is nice and all, but how is a "like button" going to help this situation in any way? The root causes don't have to do with informal peer recognition, but a public, measurable kind of recognition, expressed in publications and citations. Citations actually already are a implicit voting system for quality work. ( in theory...) Encouraging journals to publish more "boring" research, and if need be, dedicated journals, can have an actual impact. === Edit: what would be more interesting imo is similar but more extensive: a decentralized, public peer review system.

Vinnl 7y ago
(For those reading along: the_duke is talking about the project I'm involved with, https://plaudit.pub https://plaudit.pub .) I think you're right about the root cause; that's why Plaudit's endorsement data is free and open data, publicly available through CrossRef. Citations indeed are a similar implicit voting system, but they have the "problem" that they only accumulate over a larger time scale: an article first has to undergo peer review, then get published, then get read, then get used, and then people using it also need to get their results reviewed and published. We're talking several years here. That's why evaluators often use the journal name/Impact Factor as a proxy for "expected number of citations". With Plaudit, research can start accumulating endorsements from the moment the preprint is published, so it could serve a similar role. (And since Plaudit endorsements judge impact and robustness separately, articles and the journals that publish them can also get recognition for "boring" research - which is not possible using the Impact Factor. === Luckily, there are also plenty of initiatives for alternative public peer review systems. Additionally, there are also tons of "blockchain for science" projects (caveat: most appear to consist of a single whitepaper), although it's not clear to me what science-specific problem the decentralisation solves. If you'd like some pointers to some specific such projects, let me know.
- the_duke 7y ago
  The goal is laudable. My main point of criticism is that it is essentially a like button that requires almost no effort. I can see the end resulting being researchers just trading "likes" without much consideration. It could at least require a public text that justifies your approval of the paper, with a certain minimum length (100 characters plus). This would be something like a "peer review light".
  - Vinnl 7y ago
    Yes, this is the main potential problem. The hope is that this is combated somewhat by endorsements being completely transparent, traceable and open data, instead of the intransparant process of peer review - which has its own problem of nepotism.
kd0amg 7y ago
Citations actually already are a implicit voting system for quality work. ( in theory...) I'm not sure this is even true in theory. The decision to cite something looks to me to be more about relevance than quality. Citing a paper says, "they investigated a related question," rather than "their investigation was especially rigorous/thorough." Encouraging journals to publish more "boring" research, and if need be, dedicated journals, can have an actual impact. Only if tenure committees and funding agencies consider the "boring" research sufficiently important.

dsr_ 7y ago

I think it would be reasonable for a paper to cite the prior art showing negative results when: - the paper is a meta-analysis, or - the paper shows a contradiction of prior negative results, or - the paper's hypothesis explains the prior negative results Normalization of publication of negative results is also useful as a goal of its own; consider that a hundred-person high energy physics experiment probably has room for two hundred negative-result papers along with the two winners, as long as they properly pre-register their hypotheses.

Vinnl 7y ago
That is fair, and I think you're right. That said, I think those cases are uncommon enough that it's still practically a guarantee that negative results will not be racking up citations, and hence there still being very little incentive to publish those negative results. There are some exceptions to this, of course, but I think those would get published now anyway. In other words: I don't think limited publication venues is the reason few negative results get published. [1] But absolutely, it should be normalised. I also do think preregistration would be an effective tool in making that happen, if funders start requiring it - which is a challenge in its own right. [1] I think this summarises my views best, so I'm adding this to my original comment as well.

thanatropism 7y ago

Part of the problem is actually in the process of being solved: the social role that Holy Immaculate Science had been acquiring. I know this sounds a bit reactionary in times of antivaxers and even flat-earthers, but we cannot afford to trust "Science" in the way we had been doing up to the mid 2010s. How much social policy was being enacted in the name of p-hacked psychology and social science? How much of our baseline ideologies? Even the first announcement of results from the LHC was botched. Now, you may say "great, this is science, it's supposed to be conjectures and refutations". But then we can't trust it as guarantee of ground truth. Scientific papers can't be used in service of internet debates, etc.

umvi 7y ago
> How much social policy was being enacted in the name of p-hacked psychology and social science? How much of our baseline ideologies? I'm interested in how often some interest group pays for p-hacked results so that they can use it to strengthen their position in some way? In my opinion, it's very hard to have good science whenever there is external money in the picture. The temptation/pressure to slightly tweak the variables or scope of the study to achieve a desired outcome for the benefactor is just too high. The worst part is that this allows people to strengthen their biases by cherry picking "scientific studies" that seem to agree with their position even if intuitively something doesn't seem right. And science is the highest authority you can appeal to, so there is no way to refute it short of funding your own, better study that arrives at the opposite conclusion.
Vinnl 7y ago
In what way is more scepticism towards science solving the lack of publication of negative results?
bumby 7y ago
Your "p-hacked" comment reminded me of this: https://projects.fivethirtyeight.com/p-hacking/ https://projects.fivethirtyeight.com/p-hacking/ It does a pretty good job of illustrating why investigating the rigor of the methodology is as important as the statistical results.