31 ms·

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I fe

For non-experimental, non hard science, and based on the replicability studies of the last few years, and the things we have already known for a long time: I feel pretty comfortable with a prior probability of a paper's result being true positive at ~50%. For experimental, hard science, papers I still have a high belief, maybe 90%. After this I use the following heuristics: - => Lower belief + => higher belief effects multiply, of course, but each is few %. ++ => sufficient details present that I think I can attempt to replicate without contacting authors (this is rare) + => code used is available in public repository (subset of above) + => published in med-to-high ranking journal --- => published in obscure journal -- => socially desirable outcome +++ => socially undesirable outcome -- => non-US, non-EU university or nat. lab + => national lab I mostly read papers in my field (exp. physics), of course. But I began reading a bit more in medical and psyche a few years ago. My views are influenced by books like "Medical Nihilism", and "Science Fictions". Reading COVID papers was also caused me to revise my prior-probabilities downward.

pyinstallwoes 4y ago
In a world where belief can be increasingly influenced in undetected ways due to sophisticated information targeting allowing any individual or demographic to be explicitly targeted and optimized against in a relentless attempt to modify behavior, how does one form belief that is defensible?
- Bakary 4y ago
  There are a few rules of thumb you can apply: - Only a tiny minority of your beliefs have any impact on the outcome of your life. Perhaps your concept of [insert social or political policy] or [scientific topic] or [cultural idea] might have been gamed by someone at some point, but ultimately it will rarely matter whether you even hear of these concepts to begin with or not. - Of the beliefs that remain, these often become pretty defensible through practice and results. Put another way, there are plenty of barely-literate people who still have lead fascinating and worthwhile lives even in the modern era.
exmadscientist 4y ago
My favorite signal for fields I am unfamiliar with has always been --- => journal declares its ISSN prominently ---- => journal prints its ISSN on every page It has not failed me yet.
- drited 4y ago
  Is the correct interpretation of the minus signs here that you distrust journals that print ISSN on every page more than you distrust journals that print ISSN prominently? Also could you please expand on the thought process behind this heuristic?
  - exmadscientist 4y ago
    > Is the correct interpretation of the minus signs here that you distrust journals that print ISSN on every page more than you distrust journals that print ISSN prominently? Yes. At least, that was how I interpreted the grandparent post's scale and how I tried to match it. > Also could you please expand on the thought process behind this heuristic? Have... have you ever encountered these journals in the wild? They're trash. But they come up in search results because search engines are dumb, and I don't know the "good" journals in every field, so I can't filter them out myself. Until I click. Then I can! Usually what's in them is papers from researchers based in countries that have a publication count score or bonus. They'll take some overcomplicated homework problem and solve it or set up some overcomplicated scenario and show that it has property X or have a few diagrams or photos of their current overcomplicated-senior-project research and show that it will do Y when they finish it. All worthless. There is no insight whatsoever.
    - drited 4y ago
      Interesting thank you!
immibis 4y ago
You would assign a +++ score to a paper that says, for example, that certain races are genetically inferior and should be culled??
- gjm11 4y ago
  Imagine two papers of equal scientific merit: equally strong or weak evidence, equally rigorous or sloppy reasoning, etc. One of them finds that certain races are genetically inferior. One of them finds that democratic countries have stronger economies. Which is going to have an easier time getting to publication? If publication is in the Nazi Journal of Nazi Naziness, the first one will be easier to get published. If for whatever reason you're reading journals there, the second of those papers is the one you should expect to be unusually strong. If publication is in Nature, they probably don't have a lot of Nazis on their editorial team. If that paper got published there, then probably something about it is unusually good somehow despite the unpleasant conclusion. Caveats: 1. With a conclusion as horrifying as the one in your original question, though, I think it's fair to reason as follows: Even if the paper gave tremendously strong evidence for genetic differences between races, there's no way any sane editor would publish a paper that says the "... and should be culled" bit. Therefore, this paper was almost certainly railroaded through to publication by a crazy editor. Therefore, the fact that it made it is not evidence that anything about it is any good. 2. It's OK for your expectations of a paper's merits to be affected by how plausible you find its conclusions before you read it. Many "socially undesirable" ideas are that way at least partly because most people find them highly implausible, and if most people do then there's a good chance you do too, and if you do then it's fine for that to lower your expectations for the paper.
nmca 4y ago
Take logs, then the effects add.
Bakary 4y ago
Could you expand on what you define as socially desirable or undesirable? I understand the point you are making but I am not sure exactly through what lens you are quantifying these characteristics.
- avsteele 4y ago
  https://www.econlib.org/archives/2014/04/social_desirabi_1.html https://www.econlib.org/archives/2014/04/social_desirabi_1.h... The proximal desirability bias is that of publication gatekeepers. Biases of society at large also can play a significant role for certain topics.

pyinstallwoes 4y ago

In a world where belief can be increasingly influenced in undetected ways due to sophisticated information targeting allowing any individual or demographic to be explicitly targeted and optimized against in a relentless attempt to modify behavior, how does one form belief that is defensible?

Bakary 4y ago
There are a few rules of thumb you can apply: - Only a tiny minority of your beliefs have any impact on the outcome of your life. Perhaps your concept of [insert social or political policy] or [scientific topic] or [cultural idea] might have been gamed by someone at some point, but ultimately it will rarely matter whether you even hear of these concepts to begin with or not. - Of the beliefs that remain, these often become pretty defensible through practice and results. Put another way, there are plenty of barely-literate people who still have lead fascinating and worthwhile lives even in the modern era.

exmadscientist 4y ago

My favorite signal for fields I am unfamiliar with has always been --- => journal declares its ISSN prominently ---- => journal prints its ISSN on every page It has not failed me yet.

drited 4y ago
Is the correct interpretation of the minus signs here that you distrust journals that print ISSN on every page more than you distrust journals that print ISSN prominently? Also could you please expand on the thought process behind this heuristic?
- exmadscientist 4y ago
  > Is the correct interpretation of the minus signs here that you distrust journals that print ISSN on every page more than you distrust journals that print ISSN prominently? Yes. At least, that was how I interpreted the grandparent post's scale and how I tried to match it. > Also could you please expand on the thought process behind this heuristic? Have... have you ever encountered these journals in the wild? They're trash. But they come up in search results because search engines are dumb, and I don't know the "good" journals in every field, so I can't filter them out myself. Until I click. Then I can! Usually what's in them is papers from researchers based in countries that have a publication count score or bonus. They'll take some overcomplicated homework problem and solve it or set up some overcomplicated scenario and show that it has property X or have a few diagrams or photos of their current overcomplicated-senior-project research and show that it will do Y when they finish it. All worthless. There is no insight whatsoever.
  - drited 4y ago
    Interesting thank you!

immibis 4y ago

You would assign a +++ score to a paper that says, for example, that certain races are genetically inferior and should be culled??

gjm11 4y ago
Imagine two papers of equal scientific merit: equally strong or weak evidence, equally rigorous or sloppy reasoning, etc. One of them finds that certain races are genetically inferior. One of them finds that democratic countries have stronger economies. Which is going to have an easier time getting to publication? If publication is in the Nazi Journal of Nazi Naziness, the first one will be easier to get published. If for whatever reason you're reading journals there, the second of those papers is the one you should expect to be unusually strong. If publication is in Nature, they probably don't have a lot of Nazis on their editorial team. If that paper got published there, then probably something about it is unusually good somehow despite the unpleasant conclusion. Caveats: 1. With a conclusion as horrifying as the one in your original question, though, I think it's fair to reason as follows: Even if the paper gave tremendously strong evidence for genetic differences between races, there's no way any sane editor would publish a paper that says the "... and should be culled" bit. Therefore, this paper was almost certainly railroaded through to publication by a crazy editor. Therefore, the fact that it made it is not evidence that anything about it is any good. 2. It's OK for your expectations of a paper's merits to be affected by how plausible you find its conclusions before you read it. Many "socially undesirable" ideas are that way at least partly because most people find them highly implausible, and if most people do then there's a good chance you do too, and if you do then it's fine for that to lower your expectations for the paper.

nmca 4y ago

Take logs, then the effects add.

Bakary 4y ago

Could you expand on what you define as socially desirable or undesirable? I understand the point you are making but I am not sure exactly through what lens you are quantifying these characteristics.

avsteele 4y ago
https://www.econlib.org/archives/2014/04/social_desirabi_1.html https://www.econlib.org/archives/2014/04/social_desirabi_1.h... The proximal desirability bias is that of publication gatekeepers. Biases of society at large also can play a significant role for certain topics.