4 ms·

People need to stop doing meta-analysis * . Have you ever heard of meta-analysis of mathematics or physics papers? No? P = NP meta analysis maybe? Haha. I can

by Kenji 11y ago

People need to stop doing meta-analysis * . Have you ever heard of meta-analysis of mathematics or physics papers? No? P = NP meta analysis maybe? Haha. I can tell you why: Because if it's solid science, you don't need 100 papers to describe an effect. Besides, it is a proven fact that the results of meta-analysis are weaker due to how the underlying statistics works (at least if you do it in a naive way, and I do not know if it can be fixed by altering the procedure. I have to brush up on my statistics knowledge, it's good for machine learning.). * EDIT: I should have added here 'to investigate the ground-truth of underlying papers.' Of course there's nothing wrong with meta-analysis to investigate researcher behaviour and methodology.

juhanima 11y ago
Actually: https://www.win.tue.nl/~gwoegi/P-versus-NP.htm https://www.win.tue.nl/~gwoegi/P-versus-NP.htm And amusingly, the distribution of result is close to 50/50
- Kenji 11y ago
  That's actually pretty cool. The difference is that here people are completely aware that this study does not in any way tell us whether P = NP (even if it was 100/0) as long as there's no solid proof on the table. A meta-analysis in those fields is usually only used to analyze the behaviour and methodology of researchers, not to investigate the ground truth of the papers.
  - Retra 11y ago
    "The behaviour and methodology of researchers" is exactly that thing that will lead to the truths you seek. You're basically saying a bridge-builder needn't waste any time worrying about the engineering of boats.
    - Kenji 11y ago
      While I admit it was wrong to claim that all meta-analysis is bad, the main point of my argument has not been refuted. You cannot inspect 100 papers and make a truth claim that is stronger than the quality of the papers, and doing so is a folly. Nobody in maths uses meta-analysis to prove a theorem.

by Kenji 11y ago

juhanima 11y ago

Actually: https://www.win.tue.nl/~gwoegi/P-versus-NP.htm https://www.win.tue.nl/~gwoegi/P-versus-NP.htm And amusingly, the distribution of result is close to 50/50

Kenji 11y ago
That's actually pretty cool. The difference is that here people are completely aware that this study does not in any way tell us whether P = NP (even if it was 100/0) as long as there's no solid proof on the table. A meta-analysis in those fields is usually only used to analyze the behaviour and methodology of researchers, not to investigate the ground truth of the papers.
- Retra 11y ago
  "The behaviour and methodology of researchers" is exactly that thing that will lead to the truths you seek. You're basically saying a bridge-builder needn't waste any time worrying about the engineering of boats.
  - Kenji 11y ago
    While I admit it was wrong to claim that all meta-analysis is bad, the main point of my argument has not been refuted. You cannot inspect 100 papers and make a truth claim that is stronger than the quality of the papers, and doing so is a folly. Nobody in maths uses meta-analysis to prove a theorem.