3 ms·

Is there a compendium somewhere of null hypotheses as well as observed p values (or whatever statistic) for experiments with both "significant" and "insignifica

Is there a compendium somewhere of null hypotheses as well as observed p values (or whatever statistic) for experiments with both "significant" and "insignificant" results? Depending on the phenomenon and the hypothesis, an event could have a different probability of occurring and require a different threshold. It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis. This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result. The current approach also ties the experimental method to the result, requiring potentially fallible oversight of the experimental design. The hypothesis-testing aggregator reduces this reliance, which is perhaps more honest about our ability to design good experiments, would allow for a greater diversity of approaches, and could also be more efficient when evaluating results. The most interesting aspect of this is if you could somehow create relationships between standardized hypotheses found on the hypothesis menu such that you could infer other hypotheses in a more systematic way.

kerkeslager 7y ago
> It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis. You're absolutely right, but journals don't see it that way. Journals want to make money, and the articles which make the most money are the ones that prove an alternative hypothesis. This is one of the ways where for-profit publishing is harmful for science. > This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result. "Standardization of hypotheses" gets a bit tricky and I suspect the standardization process would be stifling. Part of the goal of replication is to improve on methodology, and part of that is finding ways in which your hypothesis wasn't clearly defined, or doesn't contribute to a larger theory. There needs to be some flexibility in which hypotheses scientists pursue. A more organic way might be for journals to categorize articles as testing new hypotheses or attempting to reproduce old results, and strive for some ratio between the two (1:4 or somesuch).

kerkeslager 7y ago

> It seems like we are wasting a lot of effort when any experiment is unpublished, when we could at the least be associating a data point with a particular hypothesis. You're absolutely right, but journals don't see it that way. Journals want to make money, and the articles which make the most money are the ones that prove an alternative hypothesis. This is one of the ways where for-profit publishing is harmful for science. > This would require some standardization of hypotheses, so that a researcher could select a hypothesis from a list, conduct an experiment to test it, and report those findings to some aggregator. Others would also test the hypothesis and report their findings. Eventually you have some distribution of findings that allow you to examine the experimental methods of outliers as well as modal experiments. In this way every scientific result is the product of some meta-analysis, rather than allowing a single custom experiment to produce a result. "Standardization of hypotheses" gets a bit tricky and I suspect the standardization process would be stifling. Part of the goal of replication is to improve on methodology, and part of that is finding ways in which your hypothesis wasn't clearly defined, or doesn't contribute to a larger theory. There needs to be some flexibility in which hypotheses scientists pursue. A more organic way might be for journals to categorize articles as testing new hypotheses or attempting to reproduce old results, and strive for some ratio between the two (1:4 or somesuch).