> Likelihood functions are objective facts about the data which do not depend on your state of mind. You cannot deceive somebody by reporting likelihood functio

> Likelihood functions are objective facts about the data which do not depend on your state of mind. You cannot deceive somebody by reporting likelihood functions unless you are literally lying about the data or omitting data. There's no equivalent of 'p-hacking'. I think Yudkowsky misses the mark here. Much of the replication crisis is not because of p-values or test statistics, it's about what constitutes "the data." I didn't fully appreciate this until I was a working researcher myself. Rarely are studies done as imagined by outsiders--with a pristine and perfect hypothesis and data collection scheme, amenable to coin-flip analogies. Suppose you run some psychology study (or public health study, or physiology study, or whatever). You sketch out an idea for an experiment, then collect some data. These "data" might be survey responses, biomarker measurements, sensor data, whatever. Now you have to analyze your data and work it into a paper, and now is where the trouble begins. Let's say you are doing a study on an exercise program and mental health. Hypothesis: exercise boosts mood. Some of your subjects bailed on the study after the first visit. Do you include them? Exclude them? Impute their missing values? You measured mood with a survey instrument, but you forgot to validate some fields, so for "rate your mood from 1 (bad) to 5 (good)" which was supposed to be an ordinal 1-5 scale you got a few responses of "3-4", or "3.5", or "pretty good." You tell your grad student to figure it out somehow. She decides to use values from the 0-10 visual analog scale of mood as a surrogate to estimate the 1-5 values (you collected the VAS values but you don't want to bother with those now, you'll have an undergrad look at that this summer and write it up separately). Now you have to measure your outcome. Someone at the biostats consulting center said something about "ordinal regression" but you didn't learn about that in grad school in the '90s, and all your colleagues just do t-tests anyways so you decide to do that. But hmm, maybe it makes more sense to divide people into "good mood (4 or 5)" and "bad mood (1, 2, or 3)" Ah well, let's do both. Those results aren't publishable (p = 0.10 and p=0.08), but maybe only people in bad moods improve their mood with exercise? So let's go back to the data and measure increase in mood, relative to baseline. Better do it as both percentage and as absolute increase. What do you know! Your master's student shows you a logistic model that says that among people with a bad mood initially, exercise increased the odds they'd have a good mood by the end of the study, p = 0.02. Awesome, let's write it up and submit. Wait, which of these excel sheets was the original analysis? Can you go back and double-check we excluded those three subjects with anomalous data? Great. Upload that spreadsheet with the manuscript. [...] This is where the replication crisis comes from. Likelihood functions won't save you from the situation I just outlined--which I saw unfold many, many times in the academy. The lines above commit many research sins, but the thorny thing is that sometimes you do run into unexpected or difficult data analysis situations and even honest researchers will be forced to make decisions that may affect the results. Andrew Gelman's "The Garden of Forking Paths" article is apt here [1]. I think true pre-registration is the only way to really seek truth, as it avoids many (but not all) of the issues above. [1] http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...