3 ms·

You make it sound like Bayesian-type analysis is somehow fundamentally impervious to p-hacking and is therefore clearly superior to frequentist for this purpose

• p4wnc6 11y ago
  No statistical method is impervious to malicious attempts to subvert it. That said, reasoning between two different models on the basis of magnitude of t-stat is inherently flawed. That is, such an inference process doesn't possess the properties necessary to lead to the conclusions people try to draw from it, even if no malicious intent is present. When you add this to a situation where the temptation is high for things like p-hacking, it actively encourages poor statistical hygiene. Whereas, with Bayesian statistics, you'd have to go out of your way and do something that is glaringly fishy to do the same thing. Hiding it from people is generally a lot harder than hiding what you're doing with generic p-hacking. One of the main reasons for this is that the model's reliance upon assumptions is usually far more transparent in a Bayesian model. You are forced to publicly declare your dependence on assumptions encoded in the prior. In the frequentist case, many people don't even consider that there are prior assumptions that may not hold about the data, and indeed they often report results which are incorrect because they did not account for basic things, like how their data is encoded, or non-ignorable experimental designs. One of my favorite papers about this sort of thing is "Let's Put the Garbage-Can Regression and Garbage-Can Probits Where They Belong" by Christopher Achen [1]. Of course, the same thing can happen in a Bayesian setting. Someone can use an off-the-shelf Bayesian model fitting tool without understanding what it is doing, and they could use standard prior distributions for analytical convenience or ease of coding, and then later try to package their result as some "fancy Bayesian analysis" when it really suffers from low stats quality just like the dumb t-stat comparison would have. But it's harder to do that in a Bayesian setting without it drawing a red flag (I guess, unless you just tell bald-faced lies about what code you wrote, but that's a different story). The bigger point is that you don't trust just one thing. You don't look at a single description of model fit or comparison, like two t-stats, and draw a conclusion. You should plot the residuals, make calibration plots, ROC curves, histograms of where your observations fall based on predicted outcome variables, etc. There is never any sort of cookie cutter approach that will tell you whether some backtest should be believed. As for the usefulness of managing this stuff in the form of articulating priors, one quote I always liked was in response to a critcism of Bayesian methods' reliance on a prior, where some academic said something like "So and so is standing on the front porch with a shotgun ready to shoot down any assumption that comes over the hill -- meanwhile, the back door is wide open for any assumption to run in." If we set aside the fact that any statistics procedure can be misused, or can be reported upon without proper skepticism and due diligence, then I still do further believe that frequentism, in general as an entire epistemological half of statistics is simply a failed project, especially for any type of problem that is a policy problem, which inherently requires access to a posterior distribution conditioned on some information. But, I don't want to derail this thread and turn it into yet another philosophical debate about whether frequentism can be used as a basis for probability theory that corresponds with reality or not. [1] http://www.columbia.edu/~gjw10/achen04.pdf http://www.columbia.edu/~gjw10/achen04.pdf