3 ms·
>"When model predictions are statistical, then, you want to answer the question: how "likely" is this model, given the data?" Huh? Here is a model with statist
by nonbel 8y ago
>"When model predictions are statistical, then, you want to answer the question: how "likely" is this model, given the data?"
Huh? Here is a model with statistical predictions. Its a model of flipping a fair coin 20 times a day for 365 days:
p = 0.5
nDays = 365
nPerDay = 20
dat = rbinom(nDays, nPerDay, p)
hist(dat)
https://image.ibb.co/bwuYCT/flips.png https://image.ibb.co/bwuYCT/flips.png
Here is how I compare theoretical to actual results:
> mean(dat)
[1] 10.06575
> var(dat)
[1] 5.479181
>
> nPerDay*p
[1] 10
> nPerDay*p*(1-p)
[1] 5
>"So, in the frequentist approach, you check whether the data is sufficiently more likely to have resulted from your model, rather than the null hypothesis model. The "sufficiently more likely" is quantified by a p-value."
A p-value says something about the fit of the model used to calculate it. It doesn't say anything about any other models.
- ssivark 8y agoYour analysis tells nothing about whether the model of fair coin being flipped is confirmed. You see a deviation of 0.06 in the mean and 0.47 in the variance. Is that negligible? Imagine you had much less data i.e. nDays=5 — What if your sample had a mean of 10.6 and a variance of 4.8? Could you confirm or rule out the fair coin model? Or confirm/ rule out the 0.6-heads-0.4-tails model? Within the frequenting framework, you’d need a null hypothesis to answer that question. Regarding p-value, I agree I could have phrased my comment better. In the frequentist framework, there is no way to confirm a model. The only way to arrive at a model is to falsify all “simpler” models one can think of, that might possibly have explained the data.
- nonbel 8y ago>"Your analysis tells nothing about whether the model of fair coin being flipped is confirmed. You see a deviation of 0.06 in the mean and 0.47 in the variance. Is that negligible?" Th easiest way is to just run a simulation thousands or millions of times then check how often the observed results occurred. Eg, using R again to get distributions for the mean and sd for the model and getting the one sided p-value: p = 0.5 nDays = 365 nPerDay = 20 obs = rbinom(nDays, nPerDay, p) sim = replicate(1e3, rbinom(nDays, nPerDay, p)) stats = apply(dat, 2, function(x) c(mean(x), var(x))) mean(stats[1,] > mean(obs)) mean(stats[2,] > var(obs)) >"In the frequentist framework, there is no way to confirm a model. The only way to arrive at a model is to falsify all “simpler” models one can think of, that might possibly have explained the data." That isn't frequestist. Its the so-called "NHST hybrid" which it seems confused stats 101 authors in the 1930-1940s cobbled together from the ideas of Ronald Fisher, Jerzy Neyman, and Egon Pearson. This is a decent intro to that topic, but there are thousands of papers on this: http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf http://library.mpib-berlin.mpg.de/ft/gg/GG_Mindless_2004.pdf
- ssivark 8y agoThanks for the Gigerenzer article. While he points out unfortunate problems with the way NHST is commonly/mindlessly used, none of that changes the fact that in many cases, there is a natural null hypothesis from an application of Occam's razor. (which was the point of the response to OP's question)
- nonbel 8y agoGreat to hear! Here is another you may find interesting: Meehl, Paul E (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34(2), 103–115. http://www.jstor.org/stable/10.2307/186099 http://www.jstor.org/stable/10.2307/186099