2 ms·
Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it. It is common to play fast and loose
by graycat 11y ago
Each of 0.32 and 0.001 is
the conditional probability
of type I error conditioned on
the data actually used in finding
it. It is common to play fast
and loose when are conditioning on something and when not.
This conditional probability is conditioned on the observations we used; since those are random variables, so is the conditional probability. Then the 0.32 and 0.001 are values of this
random variable on the two trials. The expectation of this random variable will be the probability of Type I error alpha.
I strained and tried to find
a difference between p value and
alpha, but in current usage
I see no difference. My guess
is that people prefer p-value
because it abbreviates probability while alpha
does not.
Maybe some people want to say
that
alpha = E[E[p value|given data]]
but this is not very operational. Maybe it is
what people mean in which case,
right, alpha is the expected
value of p-value.
So, for a given experiment and
hypothesis test, can draw a graph
with alpha on the X-axis and
beta (probability of Type II
error) on the Y-axis. Then the
graph shows beta as a function
of alpha. So, typically the
graph runs from (0,1) to
(1,0) and is convex. A more
powerful test is lower at
each value of alpha. A perfect
test is just the point at the
origin. A trivial test is
the one get by ignoring the
real data and using a random
number generator, and here the
curve is just the straight
line from (0,1) to (1,0) and
is useful for some cases of
interpolating when the
available values of alpha
are discrete, e.g., in
resampling plans for
distribution-free statistics.
Right, beta and power = 1 - beta don't have much to do with alpha or p-value except for the point that for a given statistical test in a specific context, e.g., distribution of the data and number of data points, there is that curve
I described that relate alpha and beta, and, thus, also 1 - beta, exactly. As I outlined, a different statistical test, maybe parametric instead of distribution-free, can have a different curve and more power for a given alpha.
- nonbel 11y ago>"0.001[the p value] is the conditional probability of type I error conditioned on the data" From the paper: alpha= probability of type I error So, there are two equations: p value= p(alpha|data) alpha = E[E[p value|given data]] Substituting we get: alpha = E[E[(alpha|data)|given data]] What is the difference between "given data" and "data"? How do you isolate alpha?
- deleted 11y ago[deleted]
- graycat 11y agoalpha is a number. Type I error is an event.
- nonbel 11y agoThinking about this more... first of all it should have been, where P(q)=="probability of q": alpha = E[E[P(alpha|data)|given data]] Now we can simplify. Say x=alpha, y=data, z="given data", the E operator can be an arbitrary function f, and applying function f twice is denoted as f#. We get: x=f#(P(x|y)|z)) We know P(a|b)=P(a & b)/P(b) and p(a & b)=P(a)P(b), so x=f#{[P(z)P(x)P(y)]/P(y)}=f#{P(z)P(x)} Denote the inverse of f# as F, then: F(x)=P(z)P(x) Now, z was "given data" which has probability 1. So, F(x)=P(x) In other words, there is method to your madness.
- graycat 11y agoJust routine conditional expectation. For random variables X and Y (under very meager assumptions, E[X|Y] is the almost sure unique random variable that integrates like X over all events in the sigma algebra generated by Y. In particular E[X] = E[E[X|Y]] Details follow from the Radon-Nikodym theorem in, say, Rudin, Real and Complex Analysis where Rudin give the von Neumann proof. Details are also in the standard texts on graduate probability by Loeve, Neveu, Breiman, Chung, and a few more.
- nonbel 11y ago>"alpha is the expected value of p-value" Well, you were correct about this above. You may be onto something here in terms of "actual use".
- kgwgk 11y ago> Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it. The probability of type I error of what exactly? Could you be more explicit? For example: we want to perform a test at the 5% significance level. The procedure is simple: we reject the null hypothesis when p<alpha=0.05. If the null hypothesis is true, in 1000 trials on average: - we get 0<p<0.01 10 times, we reject the null hypothesis - we get 0.01<p<0.02 10 times, we reject the null hypothesis - we get 0.02<p<0.03 10 times, we reject the null hypothesis - we get 0.03<p<0.04 10 times, we reject the null hypothesis - we get 0.04<p<0.05 10 times, we reject the null hypothesis - we get 0.05<p<0.06 10 times, we cannot reject the null hypothesis - we get 0.06<p<0.07 10 times, we cannot reject the null hypothesis, etc. The type I error of the procedure is (by construction) 5%: when the null hypothesis is true, we have a rejection (false positive) on average 50 times in 1000 trials. The "type I error rate" is a property of the test, not a property of the outcome. Using your example, we perform the test once and: - we get p=0.001, we reject the null hypothesis. The Type I error rate of the test is 5% - we get p=0.32, we do not reject the null hypothesis. The Type I error rate of the test is 5% Now, could you please fill in the dotted lines? - we get p=0.001, we [ reject the null hypothesis / do something else]. The Type I error rate of .................. is 0.1% - we get p=0.32, we [ do not reject the null hypothesis / do something else ]. The Type I error rate of .................. is 32%