4 ms·

>"I never saw a clean definition of p-value or alpha -- probability of Type I error." On re-reading this I agree. They do not make a clean comparison. They do

by nonbel 11y ago

>"I never saw a clean definition of p-value or alpha -- probability of Type I error." On re-reading this I agree. They do not make a clean comparison. They do define them though: p-value: "the probability of the observed and more extreme data given the null" alpha: "α is the long-run frequency of Type I errors" To begin with, you appear to have these reversed. Second, the p-value is dependent on the data, it will be different from experiment to experiment. The exact same experiment can give you p=0.001 one time and p=0.32 the next time. Which is the probability of type I error?

graycat 11y ago
Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it. It is common to play fast and loose when are conditioning on something and when not. This conditional probability is conditioned on the observations we used; since those are random variables, so is the conditional probability. Then the 0.32 and 0.001 are values of this random variable on the two trials. The expectation of this random variable will be the probability of Type I error alpha. I strained and tried to find a difference between p value and alpha, but in current usage I see no difference. My guess is that people prefer p-value because it abbreviates probability while alpha does not. Maybe some people want to say that alpha = E[E[p value|given data]] but this is not very operational. Maybe it is what people mean in which case, right, alpha is the expected value of p-value. So, for a given experiment and hypothesis test, can draw a graph with alpha on the X-axis and beta (probability of Type II error) on the Y-axis. Then the graph shows beta as a function of alpha. So, typically the graph runs from (0,1) to (1,0) and is convex. A more powerful test is lower at each value of alpha. A perfect test is just the point at the origin. A trivial test is the one get by ignoring the real data and using a random number generator, and here the curve is just the straight line from (0,1) to (1,0) and is useful for some cases of interpolating when the available values of alpha are discrete, e.g., in resampling plans for distribution-free statistics. Right, beta and power = 1 - beta don't have much to do with alpha or p-value except for the point that for a given statistical test in a specific context, e.g., distribution of the data and number of data points, there is that curve I described that relate alpha and beta, and, thus, also 1 - beta, exactly. As I outlined, a different statistical test, maybe parametric instead of distribution-free, can have a different curve and more power for a given alpha.
- nonbel 11y ago
  >"0.001[the p value] is the conditional probability of type I error conditioned on the data" From the paper: alpha= probability of type I error So, there are two equations: p value= p(alpha|data) alpha = E[E[p value|given data]] Substituting we get: alpha = E[E[(alpha|data)|given data]] What is the difference between "given data" and "data"? How do you isolate alpha?
  - deleted 11y ago
    [deleted]
  - graycat 11y ago
    alpha is a number. Type I error is an event.
- nonbel 11y ago
  Thinking about this more... first of all it should have been, where P(q)=="probability of q": alpha = E[E[P(alpha|data)|given data]] Now we can simplify. Say x=alpha, y=data, z="given data", the E operator can be an arbitrary function f, and applying function f twice is denoted as f#. We get: x=f#(P(x|y)|z)) We know P(a|b)=P(a & b)/P(b) and p(a & b)=P(a)P(b), so x=f#{[P(z)P(x)P(y)]/P(y)}=f#{P(z)P(x)} Denote the inverse of f# as F, then: F(x)=P(z)P(x) Now, z was "given data" which has probability 1. So, F(x)=P(x) In other words, there is method to your madness.
  - graycat 11y ago
    Just routine conditional expectation. For random variables X and Y (under very meager assumptions, E[X|Y] is the almost sure unique random variable that integrates like X over all events in the sigma algebra generated by Y. In particular E[X] = E[E[X|Y]] Details follow from the Radon-Nikodym theorem in, say, Rudin, Real and Complex Analysis where Rudin give the von Neumann proof. Details are also in the standard texts on graduate probability by Loeve, Neveu, Breiman, Chung, and a few more.
    - nonbel 11y ago

by nonbel 11y ago

graycat 11y ago

Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it. It is common to play fast and loose when are conditioning on something and when not. This conditional probability is conditioned on the observations we used; since those are random variables, so is the conditional probability. Then the 0.32 and 0.001 are values of this random variable on the two trials. The expectation of this random variable will be the probability of Type I error alpha. I strained and tried to find a difference between p value and alpha, but in current usage I see no difference. My guess is that people prefer p-value because it abbreviates probability while alpha does not. Maybe some people want to say that alpha = E[E[p value|given data]] but this is not very operational. Maybe it is what people mean in which case, right, alpha is the expected value of p-value. So, for a given experiment and hypothesis test, can draw a graph with alpha on the X-axis and beta (probability of Type II error) on the Y-axis. Then the graph shows beta as a function of alpha. So, typically the graph runs from (0,1) to (1,0) and is convex. A more powerful test is lower at each value of alpha. A perfect test is just the point at the origin. A trivial test is the one get by ignoring the real data and using a random number generator, and here the curve is just the straight line from (0,1) to (1,0) and is useful for some cases of interpolating when the available values of alpha are discrete, e.g., in resampling plans for distribution-free statistics. Right, beta and power = 1 - beta don't have much to do with alpha or p-value except for the point that for a given statistical test in a specific context, e.g., distribution of the data and number of data points, there is that curve I described that relate alpha and beta, and, thus, also 1 - beta, exactly. As I outlined, a different statistical test, maybe parametric instead of distribution-free, can have a different curve and more power for a given alpha.

nonbel 11y ago
>"0.001[the p value] is the conditional probability of type I error conditioned on the data" From the paper: alpha= probability of type I error So, there are two equations: p value= p(alpha|data) alpha = E[E[p value|given data]] Substituting we get: alpha = E[E[(alpha|data)|given data]] What is the difference between "given data" and "data"? How do you isolate alpha?
- deleted 11y ago
  [deleted]
- graycat 11y ago
  alpha is a number. Type I error is an event.
nonbel 11y ago
Thinking about this more... first of all it should have been, where P(q)=="probability of q": alpha = E[E[P(alpha|data)|given data]] Now we can simplify. Say x=alpha, y=data, z="given data", the E operator can be an arbitrary function f, and applying function f twice is denoted as f#. We get: x=f#(P(x|y)|z)) We know P(a|b)=P(a & b)/P(b) and p(a & b)=P(a)P(b), so x=f#{[P(z)P(x)P(y)]/P(y)}=f#{P(z)P(x)} Denote the inverse of f# as F, then: F(x)=P(z)P(x) Now, z was "given data" which has probability 1. So, F(x)=P(x) In other words, there is method to your madness.
- graycat 11y ago
  Just routine conditional expectation. For random variables X and Y (under very meager assumptions, E[X|Y] is the almost sure unique random variable that integrates like X over all events in the sigma algebra generated by Y. In particular E[X] = E[E[X|Y]] Details follow from the Radon-Nikodym theorem in, say, Rudin, Real and Complex Analysis where Rudin give the von Neumann proof. Details are also in the standard texts on graduate probability by Loeve, Neveu, Breiman, Chung, and a few more.
  - nonbel 11y ago