4 ms·

On a practical level, people may not be able execute a well formed experiment. I completely agree with that. However, that doesn't change the meaning of the ma

On a practical level, people may not be able execute a well formed experiment. I completely agree with that. However, that doesn't change the meaning of the mathematics, only that your reality has diverged from what you originally intended/believed. What is the meaning of the number that people call 'p-value' when it is not calculated on a well-formed experiment? I'm not sure if there is a general formula, but you may be able to find some meaning in a particular instance.

Dylan16807 11y ago
You're either defining "well-formed" as there being no such thing as a true hypothesis, or you have completely lost me. Either way I don't think there's anything more I can say. p does not tell you how likely a result is to be true.
- jsprogrammer 11y ago
  A well formed experiment tests only a null hypothesis. p-value is exactly the probability that you observed X given that the previously stated null hypothesis was true at the time of observation. The value (1 - p-value) is exactly the probability that you will make an observation consistent with your hypothesis (ie. expected replication rate). Wikipedia has a decent treatment that might help: https://en.wikipedia.org/wiki/P-value#Definition_and_interpretation https://en.wikipedia.org/wiki/P-value#Definition_and_interpr...
  - Dylan16807 11y ago
    But the importance of a p-value is showing when it's not the null hypothesis. The only time you get 95% reproduction is a result that says the null hypothesis is true. You're entirely right about that specific case. But this only happens when nothing correlates. (And almost no science has been done, because most things in fact don't correlate.) A result that disagrees with the null hypothesis at .05 does not imply any particular chance of another result that also disagrees with the null hypothesis at .05 If there is no correlation, then replication will happen 5% of the time. If there is correlation, it will be somewhere over 5%, but no particular value. When people talk about reproduction, they talk about that chance. It will only be 95% by coincidence.
    - jsprogrammer 11y ago
      >You're entirely right about that specific case. In fact, this is the only case that matters. All other (valid) cases can be reduced to a single, null hypothesis design. p-value is undefined for hypotheses that are not a null hypothesis. It is also undefined for hypotheses which do not hold. Sure, you can walk through the motions, put some numbers together, and eventually produce a number between 0 and 1. However that does not mean you have computed a p-value. If you are testing a non-null hypothesis you have not computed a p-value. If you are testing a null hypothesis that doesn't hold, you have not computed a p-value.
      - Dylan16807 11y ago
        The null hypothesis is where nothing happens. You're supposed to be showing evidence against it. If you redefine things so your "null hypothesis" is where something happens, and you're showing evidence for it, you have done something very very wrong, and you should not be using a .05 threshold either.

Dylan16807 11y ago

You're either defining "well-formed" as there being no such thing as a true hypothesis, or you have completely lost me. Either way I don't think there's anything more I can say. p does not tell you how likely a result is to be true.

jsprogrammer 11y ago
A well formed experiment tests only a null hypothesis. p-value is exactly the probability that you observed X given that the previously stated null hypothesis was true at the time of observation. The value (1 - p-value) is exactly the probability that you will make an observation consistent with your hypothesis (ie. expected replication rate). Wikipedia has a decent treatment that might help: https://en.wikipedia.org/wiki/P-value#Definition_and_interpretation https://en.wikipedia.org/wiki/P-value#Definition_and_interpr...
- Dylan16807 11y ago
  But the importance of a p-value is showing when it's not the null hypothesis. The only time you get 95% reproduction is a result that says the null hypothesis is true. You're entirely right about that specific case. But this only happens when nothing correlates. (And almost no science has been done, because most things in fact don't correlate.) A result that disagrees with the null hypothesis at .05 does not imply any particular chance of another result that also disagrees with the null hypothesis at .05 If there is no correlation, then replication will happen 5% of the time. If there is correlation, it will be somewhere over 5%, but no particular value. When people talk about reproduction, they talk about that chance. It will only be 95% by coincidence.
  - jsprogrammer 11y ago
    >You're entirely right about that specific case. In fact, this is the only case that matters. All other (valid) cases can be reduced to a single, null hypothesis design. p-value is undefined for hypotheses that are not a null hypothesis. It is also undefined for hypotheses which do not hold. Sure, you can walk through the motions, put some numbers together, and eventually produce a number between 0 and 1. However that does not mean you have computed a p-value. If you are testing a non-null hypothesis you have not computed a p-value. If you are testing a null hypothesis that doesn't hold, you have not computed a p-value.
    - Dylan16807 11y ago
      The null hypothesis is where nothing happens. You're supposed to be showing evidence against it. If you redefine things so your "null hypothesis" is where something happens, and you're showing evidence for it, you have done something very very wrong, and you should not be using a .05 threshold either.