5 ms·

Hypotheses can never be proven to be correct. I don't want to be in any world where it is believed that a hypothesis is or could be correct. This is a fundamen

Hypotheses can never be proven to be correct. I don't want to be in any world where it is believed that a hypothesis is or could be correct. This is a fundamental tenant of science. All that can be done is to reject hypotheses. You (along with Gwern) have now claimed that I don't understand p-values, but you present no alternative understanding. The reason, of course, is that when you look at the mathematics behind p-value, it is obvious that it is exactly as I claim. Edit to address your edit: >I'm suspicious of this. I don't have time to do the math right now, but an experiment that averages .01 might clear a .05 hurdle far more than 99% of the time, and would definitely be well-formed. And if you set a hurdle at .01 it would only clear it half the time, but it would still be well-formed. You are right that you need to be careful here about what you are comparing across instances. There will be variability since you are only sampling a distribution (most likely at a very low rate) and not observing the entire distribution (which, for continuous distributions, is impossible).

Dylan16807 11y ago
On a certain philosophical level you can never be absolutely sure of anything, and p-values are meaningless. On a practical level, p-values are the chance that a correlation is reported where 'reality' does not have a correlation. This is not the same number as the chance that the result agrees with 'reality'. You can reject the concept of objectivity, but you cannot reject that logic. So I have explained the alternative understanding fine, just go back and replace 'true' and 'false' and 'correct' with a philosophically-hedged version.
- jsprogrammer 11y ago
  On a practical level, people may not be able execute a well formed experiment. I completely agree with that. However, that doesn't change the meaning of the mathematics, only that your reality has diverged from what you originally intended/believed. What is the meaning of the number that people call 'p-value' when it is not calculated on a well-formed experiment? I'm not sure if there is a general formula, but you may be able to find some meaning in a particular instance.
  - Dylan16807 11y ago
    You're either defining "well-formed" as there being no such thing as a true hypothesis, or you have completely lost me. Either way I don't think there's anything more I can say. p does not tell you how likely a result is to be true.
    - jsprogrammer 11y ago
      A well formed experiment tests only a null hypothesis. p-value is exactly the probability that you observed X given that the previously stated null hypothesis was true at the time of observation. The value (1 - p-value) is exactly the probability that you will make an observation consistent with your hypothesis (ie. expected replication rate). Wikipedia has a decent treatment that might help: https://en.wikipedia.org/wiki/P-value#Definition_and_interpretation https://en.wikipedia.org/wiki/P-value#Definition_and_interpr...
      - Dylan16807 11y ago
        But the importance of a p-value is showing when it's not the null hypothesis. The only time you get 95% reproduction is a result that says the null hypothesis is true. You're entirely right about that specific case. But this only happens when nothing correlates. (And almost no science has been done, because most things in fact don't correlate.) A result that disagrees with the null hypothesis at .05 does not imply any particular chance of another result that also disagrees with the null hypothesis at .05 If there is no correlation, then replication will happen 5% of the time. If there is correlation, it will be somewhere over 5%, but no particular value. When people talk about reproduction, they talk about that chance. It will only be 95% by coincidence.
        jsprogrammer 11y ago
        >You're entirely right about that specific case. In fact, this is the only case that matters. All other (valid) cases can be reduced to a single, null hypothesis design. p-value is undefined for hypotheses that are not a null hypothesis. It is also undefined for hypotheses which do not hold. Sure, you can walk through the motions, put some numbers together, and eventually produce a number between 0 and 1. However that does not mean you have computed a p-value. If you are testing a non-null hypothesis you have not computed a p-value. If you are testing a null hypothesis that doesn't hold, you have not computed a p-value.

Dylan16807 11y ago

On a certain philosophical level you can never be absolutely sure of anything, and p-values are meaningless. On a practical level, p-values are the chance that a correlation is reported where 'reality' does not have a correlation. This is not the same number as the chance that the result agrees with 'reality'. You can reject the concept of objectivity, but you cannot reject that logic. So I have explained the alternative understanding fine, just go back and replace 'true' and 'false' and 'correct' with a philosophically-hedged version.

jsprogrammer 11y ago
On a practical level, people may not be able execute a well formed experiment. I completely agree with that. However, that doesn't change the meaning of the mathematics, only that your reality has diverged from what you originally intended/believed. What is the meaning of the number that people call 'p-value' when it is not calculated on a well-formed experiment? I'm not sure if there is a general formula, but you may be able to find some meaning in a particular instance.
- Dylan16807 11y ago
  You're either defining "well-formed" as there being no such thing as a true hypothesis, or you have completely lost me. Either way I don't think there's anything more I can say. p does not tell you how likely a result is to be true.
  - jsprogrammer 11y ago
    A well formed experiment tests only a null hypothesis. p-value is exactly the probability that you observed X given that the previously stated null hypothesis was true at the time of observation. The value (1 - p-value) is exactly the probability that you will make an observation consistent with your hypothesis (ie. expected replication rate). Wikipedia has a decent treatment that might help: https://en.wikipedia.org/wiki/P-value#Definition_and_interpretation https://en.wikipedia.org/wiki/P-value#Definition_and_interpr...
    - Dylan16807 11y ago
      But the importance of a p-value is showing when it's not the null hypothesis. The only time you get 95% reproduction is a result that says the null hypothesis is true. You're entirely right about that specific case. But this only happens when nothing correlates. (And almost no science has been done, because most things in fact don't correlate.) A result that disagrees with the null hypothesis at .05 does not imply any particular chance of another result that also disagrees with the null hypothesis at .05 If there is no correlation, then replication will happen 5% of the time. If there is correlation, it will be somewhere over 5%, but no particular value. When people talk about reproduction, they talk about that chance. It will only be 95% by coincidence.
      - jsprogrammer 11y ago
        >You're entirely right about that specific case. In fact, this is the only case that matters. All other (valid) cases can be reduced to a single, null hypothesis design. p-value is undefined for hypotheses that are not a null hypothesis. It is also undefined for hypotheses which do not hold. Sure, you can walk through the motions, put some numbers together, and eventually produce a number between 0 and 1. However that does not mean you have computed a p-value. If you are testing a non-null hypothesis you have not computed a p-value. If you are testing a null hypothesis that doesn't hold, you have not computed a p-value.