5 ms·

Sometimes dumbing down a concept can totally screw up a person's learning curve. During early days a lot of Java tutorials (in Indian engineering books) mention

Sometimes dumbing down a concept can totally screw up a person's learning curve. During early days a lot of Java tutorials (in Indian engineering books) mentioned that the reason Java has Interfaces is because otherwise it is not possible to inherit from multiple classes. While it is true that you can implement multiple interfaces, the whole point of interface is to define "interface" without forcing an implementation. It has nothing to do with the "limitation" of single inheritance. Coming back to p values. A simple google search will find you many articles that say > A small p-value (typically ≤ 0.05) indicates strong evidence against the null hypothesis, so you reject the null hypothesis. The whole idea of p-values is to warn a scientist that they should look for statistical significance. The behaviour of hypothesis over infinite trials is what that matters and hence more data => better reliance. But "more", "better" etc. are subjective ideas and in many cases when everything else considered normal <0.05 might be good but not always. There are far too many factors such as wrong sampling method, things you can not measure vs things you can measure etc. that affect this number truly. I think author nails it when he writes "Replication is the best statistic." Always think of these tests in biological evolution perspective. Do you think this hypothetis would survive the test of time where it has to frequently face the real world ?

candiodari 9y ago
It seems to me that a number of things are not clear at all. 1) p-values is a metric on a whole library of procedures, not a single check (although one of them, the Normal distribution is the one usually meant). There's 5 basic ones everyone should know, and you could fill a decent bookshelf with details on all of them. This means a true p-value should be accompanied by a) what the source data was b) how it's distributed, and why (better yet, proof). This should include assurances that there are no attempts to game the measurement (or otherwise any change in the source data directly related to this measurement), as that of course invalidates it. c) a sanity check (like a normality test, or redoing the procedure on generated data and verifying the expected result) d) what the exact claim is (e.g. this is normally distributed with a mean > X) e) what the procedure was to verify this claim (e.g. normality test + mean > X ... you need to COMBINE your p values, because if the data isn't normal your P test is invalid and of course your hypothesis should be rejected even if the numbers in the t-test say it shouldn't) f) a HUGE disclaimer that this measurement and it's actual numerical result only apply to past values, and if it's used to change something it is no longer valid, even if it's numerical value changes as a result, and you can still calculate it. 2) it's not a given that it's possible at all. The truth is that there are many things that don't follow the central limit theorem and therefore cannot correctly be used for statistical approximation. Essentially you need to be utterly convinced that every measurement is the result of somehow combinding a large number of effects that keep reappearing. For instance, planetary orbits do NOT satisfy this criterion. Sure they're the result of large numbers of influences, but almost none of those influences ever repeat (e.g. comets passing in close orbits is a big modifier of orbits, and in 4-5 billion years that'll happen once with any given asteroid. Almost every perturbation is unique and doesn't repeat, so it can't be predicted and won't follow statistical laws. That is the sort of effect that no form of statistics will ever find, and that invalidates your results. Because these are rare it works in the short term, but it doesn't work in the long term. E.g. if you repeat the experiment to determine the speed of light with Jupiter's moons, that won't work with the original data because their orbits have shifted too much)
- Karishma1234 8y ago
  You are far more articulate than I am.
tmalsburg2 9y ago
Replication is extremely important but not the solution to the problems Luck is discussing: How do you decide whether some experiment is worth a replication attempt? How do you decide whether a replication was successful? The standard (but not the only) answer is: through p-values. This shows that stats and replication are largely orthogonal issues. So what /is/ the solution to the problems with p-values? Bayesian statistics, which gives you the probability of an hypothesis given the data and some clearly defined set of assumptions, exactly what you typically want.

candiodari 9y ago

It seems to me that a number of things are not clear at all. 1) p-values is a metric on a whole library of procedures, not a single check (although one of them, the Normal distribution is the one usually meant). There's 5 basic ones everyone should know, and you could fill a decent bookshelf with details on all of them. This means a true p-value should be accompanied by a) what the source data was b) how it's distributed, and why (better yet, proof). This should include assurances that there are no attempts to game the measurement (or otherwise any change in the source data directly related to this measurement), as that of course invalidates it. c) a sanity check (like a normality test, or redoing the procedure on generated data and verifying the expected result) d) what the exact claim is (e.g. this is normally distributed with a mean > X) e) what the procedure was to verify this claim (e.g. normality test + mean > X ... you need to COMBINE your p values, because if the data isn't normal your P test is invalid and of course your hypothesis should be rejected even if the numbers in the t-test say it shouldn't) f) a HUGE disclaimer that this measurement and it's actual numerical result only apply to past values, and if it's used to change something it is no longer valid, even if it's numerical value changes as a result, and you can still calculate it. 2) it's not a given that it's possible at all. The truth is that there are many things that don't follow the central limit theorem and therefore cannot correctly be used for statistical approximation. Essentially you need to be utterly convinced that every measurement is the result of somehow combinding a large number of effects that keep reappearing. For instance, planetary orbits do NOT satisfy this criterion. Sure they're the result of large numbers of influences, but almost none of those influences ever repeat (e.g. comets passing in close orbits is a big modifier of orbits, and in 4-5 billion years that'll happen once with any given asteroid. Almost every perturbation is unique and doesn't repeat, so it can't be predicted and won't follow statistical laws. That is the sort of effect that no form of statistics will ever find, and that invalidates your results. Because these are rare it works in the short term, but it doesn't work in the long term. E.g. if you repeat the experiment to determine the speed of light with Jupiter's moons, that won't work with the original data because their orbits have shifted too much)

Karishma1234 8y ago
You are far more articulate than I am.

tmalsburg2 9y ago

Replication is extremely important but not the solution to the problems Luck is discussing: How do you decide whether some experiment is worth a replication attempt? How do you decide whether a replication was successful? The standard (but not the only) answer is: through p-values. This shows that stats and replication are largely orthogonal issues. So what /is/ the solution to the problems with p-values? Bayesian statistics, which gives you the probability of an hypothesis given the data and some clearly defined set of assumptions, exactly what you typically want.