It seems to me that a number of things are not clear at all. 1) p-values is a metric on a whole library of procedures, not a single check (although one of them

It seems to me that a number of things are not clear at all. 1) p-values is a metric on a whole library of procedures, not a single check (although one of them, the Normal distribution is the one usually meant). There's 5 basic ones everyone should know, and you could fill a decent bookshelf with details on all of them. This means a true p-value should be accompanied by a) what the source data was b) how it's distributed, and why (better yet, proof). This should include assurances that there are no attempts to game the measurement (or otherwise any change in the source data directly related to this measurement), as that of course invalidates it. c) a sanity check (like a normality test, or redoing the procedure on generated data and verifying the expected result) d) what the exact claim is (e.g. this is normally distributed with a mean > X) e) what the procedure was to verify this claim (e.g. normality test + mean > X ... you need to COMBINE your p values, because if the data isn't normal your P test is invalid and of course your hypothesis should be rejected even if the numbers in the t-test say it shouldn't) f) a HUGE disclaimer that this measurement and it's actual numerical result only apply to past values, and if it's used to change something it is no longer valid, even if it's numerical value changes as a result, and you can still calculate it. 2) it's not a given that it's possible at all. The truth is that there are many things that don't follow the central limit theorem and therefore cannot correctly be used for statistical approximation. Essentially you need to be utterly convinced that every measurement is the result of somehow combinding a large number of effects that keep reappearing. For instance, planetary orbits do NOT satisfy this criterion. Sure they're the result of large numbers of influences, but almost none of those influences ever repeat (e.g. comets passing in close orbits is a big modifier of orbits, and in 4-5 billion years that'll happen once with any given asteroid. Almost every perturbation is unique and doesn't repeat, so it can't be predicted and won't follow statistical laws. That is the sort of effect that no form of statistics will ever find, and that invalidates your results. Because these are rare it works in the short term, but it doesn't work in the long term. E.g. if you repeat the experiment to determine the speed of light with Jupiter's moons, that won't work with the original data because their orbits have shifted too much)