6 ms·

It seems clear to me that Popperian falsification is indeed the only way to separate the wheat from the chaff in theoryspace, however, generating theories gener

It seems clear to me that Popperian falsification is indeed the only way to separate the wheat from the chaff in theoryspace, however, generating theories generally requires inductive reasoning. You observe data, you make an inference that leads to a theory (induction), you then subject that theory to falsificationist testing. "It appears all the swans I've seen are white, therefore I posit all swans are white. Oh, wait, there's a black one, nevermind." On the Bayesian vs Frequentist aspect... Falsification is what you should do to theories, not model parameters. If you have a coin and you're trying to figure out the probability of heads P(H), then you have your model of coin-flipping (Bernoulli process) and you're trying to estimate the model's parameter, so you do statistical inference given some sequence of coin flips. It doesn't seem right to apply frequentist null testing because you want to estimate the model parameter, not make some binary decision. What if you had some prior data you want to include? Or you observe new data in the future? This is exactly what Bayesian inference is setup for. And a lot of science is not about falsifying theories in theoryspace but about estimating model parameters in parameter space in the case we all agree on a particular model. Moreover, a big advantage with Bayesian statistics is that it generally requires you to make your model and assumptions explicit, where it's much easier to scrutinize the model compared to a frequentist statistical test.

raverbashing 8y ago
Yeah the article is trying to associate "using p-values" with "The (Only) True Way Of Doing Science™" which to me seems like a fallacy in itself.
cbkeller 8y ago
Obligatory xckd: https://xkcd.com/2078/ https://xkcd.com/2078/
claudiawerner 8y ago
> It seems clear to me that Popperian falsification is indeed the only way to separate the wheat from the chaff in theoryspace, however, generating theories generally requires inductive reasoning. A report analyzed the entries to Nature one year and found that very, very few of the papers actually met Popper's criteria for falsifiable hypotheses, and in fact most of the papers started out with an exploratory aim and documented their findings. This means that to adopt the Popperian idea that only falsifiable claims are science means we must reject good science. One must also consider what opportunities are being missed if we were to force good science to adopt strictly falsifiable hypotheses before research commences - this would mean that every exploratory paper would need to be redefined (or rather, firstly defined) as a falsifiable hypothesis... in an attempt to please Popperians. It is also a stretch to say that science must be empirical or carried out in a particular way to which falsifiability is congenial. There are good arguments that even philosophy[1] or mathematics can be considered sciences, and indeed they were (see Wissenschaft in Kant and Hegel for instance). Furthermore, it's been argued that Popper's theory of falsification actually includes bad science (pseudoscience). From SEP[0]: >Strictly speaking, his criterion excludes the possibility that there can be a pseudoscientific claim that is refutable. According to Larry Laudan (1983, 121), it “has the untoward consequence of countenancing as ‘scientific’ every crank claim which makes ascertainably false assertions”. Astrology, rightly taken by Popper as an unusually clear example of a pseudoscience, has in fact been tested and thoroughly refuted (Culver and Ianna 1988; Carlson 1985). Similarly, the major threats to the scientific status of psychoanalysis, another of his major targets, do not come from claims that it is untestable but from claims that it has been tested and failed the tests. [0] https://plato.stanford.edu/entries/pseudo-science/#KarPop https://plato.stanford.edu/entries/pseudo-science/#KarPop [1] https://web.archive.org/web/20170621073301/http://www.philosophyisscience.com:80/p/philosophy-is-not-science.html https://web.archive.org/web/20170621073301/http://www.philos...
- cbkeller 8y ago
  Great points. It's one thing to say that a scientific theory should be falsifiable (as a necessary but not sufficient condition), but the Popperians lose me when they claim either (1) that the only valid scientific study is an attempt at falsification, (2) that corroboration (or in Popperian terms, a failed attempt at falsification) has no significance. Both of these tropes come up in this "response", where Ionides et al. use them to throw some pretty serious shade at the ASA: > This approach to the logic of scientific progress, that data can serve to falsify scientific hypotheses but not to demonstrate their truth, was developed by Popper (1959) and has broad acceptance within the scientific community. In the words of Popper (1963), “It is easy to obtain confirmations, or verifications, for nearly every theory,” while, “Every genuine test of a theory is an attempt to falsify it, or to refute it. Testability is falsifiability.” The ASA’s statement appears to be contradicting the scientific method described by Einstein and Popper. However, (1) completely ignores discovery science (something I have ranted about here previously), and implies fully accepting Popper's claim to have solved the centuries-old "problem of induction" [1]. Meanwhile, (2) seems to pop up pretty frequently even though I'm not sure Popper himself would have agreed, considering [2]: > It is easy, [Popper] argues, to obtain evidence in favour of virtually any theory, and he consequently holds that such ‘corroboration’, as he terms it, should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction Ionides et al. conveniently left out the second half of that context. Hmm, if only there were some sort of way to quantify that "riskiness". Like some sort of theorem... maybe of the form P(H|E) = P(E|H)*P(H)/P(E) [1] https://en.wikipedia.org/wiki/Problem_of_induction https://en.wikipedia.org/wiki/Problem_of_induction [2] https://plato.stanford.edu/entries/popper/ https://plato.stanford.edu/entries/popper/
  - claudiawerner 8y ago
    That's a good point, I haven't actually spent a lot of time on Popper, though I was amused to find that under some interpretation of his argument, the most extreme varieties of obvious pseudoscience fall under science. In particular I wasn't aware the article even addressed Popper, in particular what's in footnote ii: >Strictly speaking any method can be made to falsify claims with the addition of a falsification rule. However, the rule must still be shown to be reliable. At the very least, I think it's shaky to try and apply the same standard regardless of the actual method of the science - since some fields take different approaches to others. That's not to say that all approaches are valid, but that one idea of science in one field shouldn't necessarily need to hold up in other one. Some sciences obviously greatly benefit (or did in the past) from even a simple statement of falsifiability, but to others it seems like a total mismatch for the problem domain. I hadn't considered Bayes' rule to work that way (my only exposure to statistics is an intro course on information theory as part of an EE degree) but if I'm right, this opens up a discussion as to how risky is risky enough - is astrology still in the picture then?
tzs 8y ago
> "It appears all the swans I've seen are white, therefore I posit all swans are white. Oh, wait, there's a black one, nevermind." That's such a tedious way to to test a theory of swan color. You have to go find swans, which tend to live in hard to reach, dirty, obnoxious environments. It's much easier to rephrase the hypothesis into a logically exactly equivalent hypothesis, "All non-white things are not swans", and test that instead. I can test that without even leaving my house. Just looking around the room I'm in there are hundreds of non-white things, and none of them are swans. This situation is called Hempel's paradox, also known as the raven paradox or the paradox of indoor ornithology [1]. [1] https://en.wikipedia.org/wiki/Raven_paradox https://en.wikipedia.org/wiki/Raven_paradox
kevinpet 8y ago
The "black swan" story always struck me as ridiculous in confusing a phrase informally indicating rarity with a scientific belief that something doesn't exist or can't happen.
- astazangasta 8y ago
  It also confuses mathematical formalism with scientific method; the former is built on rigor, the latter trades exclusively in approximation.

raverbashing 8y ago

Yeah the article is trying to associate "using p-values" with "The (Only) True Way Of Doing Science™" which to me seems like a fallacy in itself.

cbkeller 8y ago

Obligatory xckd: https://xkcd.com/2078/ https://xkcd.com/2078/

claudiawerner 8y ago

> It seems clear to me that Popperian falsification is indeed the only way to separate the wheat from the chaff in theoryspace, however, generating theories generally requires inductive reasoning. A report analyzed the entries to Nature one year and found that very, very few of the papers actually met Popper's criteria for falsifiable hypotheses, and in fact most of the papers started out with an exploratory aim and documented their findings. This means that to adopt the Popperian idea that only falsifiable claims are science means we must reject good science. One must also consider what opportunities are being missed if we were to force good science to adopt strictly falsifiable hypotheses before research commences - this would mean that every exploratory paper would need to be redefined (or rather, firstly defined) as a falsifiable hypothesis... in an attempt to please Popperians. It is also a stretch to say that science must be empirical or carried out in a particular way to which falsifiability is congenial. There are good arguments that even philosophy[1] or mathematics can be considered sciences, and indeed they were (see Wissenschaft in Kant and Hegel for instance). Furthermore, it's been argued that Popper's theory of falsification actually includes bad science (pseudoscience). From SEP[0]: >Strictly speaking, his criterion excludes the possibility that there can be a pseudoscientific claim that is refutable. According to Larry Laudan (1983, 121), it “has the untoward consequence of countenancing as ‘scientific’ every crank claim which makes ascertainably false assertions”. Astrology, rightly taken by Popper as an unusually clear example of a pseudoscience, has in fact been tested and thoroughly refuted (Culver and Ianna 1988; Carlson 1985). Similarly, the major threats to the scientific status of psychoanalysis, another of his major targets, do not come from claims that it is untestable but from claims that it has been tested and failed the tests. [0] https://plato.stanford.edu/entries/pseudo-science/#KarPop https://plato.stanford.edu/entries/pseudo-science/#KarPop [1] https://web.archive.org/web/20170621073301/http://www.philosophyisscience.com:80/p/philosophy-is-not-science.html https://web.archive.org/web/20170621073301/http://www.philos...

cbkeller 8y ago
Great points. It's one thing to say that a scientific theory should be falsifiable (as a necessary but not sufficient condition), but the Popperians lose me when they claim either (1) that the only valid scientific study is an attempt at falsification, (2) that corroboration (or in Popperian terms, a failed attempt at falsification) has no significance. Both of these tropes come up in this "response", where Ionides et al. use them to throw some pretty serious shade at the ASA: > This approach to the logic of scientific progress, that data can serve to falsify scientific hypotheses but not to demonstrate their truth, was developed by Popper (1959) and has broad acceptance within the scientific community. In the words of Popper (1963), “It is easy to obtain confirmations, or verifications, for nearly every theory,” while, “Every genuine test of a theory is an attempt to falsify it, or to refute it. Testability is falsifiability.” The ASA’s statement appears to be contradicting the scientific method described by Einstein and Popper. However, (1) completely ignores discovery science (something I have ranted about here previously), and implies fully accepting Popper's claim to have solved the centuries-old "problem of induction" [1]. Meanwhile, (2) seems to pop up pretty frequently even though I'm not sure Popper himself would have agreed, considering [2]: > It is easy, [Popper] argues, to obtain evidence in favour of virtually any theory, and he consequently holds that such ‘corroboration’, as he terms it, should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction Ionides et al. conveniently left out the second half of that context. Hmm, if only there were some sort of way to quantify that "riskiness". Like some sort of theorem... maybe of the form P(H|E) = P(E|H)*P(H)/P(E) [1] https://en.wikipedia.org/wiki/Problem_of_induction https://en.wikipedia.org/wiki/Problem_of_induction [2] https://plato.stanford.edu/entries/popper/ https://plato.stanford.edu/entries/popper/
- claudiawerner 8y ago
  That's a good point, I haven't actually spent a lot of time on Popper, though I was amused to find that under some interpretation of his argument, the most extreme varieties of obvious pseudoscience fall under science. In particular I wasn't aware the article even addressed Popper, in particular what's in footnote ii: >Strictly speaking any method can be made to falsify claims with the addition of a falsification rule. However, the rule must still be shown to be reliable. At the very least, I think it's shaky to try and apply the same standard regardless of the actual method of the science - since some fields take different approaches to others. That's not to say that all approaches are valid, but that one idea of science in one field shouldn't necessarily need to hold up in other one. Some sciences obviously greatly benefit (or did in the past) from even a simple statement of falsifiability, but to others it seems like a total mismatch for the problem domain. I hadn't considered Bayes' rule to work that way (my only exposure to statistics is an intro course on information theory as part of an EE degree) but if I'm right, this opens up a discussion as to how risky is risky enough - is astrology still in the picture then?

tzs 8y ago

> "It appears all the swans I've seen are white, therefore I posit all swans are white. Oh, wait, there's a black one, nevermind." That's such a tedious way to to test a theory of swan color. You have to go find swans, which tend to live in hard to reach, dirty, obnoxious environments. It's much easier to rephrase the hypothesis into a logically exactly equivalent hypothesis, "All non-white things are not swans", and test that instead. I can test that without even leaving my house. Just looking around the room I'm in there are hundreds of non-white things, and none of them are swans. This situation is called Hempel's paradox, also known as the raven paradox or the paradox of indoor ornithology [1]. [1] https://en.wikipedia.org/wiki/Raven_paradox https://en.wikipedia.org/wiki/Raven_paradox

kevinpet 8y ago

The "black swan" story always struck me as ridiculous in confusing a phrase informally indicating rarity with a scientific belief that something doesn't exist or can't happen.

astazangasta 8y ago
It also confuses mathematical formalism with scientific method; the former is built on rigor, the latter trades exclusively in approximation.