12 ms·

The Statistical Crisis in Science

73 points by jonathansizz 12y ago

CurtMonash 12y ago
Between the failings in statistics and those in modeling, there's a whole lot of science that's on shaky ground.
dschiptsov 12y ago
Not only errors and misuse of statistics and misapplying of probability theory, but also abstract modeling in general. The very idea of modeling dynamic abstract processes such as finance markets, which itself are mere abstractions is a non-science, it is misuse of pseudo-scientific methods and mathematics, and what we have seen so far is nothing but failures. Too abstract or flawed abstractions and wrong premises cannot be fixed by any amount of math or modeling. They only has to be discarded. The famous "subject/object" false dichotomy in philosophy is the good example too. People could spent ages modeling reality using non-existent abstractions. Today all these multiverse "theories" are mere speculations about whether Siva, Brama or Visnu is the most powerful, forgetting that all these were nothing but anthropomorphic abstractions of the different aspects of one reality. The notion that so-called "modern science" is a new religion (a contest of unproven speculations) is already quite old. btw, a good example of the reductionist mindset (instead of pilling up abstractions) could be the Upanishadic reduction of all the Gods to one Brahman, to which Einstein accidentally discovered a formula - E = mc2, where c is a constant, implying that there is no time in the Universe).
- deleted 12y ago
  [deleted]
- hessenwolf 12y ago
  You are throwing the baby out with the bath water, with respect to financial modelling. Yes, there are failures, and, yes, the models are severely imperfect. We reduced the risk on a portfolio of 2 billion Euro, from a about a billion Euro to a risk of about 50 million Euro using hedging. The remaining 50 million was mostly basis risk, i.e., the mismatch between the underlying instruments in the liabilities and the hedge assets. Using a similar logic to yours, senior management argued that we introduced a new risk called basis risk by trading derivatives.
  - dschiptsov 12y ago
    What is wrong with financial modeling, in my opinion, is not only that models cannot grasp too complex "reality", but that it is changing while you are finishing your model, so no statistical "snapshot" or data-set is even close to be correct. Also, so-called Black swans could occur only within such models. There is no chance that one day c or even g could change (no matter what "scientists" used to say in journals). Btw, finance is a business, not science.)
    - hessenwolf 12y ago
      Yeah, so, I'm guessing you don't do a lot of financial modelling.
      - dschiptsov 12y ago
        Not everyone is so lucky.
    - codecam 12y ago
      You don't actually wan't your model to be a fit to the data -- which I think you are implying it is.
    - tripzilch 12y ago
      > There is no chance that one day c or even g could change (no matter what "scientists" used to say in journals). Well the thing is, c is defined in metres per second. And since 1983, the metre "has been defined as 'the length of the path travelled by light in vacuum during a time interval of 1/299,792,458 of a second.'".[0] Since 1983, c is defined as constant. [0] https://en.wikipedia.org/wiki/Metre https://en.wikipedia.org/wiki/Metre
  - learnstats2 12y ago
    This might be naive of me, but how did you have a billion euro risk on a portfolio of 2 billion euros? How do you assess that risk to now be instead 50 million euros? With respect, this seems like numbers were plucked out of somewhere. I propose that, if these come from a statistical model, it's the model that paints your position in the most optimistic light. I don't judge you for that, but you should really read this article and look out for other fallacies of statistics.
chriswarbo 12y ago
Scientists have tried over at least the past few hundred years (depending on your definitions) to build, from scratch, a perspective on the world which is as free from human bias as possible. At the moment, the jewel in the crown is quantum physics: an inherently statistical theory, so detached from human biases and assumptions that many smart people have struggled to understand or accept it, despite its incredible predictive power. At the heart of the whole process is statistical inference: generalising the results of experiments or observations to the Universe as a whole. A "statistical crisis in science" would be terrible news. We may have been standing on the shoulders of the misinformed, rather than giants. Our "achievements", from particle accelerators to nukes and moon rockets, could have been flukes; if the underlying statistical approach of science was flawed, the predicted behaviour and safety margins of these devices could have been way off. We may be routinely bringing the world to the edge of catastrophe, if we don't understand the consequences of our actions. Oh wait, it seems like some "political scientists" have noticed that their results tend to be influenced by external factors. I hope they realise the irony in their choice of examples: > As a hypothetical example, suppose a researcher is interested in how Democrats and Republicans perform differently in a short mathematics test when it is expressed in two different contexts, involving either healthcare or the military. The article criticises scientists' ability to navigate the statistical minefield of biases, probability estimates, modelling assumptions, etc. in a world of external, political factors like competitive funding, positive publication bias, etc. and they choose an example of measuring how political factors affect people's math skills! To me, that seems the sociological equivalent of trying to measure the thermal expansion of a ruler by reading its markings. What do you know, it's still 30cm!
- semi-extrinsic 12y ago
  Saying that quantum mechanics is an inherently statistical theory is a blatant misrepresentation. Precisely the point that makes QM so weird is that it is not caused by statistics. In a (properly set up) double slit experiment, a single electron is simultaneously travelling through both slits and causing an interference pattern.
  - ChrisLomont 12y ago
yummyfajitas 12y ago
So at least two people reading this seem to think it's about using science in the context of their pet peeves. It's not. It's about using a statistical test for a data dependent hypothesis and interpreting the test as if it were used for a data-independent hypothesis. That's all. It's not about using statistics in politics or finance. It's about first looking at the data, then formulating a hypothesis, then running a standard test which is based on the idea that you chose the hypothesis independently of the data. This is a problem in any field.
- lrei 12y ago
  I upvoted your comment but a problem in "any field" seems too strong a statement. AFAIK this isn't much of a problem in CS (my field) and never heard math or physics people complaining... Seems to only be problem in Social "Sciences" & Bio/Med where many (most?) results are statistical significance tests.
  - anon4 12y ago
    I think yummifajitas means that if done in any field, it would be a problem, not that it is necessarily observed in every field.
    - lrei 12y ago
      Yes probably what he was saying. Also, my understanding - not expert, likely wrong - is that it is sometimes done in physics (e.g analyzing data from sensors). But haven't heard any complaints (which does not mean they don't exist). Someone else would need to weight in as this is just supposition and hearsay :)
  - tveita 12y ago
    I bet you could find a fair number of statistics errors in machine learning and other empirical branches of CS.
- anon4 12y ago
  Was about to post this. Thanks.
jmmcd 12y ago
> In general, p-values are based on what would have happened under other possible data sets. As a hypothetical example, suppose a researcher is interested in how Democrats and Republicans perform differently in a short mathematics test when it is expressed in two different contexts, involving either healthcare or the military. [...] At this point a huge number of possible comparisons could be performed, all consistent with the researcher’s theory. For example, the null hypothesis could be rejected (with statistical significance) among men and not among women—explicable under the theory that men are more ideological than women. The meaning of a p-value is expressed in terms of what would have happened with a different data set, yes, but that different data set would have arisen through a different random sampling from the population. The explanation above seems to completely misunderstand the issue.
amelius 12y ago
I really don't understand the meaning of this sentence (below). Perhaps somebody could explain? > As a hypothetical example, suppose a researcher is interested in how Democrats and Republicans perform differently in a short mathematics test when it is expressed in two different contexts, involving either healthcare or the military.
- zeroxfe 12y ago
  As yummyfajitas said, this is just an example of how the actual issue manifests, but here's what that sentence means: Democrats and Republicans have their own biases. These biases may be skew their thought processes and make them perform differently in mathematics tests which are worded differently. For example, a Republican may (unconciously) overshoot the numbers for a question about healthcare costs, or a Democrat for a question about military expenditure. Although this shouldn't happen, since mathematics tests are quite rigorously worded, it might, and a researcher is interested in investigating further.
- lkbm 12y ago
  There was a paper recently that found people did poorly in math problems if the naive, wrong solution confirmed their political views: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2319992 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2319992 It's been making its rounds in the popular media with headlines like "Politics makes you dumb".
SaberTail 12y ago
A good (in my opinion) trend in physics in the past decade or two has been the rise of "blind" analyses[1]. Basically, the entire analysis is predetermined, before looking at the data. Once all the details are nailed down and everyone agrees with the approach, the blinds are taken off. There's no room for "p-hacking". This has some disadvantages, though. It requires a good understanding of the experiment so that you can figure out what an analysis will actually tell you. It's difficult to do a blind analysis on a brand new apparatus, since there can always be unanticipated problems with the data. As an example, one dark matter experiment invited a reporter to their unblinding. At first, it looked like they'd detected dark matter, but then they had to throw out most of the events because they were due to unanticipated noise in one of the photomultiplier tubes[2]. [1] http://www.slac.stanford.edu/econf/C030908/papers/TUIT001.pdf http://www.slac.stanford.edu/econf/C030908/papers/TUIT001.pd... is a quick review. [2] http://www.nytimes.com/2011/04/14/science/space/14dark.html http://www.nytimes.com/2011/04/14/science/space/14dark.html
chuckcode 12y ago
"all models are wrong, but some are useful." - George Box [1] George Box expressed early my general feeling about statistics, it is a very useful tool but remember the limitations of the methods, the data, and the people applying them. I would like to seen an emphasis on openness and transparency with data so others can replicate the analysis and the community can come up with ways to make best practices accessible to anyone. [1] http://en.wikiquote.org/wiki/George_E._P._Box http://en.wikiquote.org/wiki/George_E._P._Box

73 points by jonathansizz 12y ago

Between the failings in statistics and those in modeling, there's a whole lot of science that's on shaky ground.

Not only errors and misuse of statistics and misapplying of probability theory, but also abstract modeling in general. The very idea of modeling dynamic abstract processes such as finance markets, which itself are mere abstractions is a non-science, it is misuse of pseudo-scientific methods and mathematics, and what we have seen so far is nothing but failures. Too abstract or flawed abstractions and wrong premises cannot be fixed by any amount of math or modeling. They only has to be discarded. The famous "subject/object" false dichotomy in philosophy is the good example too. People could spent ages modeling reality using non-existent abstractions. Today all these multiverse "theories" are mere speculations about whether Siva, Brama or Visnu is the most powerful, forgetting that all these were nothing but anthropomorphic abstractions of the different aspects of one reality. The notion that so-called "modern science" is a new religion (a contest of unproven speculations) is already quite old. btw, a good example of the reductionist mindset (instead of pilling up abstractions) could be the Upanishadic reduction of all the Gods to one Brahman, to which Einstein accidentally discovered a formula - E = mc2, where c is a constant, implying that there is no time in the Universe).

deleted 12y ago
[deleted]
hessenwolf 12y ago
You are throwing the baby out with the bath water, with respect to financial modelling. Yes, there are failures, and, yes, the models are severely imperfect. We reduced the risk on a portfolio of 2 billion Euro, from a about a billion Euro to a risk of about 50 million Euro using hedging. The remaining 50 million was mostly basis risk, i.e., the mismatch between the underlying instruments in the liabilities and the hedge assets. Using a similar logic to yours, senior management argued that we introduced a new risk called basis risk by trading derivatives.
- dschiptsov 12y ago
  What is wrong with financial modeling, in my opinion, is not only that models cannot grasp too complex "reality", but that it is changing while you are finishing your model, so no statistical "snapshot" or data-set is even close to be correct. Also, so-called Black swans could occur only within such models. There is no chance that one day c or even g could change (no matter what "scientists" used to say in journals). Btw, finance is a business, not science.)
  - hessenwolf 12y ago
    Yeah, so, I'm guessing you don't do a lot of financial modelling.
    - dschiptsov 12y ago
      Not everyone is so lucky.
  - codecam 12y ago
    You don't actually wan't your model to be a fit to the data -- which I think you are implying it is.
  - tripzilch 12y ago
    > There is no chance that one day c or even g could change (no matter what "scientists" used to say in journals). Well the thing is, c is defined in metres per second. And since 1983, the metre "has been defined as 'the length of the path travelled by light in vacuum during a time interval of 1/299,792,458 of a second.'".[0] Since 1983, c is defined as constant. [0] https://en.wikipedia.org/wiki/Metre https://en.wikipedia.org/wiki/Metre
- learnstats2 12y ago
  This might be naive of me, but how did you have a billion euro risk on a portfolio of 2 billion euros? How do you assess that risk to now be instead 50 million euros? With respect, this seems like numbers were plucked out of somewhere. I propose that, if these come from a statistical model, it's the model that paints your position in the most optimistic light. I don't judge you for that, but you should really read this article and look out for other fallacies of statistics.

chriswarbo 12y ago

Scientists have tried over at least the past few hundred years (depending on your definitions) to build, from scratch, a perspective on the world which is as free from human bias as possible. At the moment, the jewel in the crown is quantum physics: an inherently statistical theory, so detached from human biases and assumptions that many smart people have struggled to understand or accept it, despite its incredible predictive power. At the heart of the whole process is statistical inference: generalising the results of experiments or observations to the Universe as a whole. A "statistical crisis in science" would be terrible news. We may have been standing on the shoulders of the misinformed, rather than giants. Our "achievements", from particle accelerators to nukes and moon rockets, could have been flukes; if the underlying statistical approach of science was flawed, the predicted behaviour and safety margins of these devices could have been way off. We may be routinely bringing the world to the edge of catastrophe, if we don't understand the consequences of our actions. Oh wait, it seems like some "political scientists" have noticed that their results tend to be influenced by external factors. I hope they realise the irony in their choice of examples: > As a hypothetical example, suppose a researcher is interested in how Democrats and Republicans perform differently in a short mathematics test when it is expressed in two different contexts, involving either healthcare or the military. The article criticises scientists' ability to navigate the statistical minefield of biases, probability estimates, modelling assumptions, etc. in a world of external, political factors like competitive funding, positive publication bias, etc. and they choose an example of measuring how political factors affect people's math skills! To me, that seems the sociological equivalent of trying to measure the thermal expansion of a ruler by reading its markings. What do you know, it's still 30cm!

semi-extrinsic 12y ago
Saying that quantum mechanics is an inherently statistical theory is a blatant misrepresentation. Precisely the point that makes QM so weird is that it is not caused by statistics. In a (properly set up) double slit experiment, a single electron is simultaneously travelling through both slits and causing an interference pattern.
- ChrisLomont 12y ago

yummyfajitas 12y ago

So at least two people reading this seem to think it's about using science in the context of their pet peeves. It's not. It's about using a statistical test for a data dependent hypothesis and interpreting the test as if it were used for a data-independent hypothesis. That's all. It's not about using statistics in politics or finance. It's about first looking at the data, then formulating a hypothesis, then running a standard test which is based on the idea that you chose the hypothesis independently of the data. This is a problem in any field.

lrei 12y ago
I upvoted your comment but a problem in "any field" seems too strong a statement. AFAIK this isn't much of a problem in CS (my field) and never heard math or physics people complaining... Seems to only be problem in Social "Sciences" & Bio/Med where many (most?) results are statistical significance tests.
- anon4 12y ago
  I think yummifajitas means that if done in any field, it would be a problem, not that it is necessarily observed in every field.
  - lrei 12y ago
    Yes probably what he was saying. Also, my understanding - not expert, likely wrong - is that it is sometimes done in physics (e.g analyzing data from sensors). But haven't heard any complaints (which does not mean they don't exist). Someone else would need to weight in as this is just supposition and hearsay :)
- tveita 12y ago
  I bet you could find a fair number of statistics errors in machine learning and other empirical branches of CS.
anon4 12y ago
Was about to post this. Thanks.

jmmcd 12y ago

> In general, p-values are based on what would have happened under other possible data sets. As a hypothetical example, suppose a researcher is interested in how Democrats and Republicans perform differently in a short mathematics test when it is expressed in two different contexts, involving either healthcare or the military. [...] At this point a huge number of possible comparisons could be performed, all consistent with the researcher’s theory. For example, the null hypothesis could be rejected (with statistical significance) among men and not among women—explicable under the theory that men are more ideological than women. The meaning of a p-value is expressed in terms of what would have happened with a different data set, yes, but that different data set would have arisen through a different random sampling from the population. The explanation above seems to completely misunderstand the issue.

amelius 12y ago

I really don't understand the meaning of this sentence (below). Perhaps somebody could explain? > As a hypothetical example, suppose a researcher is interested in how Democrats and Republicans perform differently in a short mathematics test when it is expressed in two different contexts, involving either healthcare or the military.

zeroxfe 12y ago
As yummyfajitas said, this is just an example of how the actual issue manifests, but here's what that sentence means: Democrats and Republicans have their own biases. These biases may be skew their thought processes and make them perform differently in mathematics tests which are worded differently. For example, a Republican may (unconciously) overshoot the numbers for a question about healthcare costs, or a Democrat for a question about military expenditure. Although this shouldn't happen, since mathematics tests are quite rigorously worded, it might, and a researcher is interested in investigating further.
lkbm 12y ago
There was a paper recently that found people did poorly in math problems if the naive, wrong solution confirmed their political views: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2319992 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2319992 It's been making its rounds in the popular media with headlines like "Politics makes you dumb".

SaberTail 12y ago

A good (in my opinion) trend in physics in the past decade or two has been the rise of "blind" analyses[1]. Basically, the entire analysis is predetermined, before looking at the data. Once all the details are nailed down and everyone agrees with the approach, the blinds are taken off. There's no room for "p-hacking". This has some disadvantages, though. It requires a good understanding of the experiment so that you can figure out what an analysis will actually tell you. It's difficult to do a blind analysis on a brand new apparatus, since there can always be unanticipated problems with the data. As an example, one dark matter experiment invited a reporter to their unblinding. At first, it looked like they'd detected dark matter, but then they had to throw out most of the events because they were due to unanticipated noise in one of the photomultiplier tubes[2]. [1] http://www.slac.stanford.edu/econf/C030908/papers/TUIT001.pdf http://www.slac.stanford.edu/econf/C030908/papers/TUIT001.pd... is a quick review. [2] http://www.nytimes.com/2011/04/14/science/space/14dark.html http://www.nytimes.com/2011/04/14/science/space/14dark.html

chuckcode 12y ago

"all models are wrong, but some are useful." - George Box [1] George Box expressed early my general feeling about statistics, it is a very useful tool but remember the limitations of the methods, the data, and the people applying them. I would like to seen an emphasis on openness and transparency with data so others can replicate the analysis and the community can come up with ways to make best practices accessible to anyone. [1] http://en.wikiquote.org/wiki/George_E._P._Box http://en.wikiquote.org/wiki/George_E._P._Box