18 ms·

Lessons in statistics: How not to analyze time series

28 points by xtacy 12y ago

phrixus 12y ago
Wow, the extensive comment section is eye-opening to me, in how little consensus there exists about the validity of models, line-fitting, and prediction.
- flashingleds 12y ago
  I feel like the real lesson I learned here was not about time series but about reading the comments section for any discussion involving global warming
- dxbydt 12y ago
  I am amazed at the level of erudition in the comments section as well. Shows what a highly literate intellectual audience that part of the world must be. Had this article been printed in the Times, you'd have the usual republicans vs Krugman name calling and none of this scientific minutiae.
nkurz 12y ago
This is a very interesting piece to read. The amazing thing to me is how smart people can look at the same evidence and come to very different conclusions. To get the feel of this piece, you really need to read it along with the comments. The author of the piece is a very talented statistician, with a PhD in Mathematical Statistics (I hesitate to think of what other kind of statistics someone might study?), a Masters in Atmospheric physics, and a BA in Meteorology. Without question, he is very extremely well qualified on paper to make write the post he did: http://wmbriggs.com/blog/?page_id=1085 http://wmbriggs.com/blog/?page_id=1085 The main critical commentator is David Appell. He's a freelance writer who's been concentrating and reporting on global warming issues for many years. He's got a PhD, Masters, and BA in Physics, has interviewed most of the main scientists in the field multiple times, and understands the issues as well as or better than anyone else writing about it in the mainstream press: http://davidappell.com/ http://davidappell.com/ And it appears that the two completely disagree about practically every line in the article! Without doubting the credentials or understanding of either, I can't shake the feeling that one is following the data wherever it leads, and the other could rationalize any result under the sun and not be knocked off a single one of his talking points. And I'm sure that others can read the piece and the comments and have exactly the opposite reaction. What is it about their prose that can produce generate in me this amount of certainty and trust? Somehow, one is able to signal to me that they share my worldview, and thus I'm willing the trust them on the details I'm unfamiliar with. The other loses my trust within the first paragraph. At some level I know that neither of these responses can be fully trusted, and yet I can't shake the feeling that one can be trusted and the other cannot. How can this be?
- duckingtest 12y ago
  >At some level I know that neither of these responses can be fully trusted, and yet I can't shake the feeling that one can be trusted and the other cannot. How can this be? David Appell said it himself that what he believes isn't based in fact and can't be disproven in any way, in other words, a religious belief. Seems you just prefer science to religion. "In climate science you can’t “predict” anything — climate models aren’t capable of predicting, because no one can read the future."
signalsmith 12y ago
As a mathematician, that article was a little embarrassing to wade through. While there are valid criticisms of the way statistics are sometimes misused in science, pretty much every one of them comes from lack of understanding about how statistical models work - scientists reaching for a familiar test and following a formula they were taught. I can't blame them too much - understanding the true purpose and nature of statistical models is HARD (my recommended step one: become a Bayesian). What we need is for more people to recognise when they don't have that understanding and work with somebody who does. What Briggs seems to have done, though, is decided that because HE doesn't understand statistical inference and modelling, that statistics are bunk. Taking a simplistic definition of "trend" like "the second-half average is higher than the first" and turning that into a boolean yes/no answer is the statistical equivalent of being an anti-vaccer. The most frustrating thing, though, is that all the alternative definitions of "trend" he defines can actually be expressed as statistical models! The issue is that when you express these definitions/tests for "trend" as models, you see that the statements each model makes about the underlying system are very problematic. TL;DR - Briggs doesn't understand statistical modelling, and has therefore concluded that his home-rolled tests are just as good.
- nkurz 12y ago
  (responding both to you and 'duckingtest') No, the 'answer' is not that the person you disagree with "believes in magic" or "doesn't understand" the field they work in as respected experts. Appell goes on to try to explain why climate modelling is concerned with 'projection' rather than 'prediction', and has a very clear view of the difference between science and magic. Briggs has been a professor of statistics at Cornell, written several book on statistics, and published dozens of peer reviewed papers. Claiming that he is the "statistical equivalent of an anti-vaccer" might be a useful rhetorical strategy but unless you can point to some prominent mainstream career vaccine researchers that you'd also lump into this category, it has no basis in truth. Playing to a home crowd is easy, but has left us where we are with two locked down groups each trying to persuade the undecided middle that their opponent is the devil incarnate. This doesn't lead anywhere desirable, just to further entrenchment of positions. The "way out" is trying to formulate criticism that can be understood by the party being criticized and influences them and their followers to reevaluate their beliefs and approaches. I think it has to be something stylistic that we as readers associate with a world-view (a dog whistle) rather than the actual content of the statements. Take the paragraph in "Lesson 1": If you say the mixed marriage of splicing the disjoint series does not matter, you are making a judgment. Is it true? How can you prove it? It doesn’t seem true on its face. Significance tests are circular arguments here. After the marriage, you are left with unquantifiable uncertainty. I'd assert that this paragraph can only appear on the 'denier' side of things. Is this because it embraces uncertainty, and implies we should not act until we have properly quantified it? One of the most interesting exchanges in the comments is between 'Sheri' and 'Brandon Gates', on whether it's better to accept a known-flawed model or to throw it out and have no model at all. My request to 'signalsmith', concentrating on Briggs because his positions are more self-contained: Re-reread Briggs article, ignoring the set-up paragraphs that explain what he's criticizing and why. That is, blind yourself as best as you can to the side that he is taking. Stop at the point where your pulse starts to rise and report back. Then let's see how 'duckingtest' responds to the same passage.

28 points by xtacy 12y ago

Wow, the extensive comment section is eye-opening to me, in how little consensus there exists about the validity of models, line-fitting, and prediction.

flashingleds 12y ago
I feel like the real lesson I learned here was not about time series but about reading the comments section for any discussion involving global warming
dxbydt 12y ago
I am amazed at the level of erudition in the comments section as well. Shows what a highly literate intellectual audience that part of the world must be. Had this article been printed in the Times, you'd have the usual republicans vs Krugman name calling and none of this scientific minutiae.

nkurz 12y ago

This is a very interesting piece to read. The amazing thing to me is how smart people can look at the same evidence and come to very different conclusions. To get the feel of this piece, you really need to read it along with the comments. The author of the piece is a very talented statistician, with a PhD in Mathematical Statistics (I hesitate to think of what other kind of statistics someone might study?), a Masters in Atmospheric physics, and a BA in Meteorology. Without question, he is very extremely well qualified on paper to make write the post he did: http://wmbriggs.com/blog/?page_id=1085 http://wmbriggs.com/blog/?page_id=1085 The main critical commentator is David Appell. He's a freelance writer who's been concentrating and reporting on global warming issues for many years. He's got a PhD, Masters, and BA in Physics, has interviewed most of the main scientists in the field multiple times, and understands the issues as well as or better than anyone else writing about it in the mainstream press: http://davidappell.com/ http://davidappell.com/ And it appears that the two completely disagree about practically every line in the article! Without doubting the credentials or understanding of either, I can't shake the feeling that one is following the data wherever it leads, and the other could rationalize any result under the sun and not be knocked off a single one of his talking points. And I'm sure that others can read the piece and the comments and have exactly the opposite reaction. What is it about their prose that can produce generate in me this amount of certainty and trust? Somehow, one is able to signal to me that they share my worldview, and thus I'm willing the trust them on the details I'm unfamiliar with. The other loses my trust within the first paragraph. At some level I know that neither of these responses can be fully trusted, and yet I can't shake the feeling that one can be trusted and the other cannot. How can this be?

duckingtest 12y ago
>At some level I know that neither of these responses can be fully trusted, and yet I can't shake the feeling that one can be trusted and the other cannot. How can this be? David Appell said it himself that what he believes isn't based in fact and can't be disproven in any way, in other words, a religious belief. Seems you just prefer science to religion. "In climate science you can’t “predict” anything — climate models aren’t capable of predicting, because no one can read the future."

signalsmith 12y ago

As a mathematician, that article was a little embarrassing to wade through. While there are valid criticisms of the way statistics are sometimes misused in science, pretty much every one of them comes from lack of understanding about how statistical models work - scientists reaching for a familiar test and following a formula they were taught. I can't blame them too much - understanding the true purpose and nature of statistical models is HARD (my recommended step one: become a Bayesian). What we need is for more people to recognise when they don't have that understanding and work with somebody who does. What Briggs seems to have done, though, is decided that because HE doesn't understand statistical inference and modelling, that statistics are bunk. Taking a simplistic definition of "trend" like "the second-half average is higher than the first" and turning that into a boolean yes/no answer is the statistical equivalent of being an anti-vaccer. The most frustrating thing, though, is that all the alternative definitions of "trend" he defines can actually be expressed as statistical models! The issue is that when you express these definitions/tests for "trend" as models, you see that the statements each model makes about the underlying system are very problematic. TL;DR - Briggs doesn't understand statistical modelling, and has therefore concluded that his home-rolled tests are just as good.

nkurz 12y ago
(responding both to you and 'duckingtest') No, the 'answer' is not that the person you disagree with "believes in magic" or "doesn't understand" the field they work in as respected experts. Appell goes on to try to explain why climate modelling is concerned with 'projection' rather than 'prediction', and has a very clear view of the difference between science and magic. Briggs has been a professor of statistics at Cornell, written several book on statistics, and published dozens of peer reviewed papers. Claiming that he is the "statistical equivalent of an anti-vaccer" might be a useful rhetorical strategy but unless you can point to some prominent mainstream career vaccine researchers that you'd also lump into this category, it has no basis in truth. Playing to a home crowd is easy, but has left us where we are with two locked down groups each trying to persuade the undecided middle that their opponent is the devil incarnate. This doesn't lead anywhere desirable, just to further entrenchment of positions. The "way out" is trying to formulate criticism that can be understood by the party being criticized and influences them and their followers to reevaluate their beliefs and approaches. I think it has to be something stylistic that we as readers associate with a world-view (a dog whistle) rather than the actual content of the statements. Take the paragraph in "Lesson 1": If you say the mixed marriage of splicing the disjoint series does not matter, you are making a judgment. Is it true? How can you prove it? It doesn’t seem true on its face. Significance tests are circular arguments here. After the marriage, you are left with unquantifiable uncertainty. I'd assert that this paragraph can only appear on the 'denier' side of things. Is this because it embraces uncertainty, and implies we should not act until we have properly quantified it? One of the most interesting exchanges in the comments is between 'Sheri' and 'Brandon Gates', on whether it's better to accept a known-flawed model or to throw it out and have no model at all. My request to 'signalsmith', concentrating on Briggs because his positions are more self-contained: Re-reread Briggs article, ignoring the set-up paragraphs that explain what he's criticizing and why. That is, blind yourself as best as you can to the side that he is taking. Stop at the point where your pulse starts to rise and report back. Then let's see how 'duckingtest' responds to the same passage.