4 ms·

Un-supported financial market data mining is a pet peeve of mine. In this case, there are some critical problems with your analysis: * First, your "N" of 1,57

by DavidSJ 18y ago

Un-supported financial market data mining is a pet peeve of mine. In this case, there are some critical problems with your analysis: * First, your "N" of 1,578 is way overstated, to the extent that we should be inferring that we're looking at 1,578 independent data points. What you seem to have done is to treat every month from January 1871 through June 2002 as a separate data point. However, the past 12 month performance for any two data points within 11 months of each other are correlated (highly correlated the closer in time they are). Further, the 5 year returns of any two data points within 59 months of each other are also correlated (again, very correlated for close-together data points). So you actually have just 26 independent data points here, which is a statistically dubious sample size. For the 10, 20, and 30 year versions, you have 12, 5, and 3 independent data points, respectively -- statistically meaningless. * Second (and for the same reason), your N for each subset of the data corresponding to different timing strategies is even less. Without looking at the historical data, I can't say exactly how many independent data points you really have in each subset, but I know, for example, that in the last 137 years there have been only a few stock market declines of a magnitude of 20 - 50%; certainly nowhere near the 139 that your "N" suggests. So even the 5-year study has a meaningless sample size for the individual triggers. * Third, your subset samples are biased because individual "real" data points are reflected in varying quantities depending on how many adjacent months the signal remained for. So, even if we had enough real data points in theory, your sample is skewed towards those buy signals which lasted longer. * Fourth: you say, "For 10-year investments, investing when the market is down 20% to 50% is probably going to produce better returns than a investment at a random point in time, but it's a close call". But even if your data were accurate, the 10-year charts are such a close call that there's actually no conclusion to be drawn from them. * Finally, you had no prior hypothesis. This is a fatal flaw with many attempts to mine historical stock market return data. Instead, you measured sixteen separate investment strategies (four different buy signals times four different time horizons). Without a prior hypothesis, you have not done a scientific study, but you have just mined data for patterns. In any sufficiently large corpus of data, there will be patterns simply by coincidence. A prior hypothesis is necessary to distinguish those patterns which exist by coincidence from those which we predicted ahead of time due to a correct explanatory framework. Even if your analysis had none of these flaws, we'd need to be wary of it, because of the phenomenon of publication bias (only those studies which show something interesting get noticed by people; but every so often someone predicts something, and the data seems to confirm it, simply because they had an unusual sample despite their having done everything properly). So your conclusions would merely be tentative pending followup studies (on independent data points; which means we'd have to wait decades to start to be confident). But each of these flaws, individually, are enough to completely invalidate any inferences we can make from your data. Don't feel bad, however. People have been drawing un-supported inferences from stock market data forever.

tomsaffell 18y ago
David - Thanks so much for taking the time to write this. I think your observations on the analysis are valid. However, I slightly disagree with your 0th/macro point: You said that, "Un-supported financial market data mining is a pet peeve of mine". Had you said "Un-supported financial market advice is a pet peeve of mine" I would be right there with you. But this is not advice, and I went to lengths to point that out in my blog, but I did a bad job of pointing that out in my comment on this page (I guess that's what I get for rushing). As I say in the introduction on my blog, "I want to know whether history suggests that there is anything to be gained from trying to time the market by buying after a fall". I.e. the motive of this is intellectual curiosity, not to create advice. I published it so that I can get the rich feedback that is so valuable in this process (and which is abundant on HN). Now that we have that subtlety out the way, lets look at the more substantial issues. Let me take these one at a time to explain my thoughts: 1. Yes, you're right, there is a lot of correlation. While there are 1578 months, there are far far few economic cycles since 1871, which is another way to think about the problem you are exposing (also in 2). However, this is the data that we have, and I am taking the approach of 'something is better than nothing'. I would say that what this analysis does do is to answer the question "what would the distribution of returns look like for an investment of x year, using timing strategy y, if employed since 1871 to 2008?". Would you say that is fair? If so, the next question is, "can this be generalized in a way that makes it a useful tool for predicting the future?" My answer would be, "not with significant confidence", which I think is your point - is that right? The next question would then be, "so why bother doing it at all?", and my answer would be, "intellectual curiosity" and maybe, "because it's better than nothing at all", though I can see why you would dispute the latter. 2. This essentially is the same as (1), so again I agree with your observation, but maybe not your conclusion. The "n=139" means that in my sample there were 139 months between 1871 and 2008 where the monthly price was 20%-50% lower than max price from the previous 12-months. I'm fairly sure they're accurate. 3. I'm not sure I understand - please explain further. 4. I think we totally agree here on the statistical validity of the data. Where we differ slightly is the presentation. I have not made any conclusions (again my comment on this page was misleading, but at least provocative :). This is a blog that I am using to get other people to help me answer my questions. It is not a research paper. I talk about what I see, not what I conclude. I looked at what my conclusions would be because I know that when most people read this the easiest way for them to grasp the results is to see a viewpoint of what those results would look like if translated to conclusions. I.e. the "would conclude" comments are a tool to spark debate, which seems to have worked:) That is an easy thing for me to say, I know - it may even sound like a cop-out, but if you critique this post as you would a research paper then you miss the point. 5. I could not agree more. In fact, I make this point often to others. I actually did have a hypothesis, but it wasn't crisply defined, nor did I write it down, nor did I mention it in my blog. All of this is bad practice, I know. I didn't go into this on the post because I wanted to make it more accessible (and shorter), by quickly getting to the results. Again, that's a style choice, and I would never dream of doing it in a research paper. 6. (on publication bias). Again, I wholeheartedly agree. Independent verification is a corner stone of good research - but someone has to go first.. If anyone does ever choose to followup on this then I shall be deeply flattered, and eager to see what comes of it. Thank goodness I'm not making any conclusions.. :) 7. I don't feel bad. I'm grateful that someone has taken time to give detailed feedback.

by DavidSJ 18y ago

tomsaffell 18y ago

David - Thanks so much for taking the time to write this. I think your observations on the analysis are valid. However, I slightly disagree with your 0th/macro point: You said that, "Un-supported financial market data mining is a pet peeve of mine". Had you said "Un-supported financial market advice is a pet peeve of mine" I would be right there with you. But this is not advice, and I went to lengths to point that out in my blog, but I did a bad job of pointing that out in my comment on this page (I guess that's what I get for rushing). As I say in the introduction on my blog, "I want to know whether history suggests that there is anything to be gained from trying to time the market by buying after a fall". I.e. the motive of this is intellectual curiosity, not to create advice. I published it so that I can get the rich feedback that is so valuable in this process (and which is abundant on HN). Now that we have that subtlety out the way, lets look at the more substantial issues. Let me take these one at a time to explain my thoughts: 1. Yes, you're right, there is a lot of correlation. While there are 1578 months, there are far far few economic cycles since 1871, which is another way to think about the problem you are exposing (also in 2). However, this is the data that we have, and I am taking the approach of 'something is better than nothing'. I would say that what this analysis does do is to answer the question "what would the distribution of returns look like for an investment of x year, using timing strategy y, if employed since 1871 to 2008?". Would you say that is fair? If so, the next question is, "can this be generalized in a way that makes it a useful tool for predicting the future?" My answer would be, "not with significant confidence", which I think is your point - is that right? The next question would then be, "so why bother doing it at all?", and my answer would be, "intellectual curiosity" and maybe, "because it's better than nothing at all", though I can see why you would dispute the latter. 2. This essentially is the same as (1), so again I agree with your observation, but maybe not your conclusion. The "n=139" means that in my sample there were 139 months between 1871 and 2008 where the monthly price was 20%-50% lower than max price from the previous 12-months. I'm fairly sure they're accurate. 3. I'm not sure I understand - please explain further. 4. I think we totally agree here on the statistical validity of the data. Where we differ slightly is the presentation. I have not made any conclusions (again my comment on this page was misleading, but at least provocative :). This is a blog that I am using to get other people to help me answer my questions. It is not a research paper. I talk about what I see, not what I conclude. I looked at what my conclusions would be because I know that when most people read this the easiest way for them to grasp the results is to see a viewpoint of what those results would look like if translated to conclusions. I.e. the "would conclude" comments are a tool to spark debate, which seems to have worked:) That is an easy thing for me to say, I know - it may even sound like a cop-out, but if you critique this post as you would a research paper then you miss the point. 5. I could not agree more. In fact, I make this point often to others. I actually did have a hypothesis, but it wasn't crisply defined, nor did I write it down, nor did I mention it in my blog. All of this is bad practice, I know. I didn't go into this on the post because I wanted to make it more accessible (and shorter), by quickly getting to the results. Again, that's a style choice, and I would never dream of doing it in a research paper. 6. (on publication bias). Again, I wholeheartedly agree. Independent verification is a corner stone of good research - but someone has to go first.. If anyone does ever choose to followup on this then I shall be deeply flattered, and eager to see what comes of it. Thank goodness I'm not making any conclusions.. :) 7. I don't feel bad. I'm grateful that someone has taken time to give detailed feedback.