6 ms·

Cool paper, but the fact that a silly keyword like 'color' is so significant without an author explanation makes me question their results. I really wish they d

by rm999 13y ago

Cool paper, but the fact that a silly keyword like 'color' is so significant without an author explanation makes me question their results. I really wish they dug deeper and explained unintuitive results like that. There may be a bug in their experiment. Also, why would they use the Dow Jones Industrial Average? It is a ridiculously bad average of stock market performance for many reasons. This planet money podcast goes into why: http://www.npr.org/blogs/money/2013/03/12/174139347/episode-443-dont-believe-the-hype http://www.npr.org/blogs/money/2013/03/12/174139347/episode-...

PaulHoule 13y ago
Lots of people use the Dow precisely because it is a bad indicator. If they don't like the conclusion they get with something better, like the W5000 or the S&P500, they can try it again with the Dow for a "second opinion." The whole methodology is a joke anyway. If you evaluate a huge number of search terms, some of them are going do better than others. So terms that might mean something ("debt") get mixed up with terms like ("color") that got lucky. What you have to do is come up with one strategy that trains a classifier on the combined words, tunes it up with logistic regression and derives optimal trading actions from that. And you've got to factor in what you're paying to the broker and the market makers.
- Jabbles 13y ago
  Sometimes known as data dredging. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging
  - Tobu 13y ago
    This has led to some interesting neurological findings: Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg via http://www.metafilter.com/85322/The-Wisdom-of-Salmon http://www.metafilter.com/85322/The-Wisdom-of-Salmon An extreme example, illustrating a common mistake in the field. https://xkcd.com/882/ https://xkcd.com/882/ has the gist of it.
    - shirederby 13y ago
      The "Salmon study" was an illustration of the necessity of correcting for multiple comparisons in voxel-wise testing. The problem it's addressing isn't itself data dredging, based on the definition linked above on Wikipedia. Uncorrected multiple comparisons are, of course, a big portion of the statistical dubiousness inherent it "data dredging"
- ISL 13y ago
  Relevant technical terms: 'look-elsewhere effect' and 'false alarm rate' http://en.wikipedia.org/wiki/Look-elsewhere_effect http://en.wikipedia.org/wiki/Look-elsewhere_effect http://en.wikipedia.org/wiki/False_positive_rate http://en.wikipedia.org/wiki/False_positive_rate There are more similar considerations. Statisticians have been very busy for the last century or so.
  - rubinelli 13y ago
    As my statistics professor liked to say, "Numbers are meek creatures. If you torture them long enough, they will say whatever you want to hear."
    - asperous 13y ago
      This is the key to the accounting profession
- someben 13y ago
  Logistic regression and other parametric, non-regularized linear learners tend to do poorly with NLP forecasting -type modeling. (They usually overfit.)
  - PaulHoule 13y ago
    You're right. That's why you build another threshold-type learner and then apply logistic regression to convert the score produced by learner A into a probability score. Then you can tune at the exact point of the precision-recall curve that maximizes business value.
- lmg643 13y ago
  The DJIA is a "bad" index but it is highly correlated with the S&P 500. I'm including a link to a study looking at ~70 years of data which shows a 0.95 correlation coefficient between DJIA and SP500. http://www-stat.wharton.upenn.edu/~steele/Courses/434/434Context/Indices/DowFlaws.pdf http://www-stat.wharton.upenn.edu/~steele/Courses/434/434Con... So, while the methodology might be spurious (random search terms may give false positives) i don't think the use of the Dow vs SP500 is a key criticism.
nobbis 13y ago
"Color" is slang for opinion or prediction in finance, e.g. a trader could ask an analyst for some "color" on a company before he goes and buys its stock.
- sakai 13y ago
  Sure, but that's not likely the meaning in the average Google search term with the word "color" (or whatever exactly is being displayed here).
math 13y ago
For what it's worth, I came across this yesterday and tried replicating their results. I didn't use exactly the same process - I used the S&P 500 and the trading strategy was not exactly the same (because i'm using a framework that makes something else easier to try). But my approach was close enough that I would expect to see similar results if their results were robust enough to be of any use. I didn't get similar results, in fact 'debt' went the other way, and I calculated an associated p value of about 0.8 (i.e. no significance whatsoever). Of course I could have made a mistake, and having come this far going to spend some time double checking everything today, but it'd also surprise me greatly if there was money lying around for the taking this easily..

by rm999 13y ago

PaulHoule 13y ago

Lots of people use the Dow precisely because it is a bad indicator. If they don't like the conclusion they get with something better, like the W5000 or the S&P500, they can try it again with the Dow for a "second opinion." The whole methodology is a joke anyway. If you evaluate a huge number of search terms, some of them are going do better than others. So terms that might mean something ("debt") get mixed up with terms like ("color") that got lucky. What you have to do is come up with one strategy that trains a classifier on the combined words, tunes it up with logistic regression and derives optimal trading actions from that. And you've got to factor in what you're paying to the broker and the market makers.

Jabbles 13y ago
Sometimes known as data dredging. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging
- Tobu 13y ago
  This has led to some interesting neurological findings: Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg via http://www.metafilter.com/85322/The-Wisdom-of-Salmon http://www.metafilter.com/85322/The-Wisdom-of-Salmon An extreme example, illustrating a common mistake in the field. https://xkcd.com/882/ https://xkcd.com/882/ has the gist of it.
  - shirederby 13y ago
    The "Salmon study" was an illustration of the necessity of correcting for multiple comparisons in voxel-wise testing. The problem it's addressing isn't itself data dredging, based on the definition linked above on Wikipedia. Uncorrected multiple comparisons are, of course, a big portion of the statistical dubiousness inherent it "data dredging"
ISL 13y ago
Relevant technical terms: 'look-elsewhere effect' and 'false alarm rate' http://en.wikipedia.org/wiki/Look-elsewhere_effect http://en.wikipedia.org/wiki/Look-elsewhere_effect http://en.wikipedia.org/wiki/False_positive_rate http://en.wikipedia.org/wiki/False_positive_rate There are more similar considerations. Statisticians have been very busy for the last century or so.
- rubinelli 13y ago
  As my statistics professor liked to say, "Numbers are meek creatures. If you torture them long enough, they will say whatever you want to hear."
  - asperous 13y ago
    This is the key to the accounting profession
someben 13y ago
Logistic regression and other parametric, non-regularized linear learners tend to do poorly with NLP forecasting -type modeling. (They usually overfit.)
- PaulHoule 13y ago
  You're right. That's why you build another threshold-type learner and then apply logistic regression to convert the score produced by learner A into a probability score. Then you can tune at the exact point of the precision-recall curve that maximizes business value.
lmg643 13y ago
The DJIA is a "bad" index but it is highly correlated with the S&P 500. I'm including a link to a study looking at ~70 years of data which shows a 0.95 correlation coefficient between DJIA and SP500. http://www-stat.wharton.upenn.edu/~steele/Courses/434/434Context/Indices/DowFlaws.pdf http://www-stat.wharton.upenn.edu/~steele/Courses/434/434Con... So, while the methodology might be spurious (random search terms may give false positives) i don't think the use of the Dow vs SP500 is a key criticism.

nobbis 13y ago

"Color" is slang for opinion or prediction in finance, e.g. a trader could ask an analyst for some "color" on a company before he goes and buys its stock.

sakai 13y ago
Sure, but that's not likely the meaning in the average Google search term with the word "color" (or whatever exactly is being displayed here).

math 13y ago

For what it's worth, I came across this yesterday and tried replicating their results. I didn't use exactly the same process - I used the S&P 500 and the trading strategy was not exactly the same (because i'm using a framework that makes something else easier to try). But my approach was close enough that I would expect to see similar results if their results were robust enough to be of any use. I didn't get similar results, in fact 'debt' went the other way, and I calculated an associated p value of about 0.8 (i.e. no significance whatsoever). Of course I could have made a mistake, and having come this far going to spend some time double checking everything today, but it'd also surprise me greatly if there was money lying around for the taking this easily..