7 ms·
Google Search Terms Can Predict the Stock Market
- Symmetry 13y agoIgnoring methodology concerns for a moment, there have been times in the past where people have found statistical regularities like this in the stock market. Sometimes people have made money on these discoveries. If not someone else will as soon as the researcher publishes. But making money on them gets rid of them, and given that this is now publicly available information those regularities certainly no longer exist, if they ever existed in the first place. EDIT: Ok, I couldn't resist the xkcd reference: http://xkcd.com/882/ http://xkcd.com/882/
- tgrass 13y agoAn economist and his friend are walking down the street when the friend sees a ten dollar bill on the sidewalk. “Look,” he says, “it’s a ten dollar bill”. “Nonsense,” says the economist. “If that was a ten dollar bill, someone would have picked it up by now.”
- Symmetry 13y agoI'm (and economists) aren't saying that there are never $10 bills on the sidewalk, just that they don't stay there for long after someone notices them. Stock prices went down on the weekends for decades before someone noticed. But once it was noticed it stopped pretty quickly.
- tgrass 13y agoUnderstood. No offense, just being goofy.
- mikeyouse 13y agoI'm more interested in the persistent market inefficiencies; growth vs. value, January effect, low P/E, all interesting examples that haven't been arbitraged away.
- someben 13y agoCheck out the behavioral finance literature, like the Shleifer book: http://www.amazon.com/Inefficient-Markets-Introduction-Behavioral-Clarendon/dp/0198292279 http://www.amazon.com/Inefficient-Markets-Introduction-Behav...
- apalmer 13y agoI am just trying to double check to make sure I understood so bear with me. Are you telling me that stock prices systematically went down for decades and no one took this into account to make money? there is hope after all...
- a1k0n 13y agoThere's always someone on the other side of a trade.
- kragen 13y agoMost of the time, the economist is right about that: the vast majority of ten-dollar-bill-appearing things found on sidewalks are obviously not real ten-dollar bills once you pick them up. They're advertisements for nightclubs or get-rich-quick schemes and whatnot.
- dawilster 13y agoI was thinking of something very similar the other day but in regards to bitcoins. They are very volatile at this stage and whenever they're in the public eye they either seem to increase or decrease in value. Perhaps somebody could perform a study as to a relation between the volume of bitcoin related search terms and it's market value.
- jvanderwal 13y agoI'm actually looking into this a little right now since I happen to work for a media monitoring company. I can tell you there is a definite correlation between them, but haven't been able to determine if it's a leading or lagging indicator.
- hammock 13y agoAs soon as someone figures out bitcoin derivatives (e.g. options), you'll be able to buy or sell the volatility. More broadly, such a development may also stabilize those price fluctuations.
- someben 13y agoWhat is your source for the ole' "derivatives reduce volatility in the spot market" assumption?
- fennecfoxen 13y agoTheory. If you expect that the price of bitcoins is going to be somewhere in the next month, but it's wildly deviated from that on the spot market, there is a monetary incentive to buy/sell bitcoin until the prices are in line with long-term expectations -- discounted for risk, of course. Derivatives like futures options reduce that risk, because you can lock in that future price right now. If anyone's offering, that is. Derivatives could also reduce risk for users. For instance: you're a business that accepts bitcoins for payment. But the price of bitcoins can fall by 50% in a day! So you'd better transfer those bitcoins to cash the moment you get them, or you're subject to losing half your cash any day. Something that's too hot to handle like that kind of damages the ability to use it as a medium of exchange, doesn't it? But you could alternatively buy bitcoin currency futures, essentially locking in your current price. So maybe you could hang onto those bitcoins a while and use them to pay people for your referral program or something. (Oh, sure, there's a cost associated with it, but there are costs associated with translating cash back and forth as well. Depending on the price of the derivatives, it might make sense.)
- rm999 13y agoCool paper, but the fact that a silly keyword like 'color' is so significant without an author explanation makes me question their results. I really wish they dug deeper and explained unintuitive results like that. There may be a bug in their experiment. Also, why would they use the Dow Jones Industrial Average? It is a ridiculously bad average of stock market performance for many reasons. This planet money podcast goes into why: http://www.npr.org/blogs/money/2013/03/12/174139347/episode-443-dont-believe-the-hype http://www.npr.org/blogs/money/2013/03/12/174139347/episode-...
- PaulHoule 13y agoLots of people use the Dow precisely because it is a bad indicator. If they don't like the conclusion they get with something better, like the W5000 or the S&P500, they can try it again with the Dow for a "second opinion." The whole methodology is a joke anyway. If you evaluate a huge number of search terms, some of them are going do better than others. So terms that might mean something ("debt") get mixed up with terms like ("color") that got lucky. What you have to do is come up with one strategy that trains a classifier on the combined words, tunes it up with logistic regression and derives optimal trading actions from that. And you've got to factor in what you're paying to the broker and the market makers.
- Jabbles 13y agoSometimes known as data dredging. https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging
- Tobu 13y agoThis has led to some interesting neurological findings: Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg http://prefrontal.org/files/posters/Bennett-Salmon-2009.jpg via http://www.metafilter.com/85322/The-Wisdom-of-Salmon http://www.metafilter.com/85322/The-Wisdom-of-Salmon An extreme example, illustrating a common mistake in the field. https://xkcd.com/882/ https://xkcd.com/882/ has the gist of it.
- ericb 13y agoHow do you get access to google search terms for something like this? Also, anyone have any other hypothesis about search terms that would predict movement?
- teraflop 13y agohttp://www.google.com/trends/ http://www.google.com/trends/ Hal Varian, Google's chief economist, has been talking about using Google Trends this way for a while: http://www.frbsf.org/economics/conferences/1103/Varian-part_1.pdf http://www.frbsf.org/economics/conferences/1103/Varian-part_...
- pdxandi 13y agoYou use Google Trends: http://www.google.com/trends/ http://www.google.com/trends/ What's interesting is that I had this exact idea with precious metals. The frequency of the terms searched (like gold or buy gold) matches the gold spot market index impressively closely.
- DoubleMalt 13y agoGoogle search terms could also predict a completely random time series. You just have to pick the right search terms. If you look at enough words the chances, that some of them have predictive value approaches 100%.
- Iterated 13y agoI used to backtest trading indicators a lot and the majority of time the indicator would work on a stock like Apple but wouldn't work on a stock like Tyson Foods. Or any other stock. I was just fitting the curve.
- someben 13y agoOut-of-sample backtest or it didn't happen. Out of a bunch of random search queries, chances are several of them will be "predictive" of futures moves in an index. Big whoop.
- xedarius 13y agoMany many moons ago I had a similar idea, although my idea was to spot patterns of eBay purchases. I had a feeling that people in various parts of the world would buy certain types of items depending on economic trends. Once I had identified purchasing trends, it seemed to me that I might be able to predict trends.
- deleted 13y ago[deleted]
- zazerr 13y agoRelated: The time Eric Schmidt said "One day we had a conversation where we figured we could just try to predict the stock market, and then we decided it was illegal. So we stopped doing that."
- someben 13y agoOr it would cannibalize their real business, reputation-wise. Imagine if Quicken started trading TurboTax data...
- fennecfoxen 13y agoWhy would it be illegal? :( I mean, subject to a variety of risk and regulatory issues, perhaps, and not a core competency, and a variety of other things, but... the fundamental reason for insider-trading laws is to make sure the insiders don't abuse their positions and act against the interests of the company's owners (shareholders) by trading in stock tips instead of building shareholder value. If you get knowledge independently -- like if you sent your analyst out to count the number of cars in a firm's parking lots to estimate hiring/firing -- that's all well and good (and is something that hedge funds actually do.) Heck, if it works, this sort of thing would be great. Moving the market earlier means fewer people buy and sell companies at the wrong price. That sort of knowledge is worth billions. (Think about it from the perspective of startups: if you could see the future and know whether a company would work out before you even founded it, then you would build only successful companies, and you'd be certain they'd be funded well. This is but a small fraction of that power, but it's still quite meaningful.)
- mynegation 13y agoThe legality of information you obtain is not about whether you obtained it independently, but based on whether it is public knowledge (or can be derived from it) or not. Let's say you are not connected to company X, but you have a friend that works there. One day your friend tells you material fact. Now, your friend probably broke a couple of rules, but _you_ cannot control what people tell you and you cannot be held liable for what you hear. However, acting on this information is illegal. Even if you presumably do not use this information, but trade shares of company X before the information you know becomes public, you are in murky waters. Same with Google. Google cannot control what people search for. However, acting on this information would be illegal, unless they are absolutely sure that information in the search terms is consistent with public knowledge about the company.
- alakin 13y agoBeen playing around with google trends and trading for a little while now, and there could be something to this. If anyone in SF is also into this stuff, we should grab coffee. Email is antonlakin at gmail.
- bdg 13y agoThis concept is very similar to the "Twitter can predict the stock market" from a long time back: http://www.wired.com/wiredscience/2010/10/twitter-crystal-ball/ http://www.wired.com/wiredscience/2010/10/twitter-crystal-ba... See also, a collection of other related links (helpful if you want to try your hand at making something like this): https://news.ycombinator.com/item?id=2055857 https://news.ycombinator.com/item?id=2055857
- someben 13y agoThat Twitter paper has been torn to shreds over the last couple of years. Here is me snarking: http://blog.someben.com/2011/05/sour-grapes-seven-reasons-why-that-twitter-prediction-model-is-cooked/ http://blog.someben.com/2011/05/sour-grapes-seven-reasons-wh...
- ianstallings 13y agoSomeone somewhere at a hedge fund just chuckled and said "ya think?"
- st8ic 13y ago1) This has been around for many, many years. 2) No free lunch.
- AznHisoka 13y agoDon't think it's that simple. Example: There's a sudden burst in searches for "Macys". They could have started a huge advertising campaign. Turns out that campaign costed them billions of dollar, and didnt have high ROI.
- fennecfoxen 13y agoThis seems more like a macroeconomic-leading-indicator strategy. The algorithm worked against the Dow Jones Industrials, not against individual stocks.
- mattparcher 13y agoSomewhat related, on a more micro level: some have suggested a trend in which news about Anne Hathaway drives up the price Berkeshire Hathaway’s stock. Six examples from 2008-2010: http://www.huffingtonpost.com/dan-mirvish/the-hathaway-effect-how-a_b_830041.html http://www.huffingtonpost.com/dan-mirvish/the-hathaway-effec... A computer scientist who works with hedge funds: "We come across all sorts of strange things in our line of business, strange correlations" http://www.theatlantic.com/technology/print/2011/03/does-anne-hathaway-news-drive-berkshire-hathaways-stock/72661/ http://www.theatlantic.com/technology/print/2011/03/does-ann... As the intelligence of our technology grows, I find it amusing to consider these less-logical patterns in the algorithms (indirectly) responsible for so much of our economy.
- OGC 13y agoSo it's just "Correlation does not imply causation"?
- brador 13y agoHow easy is it to scrape Google trends? Is there an API?
- andylei 13y agogood luck getting 'color' to work out of sample
- Guvante 13y agoYou are always going to be able to find trends between data points if you have enough information. In order to tell whether you actually have predictable information you need to do a double test, which they don't seem to mention anywhere. Typically you split your data pool in half and use data analysis to determine some kind of relation, such as the change in "color" seems to correlate to a reverse change in DOW Jones. You then generate a prediction algorithm based on that. Finally you use your prediction system on the other half of your data, to see if you are actually predicting or just correlating. Feel free to adjust any of your methodology but make sure you don't include the second half in your generation step or else you have only shown correlation not prediction.