5 ms·

Can a Machine Learning Model Predict the SP500 by Looking at Candlesticks?

28 points by mariofilho 8y ago

jeletonskelly 8y ago
Isn't that what quantitative trading is?
- mruts 8y ago
  Quantitative trading isn't trying to predict the S&P500. It's trying to predict the distribution of returns, which is a (slightly) easier problem. Quants are just trying to make money, not predict the future. And judging by the returns of quant funds like Bridgewater or Renaissance, the answer is that while they might not be able to predict the S&P, they can generate alpha year over year.
- ivalm 8y ago
  I think most quants trade on 1) volatility (ie mis-pricing of the expected distribution of returns) 2) front running large orders 3) rapid news analysis I don't think there is a successful model for market directional prediction based on previous price action (ie charting).
- codesushi42 8y ago
  No. Hedge funds make money on asset allocation and asymmetric information, i.e. exclusive market information (see Quandl). They don't make alpha with complex trading rules.
notroot 8y ago
Yes, no, maybe, I don't know Can you repeat the question?
- notroot 8y ago
  Remind me what the ROC curve looks like for `can` again.
lawlorino 8y ago
No. Saved you a click.
kuhhk 8y ago
The book “A random walk down Wall Street” concluded “NO”
- mruts 8y ago
  I mean, that book was looking at technical analysis and charting. Also, he was just looking at mutual funds (which can only go long and suffer from a host of other problems). Hedge funds weren't really invented/popular back then and quant investing was practically non-existent. Economists love to talk about EMH, and there's a great joke that illustrates the difference between economists and traders: The economist is asked what he would do if he saw a $20 dollar on the street. He replies "well it wouldn't ever happen, because someone would have already picked it up!" The trader is the one picking up the $20, and the economist is the one who never believes it can exist. Back to the point, it seems clear to me that there are few quant hedge funds that can have consistently outperformed the market that disproves the null hypothesis (no out performance) with a P < 0.05. Names like RenTech, Bridgewater, AQR, 2 Sigma.
  - ramblerman 8y ago
    > I mean, that book was looking at technical analysis and charting Trying to predict the future by looking at candlesticks is pretty much the definition of charting.
    - mruts 8y ago
      Yeah that’s a good point. But I think the author of a random walk on Wall St was trying to make a stronger point: that all out performance on Wall St is just a random walk.
  - astazangasta 8y ago
    >it seems clear to me that there are few quant hedge funds that can have consistently outperformed the market that disproves the null hypothesis (no out performance) with a P < 0.05 The meaning p < 0.05 is that by random chance you expect to find 5% of companies doing this well relative to the rest with no actual underlying cause. The existence of a few companies that manage this is proof of exactly nothing about those companies.
MisterOctober 8y ago
12:40am : restate my assumptions
- vingummibamsen 8y ago
  π
- sj4nz 8y ago
  :) For anyone else who doesn't recognize this: https://www.imdb.com/title/tt0138704/ https://www.imdb.com/title/tt0138704/
module0000 8y ago
No, not at all. Move along. Everyone tries this initially, and (quickly) learns this lesson. I suppose it cost you a lot less to find out the answer via posting to HN though.
ataturk 8y ago
No.
jimrandomh 8y ago
"The reason the stock market is hard to predict is because it is a prediction." --Andrew Critch
ohiovr 8y ago
If a well trained ai at scale to read the sticks and only the sticks fails, why would it be considered possible for humans to do it?
- ivalm 8y ago
  They can't. Charting is garbage, everyone knows this.
  - ohiovr 8y ago
    https://www.investopedia.com/university/charts/ https://www.investopedia.com/university/charts/ seems some people do. Charting is probably bunk and it might be impossible to get reliable trades. if that could be scientifically proven people wouldn’t lose their money tryng to use it.
    - TuringNYC 8y ago
      >> seems some people do. Yes, people do charting. No, they dont actually make money on it. There is a vast market to manage money very profitably for anyone who can demonstrate consistent performance (mutual funds, etfs, hedge funds, etc.) There are also many sites now that will audit your performance and prove you are performing well by tracing outcomes. If indeed charting was profitable, there would be proof of it and people trying to profit off it by managing money using it.
uptownfunk 8y ago
I was always curious if you trained a model on a literal visual representation (pixels/image) of the charts or candlesticks, would the model be able to “see” something that we can’t.
- IshKebab 8y ago
  Of course not. It's the same data but presented in a harder to process format.
  - uptownfunk 8y ago
    It may be the similar but as they say, a picture speaks a thousand words, the visual features that a CNN might pick up could be something completely different than the features someone could think of. It is all about data representation. Hypothetically, the data representation shouldn't matter, but I think it is like viewing the optimization surface from a different angle, it is possible to get something different out of it.
- mariofilho 8y ago
  This is something I got curious about too. It's very likely the answer is no, but I would like to test it at some point.
HockeyPlayer 8y ago
Using RMSE doesn't make sense to me, your losses are linear to how wrong you are. Shouldn't the error just be how far off you were?
Nasrudith 8y ago
I wonder how well a literal candlestick model would work - as in literally acting like a diviner watching candles melt and trying to map something from it. Given how much is up to chance and comingled variables (how the market performs correlated to temperature for instance could make it technically correct if epistemologically utter stupid) it may wind up ironically better than random chance. Still not something to stake your life savings on.
stevespang 8y ago
As a former tradestation trader, whether you use bar charts or candlesticks or whatever, they all simply reflect price. Using past historical data for backtesting is very tricky because too often in continuous improvement you will merely "fit" the algorithms to the historical data - - - but when you trade it forward for a year or more you get to see how really bad your model sucks. I once had a algorithm that made millions on 5 years of historical SP futures data (15 min bars), but testing it on live data foward it puked within 2 months and lost a fortune.
dbs 8y ago
How to set yourself to failure by starting with "one of the most widely known techniques". You can't outperform the average by using the same tools as everybody else.

28 points by mariofilho 8y ago

jeletonskelly 8y ago

Isn't that what quantitative trading is?

mruts 8y ago
Quantitative trading isn't trying to predict the S&P500. It's trying to predict the distribution of returns, which is a (slightly) easier problem. Quants are just trying to make money, not predict the future. And judging by the returns of quant funds like Bridgewater or Renaissance, the answer is that while they might not be able to predict the S&P, they can generate alpha year over year.
ivalm 8y ago
I think most quants trade on 1) volatility (ie mis-pricing of the expected distribution of returns) 2) front running large orders 3) rapid news analysis I don't think there is a successful model for market directional prediction based on previous price action (ie charting).
codesushi42 8y ago
No. Hedge funds make money on asset allocation and asymmetric information, i.e. exclusive market information (see Quandl). They don't make alpha with complex trading rules.

notroot 8y ago

Yes, no, maybe, I don't know Can you repeat the question?

notroot 8y ago
Remind me what the ROC curve looks like for `can` again.

lawlorino 8y ago

No. Saved you a click.

kuhhk 8y ago

The book “A random walk down Wall Street” concluded “NO”

mruts 8y ago
I mean, that book was looking at technical analysis and charting. Also, he was just looking at mutual funds (which can only go long and suffer from a host of other problems). Hedge funds weren't really invented/popular back then and quant investing was practically non-existent. Economists love to talk about EMH, and there's a great joke that illustrates the difference between economists and traders: The economist is asked what he would do if he saw a $20 dollar on the street. He replies "well it wouldn't ever happen, because someone would have already picked it up!" The trader is the one picking up the $20, and the economist is the one who never believes it can exist. Back to the point, it seems clear to me that there are few quant hedge funds that can have consistently outperformed the market that disproves the null hypothesis (no out performance) with a P < 0.05. Names like RenTech, Bridgewater, AQR, 2 Sigma.
- ramblerman 8y ago
  > I mean, that book was looking at technical analysis and charting Trying to predict the future by looking at candlesticks is pretty much the definition of charting.
  - mruts 8y ago
    Yeah that’s a good point. But I think the author of a random walk on Wall St was trying to make a stronger point: that all out performance on Wall St is just a random walk.
- astazangasta 8y ago
  >it seems clear to me that there are few quant hedge funds that can have consistently outperformed the market that disproves the null hypothesis (no out performance) with a P < 0.05 The meaning p < 0.05 is that by random chance you expect to find 5% of companies doing this well relative to the rest with no actual underlying cause. The existence of a few companies that manage this is proof of exactly nothing about those companies.

MisterOctober 8y ago

12:40am : restate my assumptions

vingummibamsen 8y ago
π
sj4nz 8y ago
:) For anyone else who doesn't recognize this: https://www.imdb.com/title/tt0138704/ https://www.imdb.com/title/tt0138704/

module0000 8y ago

No, not at all. Move along. Everyone tries this initially, and (quickly) learns this lesson. I suppose it cost you a lot less to find out the answer via posting to HN though.

ataturk 8y ago

No.

jimrandomh 8y ago

"The reason the stock market is hard to predict is because it is a prediction." --Andrew Critch

ohiovr 8y ago

If a well trained ai at scale to read the sticks and only the sticks fails, why would it be considered possible for humans to do it?

ivalm 8y ago
They can't. Charting is garbage, everyone knows this.
- ohiovr 8y ago
  https://www.investopedia.com/university/charts/ https://www.investopedia.com/university/charts/ seems some people do. Charting is probably bunk and it might be impossible to get reliable trades. if that could be scientifically proven people wouldn’t lose their money tryng to use it.
  - TuringNYC 8y ago
    >> seems some people do. Yes, people do charting. No, they dont actually make money on it. There is a vast market to manage money very profitably for anyone who can demonstrate consistent performance (mutual funds, etfs, hedge funds, etc.) There are also many sites now that will audit your performance and prove you are performing well by tracing outcomes. If indeed charting was profitable, there would be proof of it and people trying to profit off it by managing money using it.

uptownfunk 8y ago

I was always curious if you trained a model on a literal visual representation (pixels/image) of the charts or candlesticks, would the model be able to “see” something that we can’t.

IshKebab 8y ago
Of course not. It's the same data but presented in a harder to process format.
- uptownfunk 8y ago
  It may be the similar but as they say, a picture speaks a thousand words, the visual features that a CNN might pick up could be something completely different than the features someone could think of. It is all about data representation. Hypothetically, the data representation shouldn't matter, but I think it is like viewing the optimization surface from a different angle, it is possible to get something different out of it.
mariofilho 8y ago
This is something I got curious about too. It's very likely the answer is no, but I would like to test it at some point.

HockeyPlayer 8y ago

Using RMSE doesn't make sense to me, your losses are linear to how wrong you are. Shouldn't the error just be how far off you were?

Nasrudith 8y ago

I wonder how well a literal candlestick model would work - as in literally acting like a diviner watching candles melt and trying to map something from it. Given how much is up to chance and comingled variables (how the market performs correlated to temperature for instance could make it technically correct if epistemologically utter stupid) it may wind up ironically better than random chance. Still not something to stake your life savings on.

stevespang 8y ago

As a former tradestation trader, whether you use bar charts or candlesticks or whatever, they all simply reflect price. Using past historical data for backtesting is very tricky because too often in continuous improvement you will merely "fit" the algorithms to the historical data - - - but when you trade it forward for a year or more you get to see how really bad your model sucks. I once had a algorithm that made millions on 5 years of historical SP futures data (15 min bars), but testing it on live data foward it puked within 2 months and lost a fortune.

dbs 8y ago

How to set yourself to failure by starting with "one of the most widely known techniques". You can't outperform the average by using the same tools as everybody else.