10 ms·

Here's the problem with trying to create your own trading system. How do you back test it to know that it works. If you back test over the past 5 years then y

Here's the problem with trying to create your own trading system. How do you back test it to know that it works. If you back test over the past 5 years then you are only testing your model against a huge bull market. If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets, HFT's, huge numbers of hedge funds and the money they bring, and huge passive investing via ETF's, all those factors were there 20 years ago, but they weren't the major market drivers that they are now. So for back testing you are damned if you do and damned if you don't:( I think I've said this before but in my experience, a decent trading system is 70%+ market insight. And there are only really 3 ways to get this insight, hard earned experience, applying new streams of data to the markets and applying math to existing streams of data to unlock new insights. All 3 are being done every day by scores of MIT Phd's. So if you are writing your own trading models you should start by answering the question of what's your alpha when compared to everyone else....

CPLX 11y ago
> How do you back test it to know that it works. You don't. You turn it on and see if it works. Which does have a certain degree of intellectual rigor to it. > So if you are writing your own trading models you should start by answering the question of what's your alpha when compared to everyone else.... Well right, that's the core question isn't it. But clearly the allure of wanting to test ones own set of biases and insights for alpha is fairly understandable and appealing.
ericjang 11y ago
One clarification - these mom & pop prop shops aren't necessarily competing against big baskets of MIT PhDs. If your book size is only $200K, the big players (i.e. scores of MIT PhDs) won't even bother competing with you on the same strategies. Alpha from these strategies may very well be orthogonal to hedge fund alpha. That said, you're still playing a zero sum game with other mom & pop shops and the general large-scale movements of the market. You need to be confident that you have better information / forecasting ability than the others, or technology infrastructure that enables strategies no one else can execute. Source: I work in the industry.
- deleted 11y ago
  [deleted]
- exelius 11y ago
  Yeah, most people don't realize that finance is a zero-sum game. That leads to arms races which over time remove the lion's share of the profit (companies will spend money on a better solution to a problem until such time as a better solution costs more than the value of the opportunity). I expect this to replicate itself on the low-end as well.
  - joosters 11y ago
    The 'zero-sum game' part is meaningless and adds no insight here. It doesn't matter whether you believe trading/gambling is zero-sum or not. For instance, imagine you have a poker-playing bot at a poker table, where the house takes no cut of the stakes. Zero-sum? Yes. But your bot has to be better than the other players, and win more than its running costs in order to profit. Now imagine the same situation but the house does take a cut. Zero-sum? Not any more. But your bot still has to be better than the other players and its running costs in order to profit. The house 'rake' just increases your operating costs. I think you are confusing the issue with large scale HFT (where companies pay more and more for slightly faster comms), which is a world away from the trading strategies the article is talking about.
    - SonicSoul 11y ago
      Think you missed the point. All he was saying (I believe) that no matter how good your models are it's still a race between the players to get the biggest share of the pot. The more players the less chance you have
      - joosters 11y ago
        That's true of many endeavors, and is unrelated to any 'zero-sum' argument.
    - jsprogrammer 11y ago
kittenfluff 11y ago
> If you back test over the past 5 years then you are only testing your model against a huge bull market. If your model is long as often as it is short (either cross-sectionally or in a time series sense) then this is less likely to be a problem. A far bigger source of error for inexperienced researchers is incorrectly accounting (or not accounting at all) for trading costs, financing costs, roll costs, liquidity constraints, data delays, market impact etc.
jsprogrammer 11y ago
>How do you back test it to know that it works. If you back test over the past 5 years then you are only testing your model against a huge bull market. >If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets, HFT's, huge numbers of hedge funds and the money they bring, and huge passive investing via ETF's, all those factors were there 20 years ago, but they weren't the major market drivers that they are now. I imagine you wouldn't want to 'back test' over some arbitrarily selected number of years of data. You'd want an adaptive algorithm that could exploit 'trends' over any time period. But, like someone else mentioned, backtesting is near meaningless: the only thing that matters is actual performance.
- tomp 11y ago
  > You'd want an adaptive algorithm that could exploit 'trends' over any time period. Of course that what you want, but how will you find it, and more importantly, how will you know you found it? In a bull market, an "adaptable general algorithm" that makes 10% in a year is statistically indistinguishable from an algorithm that buys and holds S&P500, unless you design your backtests very carefully.
  - jsprogrammer 11y ago
    Obviously it is a problem, but it will get you much further than the original suggestion to select some arbitrary number of days to go back.
baddox 11y ago
You could say the same thing about starting a business, or choosing a college degree, or taking a certain job.
namecast 11y ago
> If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets +1, and I'll add a link that argues that data pre-2009 is worthless for a great many (intraday) strategies: http://www.priceactionlab.com/Blog/2015/07/historical-data-trading-system-development/ http://www.priceactionlab.com/Blog/2015/07/historical-data-t... And this is before we even start worrying about in-sample vs. out-of-sample...!
chejazi 11y ago
One problem I encountered when backtesting a futures algorithm was fill rate. In real world scenarios, getting your orders filled can be a slippery slope. Latency can be a huge bust, especially as the market price accelerates away. Accounting for this is more important with HFT, where a position will only be held for a few pips.
trhway 11y ago
>If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets, HFT's, huge numbers of hedge funds and the money they bring, and huge passive investing via ETF's, all those factors were there 20 years ago, but they weren't the major market drivers that they are now. well, giving the fractal nature of price movement, i'd guess one can backtest against last several days/weeks taken at very high resolution and scaled out to represent months/years :)
vegabook 11y ago
> So if you are writing your own trading models you should start by answering the question of what's your alpha when compared to everyone else.... This has been true of markets since well before computers got involved. When thousands of atomic participants are involved, what's your edge? The truth though is that participants "cluster" and "herd", and algos do too. Witness the correlation between all the algo hedge funds. Markets are prone to group psychology. Savvy, often contrarian-minded (or at the least "independent" minded) individuals always did, and still do, have a chance, and that includes algorithmic originality.
gnaritas 11y ago
Back testing can't tell you something works, you're merely curve fitting no matter how careful you're trying to be; the role of back testing is making sure your algo trades when you expect it to. You need to forward test to see if it really works.

CPLX 11y ago

> How do you back test it to know that it works. You don't. You turn it on and see if it works. Which does have a certain degree of intellectual rigor to it. > So if you are writing your own trading models you should start by answering the question of what's your alpha when compared to everyone else.... Well right, that's the core question isn't it. But clearly the allure of wanting to test ones own set of biases and insights for alpha is fairly understandable and appealing.

ericjang 11y ago

One clarification - these mom & pop prop shops aren't necessarily competing against big baskets of MIT PhDs. If your book size is only $200K, the big players (i.e. scores of MIT PhDs) won't even bother competing with you on the same strategies. Alpha from these strategies may very well be orthogonal to hedge fund alpha. That said, you're still playing a zero sum game with other mom & pop shops and the general large-scale movements of the market. You need to be confident that you have better information / forecasting ability than the others, or technology infrastructure that enables strategies no one else can execute. Source: I work in the industry.

deleted 11y ago
[deleted]
exelius 11y ago
Yeah, most people don't realize that finance is a zero-sum game. That leads to arms races which over time remove the lion's share of the profit (companies will spend money on a better solution to a problem until such time as a better solution costs more than the value of the opportunity). I expect this to replicate itself on the low-end as well.
- joosters 11y ago
  The 'zero-sum game' part is meaningless and adds no insight here. It doesn't matter whether you believe trading/gambling is zero-sum or not. For instance, imagine you have a poker-playing bot at a poker table, where the house takes no cut of the stakes. Zero-sum? Yes. But your bot has to be better than the other players, and win more than its running costs in order to profit. Now imagine the same situation but the house does take a cut. Zero-sum? Not any more. But your bot still has to be better than the other players and its running costs in order to profit. The house 'rake' just increases your operating costs. I think you are confusing the issue with large scale HFT (where companies pay more and more for slightly faster comms), which is a world away from the trading strategies the article is talking about.
  - SonicSoul 11y ago
    Think you missed the point. All he was saying (I believe) that no matter how good your models are it's still a race between the players to get the biggest share of the pot. The more players the less chance you have
    - joosters 11y ago
      That's true of many endeavors, and is unrelated to any 'zero-sum' argument.
  - jsprogrammer 11y ago

kittenfluff 11y ago

> If you back test over the past 5 years then you are only testing your model against a huge bull market. If your model is long as often as it is short (either cross-sectionally or in a time series sense) then this is less likely to be a problem. A far bigger source of error for inexperienced researchers is incorrectly accounting (or not accounting at all) for trading costs, financing costs, roll costs, liquidity constraints, data delays, market impact etc.

jsprogrammer 11y ago

>How do you back test it to know that it works. If you back test over the past 5 years then you are only testing your model against a huge bull market. >If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets, HFT's, huge numbers of hedge funds and the money they bring, and huge passive investing via ETF's, all those factors were there 20 years ago, but they weren't the major market drivers that they are now. I imagine you wouldn't want to 'back test' over some arbitrarily selected number of years of data. You'd want an adaptive algorithm that could exploit 'trends' over any time period. But, like someone else mentioned, backtesting is near meaningless: the only thing that matters is actual performance.

tomp 11y ago
> You'd want an adaptive algorithm that could exploit 'trends' over any time period. Of course that what you want, but how will you find it, and more importantly, how will you know you found it? In a bull market, an "adaptable general algorithm" that makes 10% in a year is statistically indistinguishable from an algorithm that buys and holds S&P500, unless you design your backtests very carefully.
- jsprogrammer 11y ago
  Obviously it is a problem, but it will get you much further than the original suggestion to select some arbitrary number of days to go back.

baddox 11y ago

You could say the same thing about starting a business, or choosing a college degree, or taking a certain job.

namecast 11y ago

> If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets +1, and I'll add a link that argues that data pre-2009 is worthless for a great many (intraday) strategies: http://www.priceactionlab.com/Blog/2015/07/historical-data-trading-system-development/ http://www.priceactionlab.com/Blog/2015/07/historical-data-t... And this is before we even start worrying about in-sample vs. out-of-sample...!

chejazi 11y ago

One problem I encountered when backtesting a futures algorithm was fill rate. In real world scenarios, getting your orders filled can be a slippery slope. Latency can be a huge bust, especially as the market price accelerates away. Accounting for this is more important with HFT, where a position will only be held for a few pips.

trhway 11y ago

>If you back test over the past 20 years then I'm not sure it helps much as the market of 20 years ago didn't really have any of the major market drives of today's markets, HFT's, huge numbers of hedge funds and the money they bring, and huge passive investing via ETF's, all those factors were there 20 years ago, but they weren't the major market drivers that they are now. well, giving the fractal nature of price movement, i'd guess one can backtest against last several days/weeks taken at very high resolution and scaled out to represent months/years :)

vegabook 11y ago

> So if you are writing your own trading models you should start by answering the question of what's your alpha when compared to everyone else.... This has been true of markets since well before computers got involved. When thousands of atomic participants are involved, what's your edge? The truth though is that participants "cluster" and "herd", and algos do too. Witness the correlation between all the algo hedge funds. Markets are prone to group psychology. Savvy, often contrarian-minded (or at the least "independent" minded) individuals always did, and still do, have a chance, and that includes algorithmic originality.

gnaritas 11y ago

Back testing can't tell you something works, you're merely curve fitting no matter how careful you're trying to be; the role of back testing is making sure your algo trades when you expect it to. You need to forward test to see if it really works.