8 ms·

I tried doing some forecasting with various neural network models after assembling what I thought was a good amount of forex data. The neural net (I tried vario

I tried doing some forecasting with various neural network models after assembling what I thought was a good amount of forex data. The neural net (I tried various architectures) couldn't do any better than chance. After playing around with it and trying to double-check everything, that was as far as I could get. This puts me ahead of most traders, since most of them lose money, then quit. This makes me wonder what kind of trading systems can actually have any kind of edge, since some kind of autoregressive time series forecasting system seems pretty unreliable. On a more general note, how do you move beyond it being gambling? Just because a system backtests well doesn't mean a phenomenon will continue to happen, especially if your system will significantly impact the market you're in. If you make a trend-following system, every time you trade, you're gambling that the trend is more likely to continue than not. If you're right, you'll come out ahead over many trades. If you don't have enough capital to withstand drawdown the way most beginners don't, you won't be able to last long enough for whatever phenomenon you've found to average out. It takes a lot of time, effort and risk to do all this, so, this is a long-winded way of saying I don't think it's for me. If you build a SaaS product and it fails, at least you can talk about what you learned from building it and use that in future endeavors. If you lose money trading because your algorithm doesn't work, what do you learn from that besides that your algorithm doesn't work?

wavepruner 6y ago
You need more data to input besides just the price time-series. Successful human traders balance and synthesize a myriad of data sources to make decisions. I depend on an in-depth understanding of human psychology as one of my data sources. You can't turn something like that into data and input to a model. It is something learned through life experience and study.
- henning 6y ago
  This is another unhelpful thing about trading: when you seek out information, everyone pipes in telling you that what you're doing is wrong, what they do is right, what you want to do can't be done without giving any explanation or elaboration, etc.
  - _e 6y ago
    If your algorithm stops working or doesn't work... do you have the experience to know why? wavepruner is saying that models can't capture everything like intuition and experience which comes with time.
- akg_67 6y ago
  +1 Any trading strategy based only on price is fool’s errand. The information that impacts price need to be included in the trading strategy. A lot of short term price movement is news driven, thus unstructured text processing of news, social media, relevant documents will be a key component of such trading strategies. I am not that familiar with forex market compared to equity market. But I expect forex to be impacted by changes in political and economic situational news of host countries of relevant currency pairs. All these need to be coded into forex trading strategies. If trading based on just price was so simple , everybody would be doing it successfully.
  - smabie 6y ago
    No successful strategy ever has been based on price. Price isn't stationary so you can't do anything with it. You need to be looking at the log returns. Price is completely irrelevant, at least for equities.
    - akg_67 6y ago
      Do you know how return is calculated?
    - Jimmc414 6y ago
      Sorry, but that is wrong. All I use is price and time. See Elliot Wave Theory. Most indicators are perfectly correlated with price meaning unless you are a HFT you can't trade fast enough to act on them.
      - smabie 6y ago
        Elliot wave theory is quackery. Price is not suitable for any statistical analysis.
        Jimmc414 6y ago
        It has changed and it is just observations of competing waves of pessimism and optimism and the patterns they demonstrate. I'll put a model trained on TQQQ and SQQQ price and time data only. I will bet you 5k mine will beat yours using whatever inputs you choose over any reasonable time you specify.
- tatrajim 6y ago
  Personally over the past few decades investing in US equities, I have found arbitraging information found in Asian-language tech sites and real-world locations such as Shenzhen surprisingly profitable. My best example was a tip I garnered in 2011 from a Beijing KO employee brandishing her new iPhone 4. When I idly asked her if other employees in the huge KO (hundreds of staff) had iPhones, she offhandedly exclaimed 我们都有！ ("we all have them"). That moment set me up for a very comfortable retirement.
darkteflon 6y ago
Most people seem to think indexing is a boring cop-out, but imo it’s the place of humility you get to after you’ve dashed yourself against the rocks of trying to outperform for a few years - or decades - and then realising the whole endeavour is insanity. It’s accepting that you’ll receive what the market gives you and not a dollar more, and that that’s the best you’re ever going to get.
- winrid 6y ago
  +1 for indexes. This is the route most successful traders seem to take - alternating between indexes and bonds. You won't see Warren Buffet buying stocks on Robinhood.
  - smabie 6y ago
    You don't see Warren Buffet buying indices either.. If you aren't willing to put in the work, sure just hold an all weather portfolio or whatever. But I don't think it's super hard to beat the market on a risk-adjusted basis, especially as a retail investor. Most of all, it takes a passion for it (trading is a hobby) and also a reasonable amount of time dedication. For example, there are a number of persistent factors that don't go away. Things like value, low-beta, size factors, etc. It's not super hard to leverage the tons and tons of papers that have been published to construct a portfolio that does really well. And, due to structural reasons, will do well until the laws change (almost all persistent factors exist due to some law). Especially in the current market environment, it's really easy to make a killing. The market is moving purely on sentiment and day traders have had some really great opportunities to make stacks in the past couple of months.
    - darkteflon 6y ago
      Friend, if you are up on Hackernews making comments like this, I can guarantee with near 100% certainty that you are not capable of sustained outperformance in the markets. Whether you realise it now, later or never is no skin off my nose, but sooner would be cheaper. Comments like this are why these "beat-the-market" threads are evergreen on HN. They do a disservice to the community.
smabie 6y ago
The most popular kind of quant trading is using a factor model. The first step is developing some alpha factor, a number that is predictive of how much money you'll make from each stock. So let's say my alpha factor is "companies with good earnings per share will go up." So I first take the EPS for all the stocks in my universe and maybe rank, then zscore. Now I have some positive numbers and some negative numbers. These represent the weights of my portfolio. The positive weighted companies I go long, the negative ones I go short. The bigger the number, the larger my allocations. Now that I have my alpha factor I backtest it and whatever. Since the mean of a zscore is zero, I know I'm market neutral, so (ignoring some stuff) my factor should have little exposure to the market. If I think it's good, I add it to my other alpha factors and combine them somehow. Could be as simple as adding them all up, or maybe something like using random forests to figure out the best way to combine them, or whatever. Now that I have a bunch of alpha factors all combined, I can run them through the optimization engine. The optimization engine will adjust the weights of my "ideal" portfolio in order to reduce exposure to various risk factors (thus lowering volatility). My optimizer will also figure out how often I need to rebalance. There's generally a bunch of terms in there that try to reduce trading costs and zero out exposure while not diluting the "ideal" portfolio too much (or else the alpha could be wiped out). Now, after all of this, I'm ready to trade. In short, what we're trying to do is reduce our exposure to as many factors as possible and just get exposure to our alpha factor. We don't want the market, price of oil, sex scandal of a CEO, or anything else affecting our portfolio. We are trying to dig up this latent, unearthed, alpha that exists in the market, but doesn't belong to one company or asset.
- steephills 6y ago
  Taking EPS as proxy for alpha is like trying to recognise banana pictures by looking at average yellow content: it is plausible, but roughly 100 years behind the other market participants. Don't do this kind of stuff with your hard earned money...
  - DSingularity 6y ago
    He was using that as a simple example.
hogFeast 6y ago
There are lots of ways to produce an edge. Forex is slightly different because you are trading a currency (this actually makes things easier in some ways) but, a few years ago, a lot of the cutting edge was news releases. So inflation comes out at X% and then you try to jump ahead of other people reacting to the news. Speaking very generally, you are looking for data that has information about future returns. So this may include past values of the time series (this is kind of complex though because a stock price does trend, that company is investing capital to earn a return which compounds in the price so stationarity is...complex) but may include other time series/their past values i.e. price of other stocks, economic data, etc. So this could be responding to changes in liquidity, it could be seeing some repeatable behaviour by investors and jumping ahead of it, etc. Quant is not about adding to the efficiency of markets though. They aren't using these models to determine the value of something, they are more about looking at the value of other things to determine the value of a given asset. So these strategies end up being correlated to liquidity in a lot of instances (but not all). This is a generalisation but...it is a very odd thing to have occurring in society...would this exist if investors didn't have an irrational demand for microsecond liquidity? Probably not. Also, determining whether something is a real signal is just part of statistics, isn't it? This has definitely been an area where there has been quite a lot of innovation as increases in computational power has made non-parametric stuff more feasible (I am not an expert on this, it is just my understanding). Btw, I should add I used to work in finance and I have some experience with this kind of thing as I do quite a bit of "quant investing" but in gambling (it is far easier to just copy what people do in finance and apply it to gambling then come up with it yourself). And just based on my experience, it makes most sense to employ a mixed approach. So learn about the business valuation, and then build a five-factor model...watch what it does, and then filter its picks with your knowledge. A lot of quant strategies are vaguely ludicrous if you have an understanding of the fundamentals of investing, like you are trying to use a computer to replicate a human...and people wonder why it doesn't work? It is an overcomplicated shortcut (to give you a concrete example, the blowup of value and funds like AQR was very obvious...you just had to look at the utter garbage stocks they owned). So I think a combination of human and computer beats either separately (one fund that does is Marshall Wace).
gorgoiler 6y ago
In a literal market, you can find alpha in tomato trading by: (1) monitoring other people’s tomato transactions in as much detail as possible; (2) spying on people who are about to buy tomatoes and rapidly make changes behind the scenes just before they make a purchase; or (3) pointing voice analysis at The Food Network looking out for recipes that call for fresh tomatoes, tracking tomato tankers in major sea lanes, monitoring storm tracks in the top tomato growing zones, etc, and adjusting your position appropriately. It sounds like you are trying (1) when (3) might be better, or even (2) if you are not jail-averse (or your local jurisdiction has institutionalized high speed market fiddling to the point of being legal.)
- y7F85OGwp 6y ago
  Mind providing a source for what a 'literal market' means? Thanks.
  - gorgoiler 6y ago
    As in a town square market, as seen in human culture for the last few thousand years. For non-native speakers, market has a specific meaning in US English. In the US a “market” typically means a small shop selling the most common fruits, vegetables, snacks, and household items. In Europe a market is a larger and more general area for autonomous small traders, usually also trading food and household goods. The autonomy and low barrier to entry make them a good analogy to a stock market. (They also suffer from availability and quality issues. A large business can arrange to purchase and retail all of these goods on behalf of consumers under one roof — a supermarket.)
tcgv 6y ago
> since some kind of autoregressive time series forecasting system seems pretty unreliable. A few months ago I tried to evaluate autoregressive behavior in stock returns. To my surprise it seemed strong on some periods, but then weak on others [1], and as you said not reliable enough to rely on. My impression is that a lot more information aggregation and processing is required to obtain a sustainable edge worth tranding on than what a single developer can achieve in his/her spare time. Top investment shops have dedicated teams of sw engineers just to deal with the infrastructure that support their data pipelines, financial model backtesting and deployment. [1] https://thomasvilhena.com/2020/01/likelihood-of-autoregression-in-stock-returns https://thomasvilhena.com/2020/01/likelihood-of-autoregressi...
- Galanwe 6y ago
  You don't have to outperform to have a viable strategy. Harnessing small returns from multiple un correlated sources is a perfectly viable plan.
- michaf 6y ago
  There must be some information content left in stock price time series data, as evidenced e.g. by the price Momentum factor [1], which was been replicated in a number of studies (e.g. [2]), observable over the last couple of decades. [1] https://en.wikipedia.org/wiki/Momentum_investing https://en.wikipedia.org/wiki/Momentum_investing [2] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2961979 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2961979
KKKKkkkk1 6y ago
Let's say I want to predict the crop yield of a field. Sure, looking at the yield in previous years would help. But the yield is just a nonlinear projection of a point in high-dimensional space that has dimensions like weather, water availability, pest infestations, farmer skill, etc. All of these dimensions are incredibly relevant to forecasting, but once we've projected our points onto the yield axis, most of this information is gone. So if you want to take advantage of this information, you need to do your fitting in the original high-dimensional space.
user5994461 6y ago
>>> This makes me wonder what kind of trading systems can actually have any kind of edge. The secret is simply to have an edge. If you're trading on behalf of clients. You don't care what happens to the market because you don't depend on the high or low to make money. If you're buying or selling for yourself, same thing. Guess who's buying coal and oil, power plants and refineries and assimilated. They sell what they have and buy what they need. If you're making money on arbitrage, making sure the New York and the London stock exchange have the same USD to GBP to EUR price and vice versa. You could make money but you better be faster than other corporations and more careful at the same time because you're not the only one doing that. Anytime you buy one side, the other side might have changed because you can balance out. There are clear factors that drive many markets. When the weather is cold people consume more energy for heating. When it's hot, they go out to make barbecues and buy more sausages. When there is a drought or crop sickness, wiping agriculture exploitation, prices of food and meat go up. That's some examples that are easy to understand. The stock market is not about speculation. It's about buying real items in the real world and providing services.
deleted 6y ago
[deleted]
dbs 6y ago
You dont learn. There are two rules, you dont talk about what works and you dont talk about what doesnt work. I worked professionally in the quant space up to 2008, and i still get calls for interviews, with people wanting to dig out what i am doing nowadays. What is popular or known loses alpha pretty quickly due to overcrowding.

wavepruner 6y ago

You need more data to input besides just the price time-series. Successful human traders balance and synthesize a myriad of data sources to make decisions. I depend on an in-depth understanding of human psychology as one of my data sources. You can't turn something like that into data and input to a model. It is something learned through life experience and study.

henning 6y ago
This is another unhelpful thing about trading: when you seek out information, everyone pipes in telling you that what you're doing is wrong, what they do is right, what you want to do can't be done without giving any explanation or elaboration, etc.
- _e 6y ago
  If your algorithm stops working or doesn't work... do you have the experience to know why? wavepruner is saying that models can't capture everything like intuition and experience which comes with time.
akg_67 6y ago
+1 Any trading strategy based only on price is fool’s errand. The information that impacts price need to be included in the trading strategy. A lot of short term price movement is news driven, thus unstructured text processing of news, social media, relevant documents will be a key component of such trading strategies. I am not that familiar with forex market compared to equity market. But I expect forex to be impacted by changes in political and economic situational news of host countries of relevant currency pairs. All these need to be coded into forex trading strategies. If trading based on just price was so simple , everybody would be doing it successfully.
- smabie 6y ago
  No successful strategy ever has been based on price. Price isn't stationary so you can't do anything with it. You need to be looking at the log returns. Price is completely irrelevant, at least for equities.
  - akg_67 6y ago
    Do you know how return is calculated?
  - Jimmc414 6y ago
    Sorry, but that is wrong. All I use is price and time. See Elliot Wave Theory. Most indicators are perfectly correlated with price meaning unless you are a HFT you can't trade fast enough to act on them.
    - smabie 6y ago
      Elliot wave theory is quackery. Price is not suitable for any statistical analysis.
      - Jimmc414 6y ago
        It has changed and it is just observations of competing waves of pessimism and optimism and the patterns they demonstrate. I'll put a model trained on TQQQ and SQQQ price and time data only. I will bet you 5k mine will beat yours using whatever inputs you choose over any reasonable time you specify.
tatrajim 6y ago
Personally over the past few decades investing in US equities, I have found arbitraging information found in Asian-language tech sites and real-world locations such as Shenzhen surprisingly profitable. My best example was a tip I garnered in 2011 from a Beijing KO employee brandishing her new iPhone 4. When I idly asked her if other employees in the huge KO (hundreds of staff) had iPhones, she offhandedly exclaimed 我们都有！ ("we all have them"). That moment set me up for a very comfortable retirement.

darkteflon 6y ago

Most people seem to think indexing is a boring cop-out, but imo it’s the place of humility you get to after you’ve dashed yourself against the rocks of trying to outperform for a few years - or decades - and then realising the whole endeavour is insanity. It’s accepting that you’ll receive what the market gives you and not a dollar more, and that that’s the best you’re ever going to get.

winrid 6y ago
+1 for indexes. This is the route most successful traders seem to take - alternating between indexes and bonds. You won't see Warren Buffet buying stocks on Robinhood.
- smabie 6y ago
  You don't see Warren Buffet buying indices either.. If you aren't willing to put in the work, sure just hold an all weather portfolio or whatever. But I don't think it's super hard to beat the market on a risk-adjusted basis, especially as a retail investor. Most of all, it takes a passion for it (trading is a hobby) and also a reasonable amount of time dedication. For example, there are a number of persistent factors that don't go away. Things like value, low-beta, size factors, etc. It's not super hard to leverage the tons and tons of papers that have been published to construct a portfolio that does really well. And, due to structural reasons, will do well until the laws change (almost all persistent factors exist due to some law). Especially in the current market environment, it's really easy to make a killing. The market is moving purely on sentiment and day traders have had some really great opportunities to make stacks in the past couple of months.
  - darkteflon 6y ago
    Friend, if you are up on Hackernews making comments like this, I can guarantee with near 100% certainty that you are not capable of sustained outperformance in the markets. Whether you realise it now, later or never is no skin off my nose, but sooner would be cheaper. Comments like this are why these "beat-the-market" threads are evergreen on HN. They do a disservice to the community.

smabie 6y ago

The most popular kind of quant trading is using a factor model. The first step is developing some alpha factor, a number that is predictive of how much money you'll make from each stock. So let's say my alpha factor is "companies with good earnings per share will go up." So I first take the EPS for all the stocks in my universe and maybe rank, then zscore. Now I have some positive numbers and some negative numbers. These represent the weights of my portfolio. The positive weighted companies I go long, the negative ones I go short. The bigger the number, the larger my allocations. Now that I have my alpha factor I backtest it and whatever. Since the mean of a zscore is zero, I know I'm market neutral, so (ignoring some stuff) my factor should have little exposure to the market. If I think it's good, I add it to my other alpha factors and combine them somehow. Could be as simple as adding them all up, or maybe something like using random forests to figure out the best way to combine them, or whatever. Now that I have a bunch of alpha factors all combined, I can run them through the optimization engine. The optimization engine will adjust the weights of my "ideal" portfolio in order to reduce exposure to various risk factors (thus lowering volatility). My optimizer will also figure out how often I need to rebalance. There's generally a bunch of terms in there that try to reduce trading costs and zero out exposure while not diluting the "ideal" portfolio too much (or else the alpha could be wiped out). Now, after all of this, I'm ready to trade. In short, what we're trying to do is reduce our exposure to as many factors as possible and just get exposure to our alpha factor. We don't want the market, price of oil, sex scandal of a CEO, or anything else affecting our portfolio. We are trying to dig up this latent, unearthed, alpha that exists in the market, but doesn't belong to one company or asset.

steephills 6y ago
Taking EPS as proxy for alpha is like trying to recognise banana pictures by looking at average yellow content: it is plausible, but roughly 100 years behind the other market participants. Don't do this kind of stuff with your hard earned money...
- DSingularity 6y ago
  He was using that as a simple example.

hogFeast 6y ago

There are lots of ways to produce an edge. Forex is slightly different because you are trading a currency (this actually makes things easier in some ways) but, a few years ago, a lot of the cutting edge was news releases. So inflation comes out at X% and then you try to jump ahead of other people reacting to the news. Speaking very generally, you are looking for data that has information about future returns. So this may include past values of the time series (this is kind of complex though because a stock price does trend, that company is investing capital to earn a return which compounds in the price so stationarity is...complex) but may include other time series/their past values i.e. price of other stocks, economic data, etc. So this could be responding to changes in liquidity, it could be seeing some repeatable behaviour by investors and jumping ahead of it, etc. Quant is not about adding to the efficiency of markets though. They aren't using these models to determine the value of something, they are more about looking at the value of other things to determine the value of a given asset. So these strategies end up being correlated to liquidity in a lot of instances (but not all). This is a generalisation but...it is a very odd thing to have occurring in society...would this exist if investors didn't have an irrational demand for microsecond liquidity? Probably not. Also, determining whether something is a real signal is just part of statistics, isn't it? This has definitely been an area where there has been quite a lot of innovation as increases in computational power has made non-parametric stuff more feasible (I am not an expert on this, it is just my understanding). Btw, I should add I used to work in finance and I have some experience with this kind of thing as I do quite a bit of "quant investing" but in gambling (it is far easier to just copy what people do in finance and apply it to gambling then come up with it yourself). And just based on my experience, it makes most sense to employ a mixed approach. So learn about the business valuation, and then build a five-factor model...watch what it does, and then filter its picks with your knowledge. A lot of quant strategies are vaguely ludicrous if you have an understanding of the fundamentals of investing, like you are trying to use a computer to replicate a human...and people wonder why it doesn't work? It is an overcomplicated shortcut (to give you a concrete example, the blowup of value and funds like AQR was very obvious...you just had to look at the utter garbage stocks they owned). So I think a combination of human and computer beats either separately (one fund that does is Marshall Wace).

gorgoiler 6y ago

In a literal market, you can find alpha in tomato trading by: (1) monitoring other people’s tomato transactions in as much detail as possible; (2) spying on people who are about to buy tomatoes and rapidly make changes behind the scenes just before they make a purchase; or (3) pointing voice analysis at The Food Network looking out for recipes that call for fresh tomatoes, tracking tomato tankers in major sea lanes, monitoring storm tracks in the top tomato growing zones, etc, and adjusting your position appropriately. It sounds like you are trying (1) when (3) might be better, or even (2) if you are not jail-averse (or your local jurisdiction has institutionalized high speed market fiddling to the point of being legal.)

y7F85OGwp 6y ago
Mind providing a source for what a 'literal market' means? Thanks.
- gorgoiler 6y ago
  As in a town square market, as seen in human culture for the last few thousand years. For non-native speakers, market has a specific meaning in US English. In the US a “market” typically means a small shop selling the most common fruits, vegetables, snacks, and household items. In Europe a market is a larger and more general area for autonomous small traders, usually also trading food and household goods. The autonomy and low barrier to entry make them a good analogy to a stock market. (They also suffer from availability and quality issues. A large business can arrange to purchase and retail all of these goods on behalf of consumers under one roof — a supermarket.)

tcgv 6y ago

> since some kind of autoregressive time series forecasting system seems pretty unreliable. A few months ago I tried to evaluate autoregressive behavior in stock returns. To my surprise it seemed strong on some periods, but then weak on others [1], and as you said not reliable enough to rely on. My impression is that a lot more information aggregation and processing is required to obtain a sustainable edge worth tranding on than what a single developer can achieve in his/her spare time. Top investment shops have dedicated teams of sw engineers just to deal with the infrastructure that support their data pipelines, financial model backtesting and deployment. [1] https://thomasvilhena.com/2020/01/likelihood-of-autoregression-in-stock-returns https://thomasvilhena.com/2020/01/likelihood-of-autoregressi...

Galanwe 6y ago
You don't have to outperform to have a viable strategy. Harnessing small returns from multiple un correlated sources is a perfectly viable plan.
michaf 6y ago
There must be some information content left in stock price time series data, as evidenced e.g. by the price Momentum factor [1], which was been replicated in a number of studies (e.g. [2]), observable over the last couple of decades. [1] https://en.wikipedia.org/wiki/Momentum_investing https://en.wikipedia.org/wiki/Momentum_investing [2] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2961979 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2961979

KKKKkkkk1 6y ago

Let's say I want to predict the crop yield of a field. Sure, looking at the yield in previous years would help. But the yield is just a nonlinear projection of a point in high-dimensional space that has dimensions like weather, water availability, pest infestations, farmer skill, etc. All of these dimensions are incredibly relevant to forecasting, but once we've projected our points onto the yield axis, most of this information is gone. So if you want to take advantage of this information, you need to do your fitting in the original high-dimensional space.

user5994461 6y ago

>>> This makes me wonder what kind of trading systems can actually have any kind of edge. The secret is simply to have an edge. If you're trading on behalf of clients. You don't care what happens to the market because you don't depend on the high or low to make money. If you're buying or selling for yourself, same thing. Guess who's buying coal and oil, power plants and refineries and assimilated. They sell what they have and buy what they need. If you're making money on arbitrage, making sure the New York and the London stock exchange have the same USD to GBP to EUR price and vice versa. You could make money but you better be faster than other corporations and more careful at the same time because you're not the only one doing that. Anytime you buy one side, the other side might have changed because you can balance out. There are clear factors that drive many markets. When the weather is cold people consume more energy for heating. When it's hot, they go out to make barbecues and buy more sausages. When there is a drought or crop sickness, wiping agriculture exploitation, prices of food and meat go up. That's some examples that are easy to understand. The stock market is not about speculation. It's about buying real items in the real world and providing services.

deleted 6y ago

[deleted]

dbs 6y ago

You dont learn. There are two rules, you dont talk about what works and you dont talk about what doesnt work. I worked professionally in the quant space up to 2008, and i still get calls for interviews, with people wanting to dig out what i am doing nowadays. What is popular or known loses alpha pretty quickly due to overcrowding.