8 ms·

> In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Well, that is mostly true. But never discount

by smdz 6y ago

> In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Well, that is mostly true. But never discount anything. There are people like me who used to love the data analysis and prediction part in these markets. I got hooked to the markets because of it. I was not interested in making money and naively thought my average pay was good enough. When I first built (or my machine built) a working strategy (in early 2008), live traded/tested it for a couple of months and told few colleagues about the details about the strategy - they did not take me seriously. This was even before I understood NNs or any of scikit-learn tooling. I knew I wanted to get into financial markets - went to a broker to sell the automated strategy and seeking a full time job as an algo-trader - they thought I was trying to scam them even after seeing the contract notes. Plus algotrading had not picked up back then. I found later about such scams. It took me 3 more years and a financial crisis to understand the value of making "much more than enough" money. And retrospectively I know those were just stupid attempts trying to convince others and attempting to give it away.

traK6Dcm 6y ago
You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn. You're right that there are probably some gems and people writing up good posts and articles. However, 99% of what comes to my inbox, which is certain newsletters and arXiv subscriptions, is clearly BS. I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. In trading the opposite seems to be true. Seeing a good trading paper on arXiv is incredibly rare. I would even go as far as saying that reddit is a significantly better source of information than arXiv for this field.
- p1esk 6y ago
  So how should we evaluate the quality of a paper on trading AI? I mean the authors might not have access to real data, but their ideas might still be good.
  - lumost 6y ago
    There are some ML problems where it is fundamentally impossible to use historical data to make accurate forward looking predictions as its not IID. These fields require you very carefully capture data on sub-optimal choices. In the case of trading this means making explicitly bad trading decisions some portion of the time, and teams that have done this at any scale are unlikely to share the data. In the case of trading, any paper not tackling these issues head on is not likely to be useful.
    - p1esk 6y ago
      I don't get it - if you have accurate historical data, how is this different from having access to current real-time data? Why can't you pretend you live 20 years in the past and use the data you have as if it were real-time?
      - apoorvkumar 6y ago
        For 1 because your trading existence in that universe would change the future which you can't account for. Your activity influences decisions of other HFTs in real time whereas with a static history you're claiming to be able to trade without perturbing the markets.
      - traK6Dcm 6y ago
        Data distribution shift. The market changes over time and your current data does not come from the same distribution as old data. That limits the amount of data you can use for training and testing. You need to be very careful not to overfit. That's especially true for something like daily or hourly data - there isn't much data to begin with and you won't have much left if you look at only a few weeks or months. Market data already has a low signal/noise ratio to begin with, so you need a good chunk of data to learn from. As you go to shorter time scales you get more usable data, but then you also need to deal with other issues such as latencies/jitter, market impact, complex order types, order book queues, etc. It becomes a different game.
- scottlocklin 6y ago
  >I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. You should really google up something called the Gell-Mann amnesia effect. 99.9% of everything is shit. Including biology, CS, ML and especially "AI." Of course trading papers are even more universally shit, but once in a while someone publishes a non obvious to me risk factor.
  - machinehermit 6y ago
    Thanks for the Gell-Mann amnesia effect.
- smabie 6y ago
  Have you tried quantpedia? https://quantpedia.com/ https://quantpedia.com/ It's expensive but I find it a really good source for ideas.
godelmachine 6y ago
Would you kindly give me a fair idea as to what’s a good amount of money to be made in this field?
- imcoconut 6y ago
  As with most industries, it depends. But junior people typically make in the 200-500K range. Then as you gain experience, develop your own ideas/strategies and are able to manage risk appropriately, the sky is the limit. The closer you are to managing money that's being invested the more you make. If you can run a 1.5 - 2 Sharpe strategy and never dip below a ~5% drawdown, i.e. probably have substantial positive skew in returns, you can make in the millions or tens of millions at the right fund. Note that as the OP correctly alluded to, this is much more difficult to do live than in a backtest.
  - traK6Dcm 6y ago
    Is 200-500k still true? It used to be, but I think it has decreased significantly over the last decade. I'd say most junior people in this field are making about the same or less than software engineers these days. But like you said, the range here is incredibly wide and largely depends on how well your strategies do and if you have your own desk/fund.
    - derriz 6y ago
      I don’t think it is. I worked for one of the major HFTs and new graduates earned far less than that. In addition the churn rate was high - I’d say most new graduate hires didn’t last more than 2 years and what you learned in those 2 years was often not much use in terms of experience useful for other career paths. Bonus distribution was reverse exponential. Like traditional consultancy partnerships, a small few of the old hands made serious money but those at the “bottom” made ok money but after a few years their FAANG based contemporaries were doing better. Advancing up the ranks was not guaranteed even if you survived the frequent blood lettings.
  - ganzuul 6y ago
    I wonder if this isn't the cause of lack of progress in science. - Why create wealth for all when you can acquire currency for yourself by managing other people's money? It seems to me like those tragic stories of genuises who died young. What could have been if their ideas had reached the world? But instead of dying the geniuses got sequestered into finance and secrecy, volunteering to make no mark at all on the world of their passing.

by smdz 6y ago

traK6Dcm 6y ago

You make a good point. I've also gotten into trading because I enjoy the algorithmic and mathematical aspects, and I would love to share more of what has been working for me and write extensively about it. And there are probably more people like that out there. However, trading has such a bad reputation and uncertain future that I am not sure that's a good career move. I'm torn. You're right that there are probably some gems and people writing up good posts and articles. However, 99% of what comes to my inbox, which is certain newsletters and arXiv subscriptions, is clearly BS. I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. In trading the opposite seems to be true. Seeing a good trading paper on arXiv is incredibly rare. I would even go as far as saying that reddit is a significantly better source of information than arXiv for this field.

p1esk 6y ago
So how should we evaluate the quality of a paper on trading AI? I mean the authors might not have access to real data, but their ideas might still be good.
- lumost 6y ago
  There are some ML problems where it is fundamentally impossible to use historical data to make accurate forward looking predictions as its not IID. These fields require you very carefully capture data on sub-optimal choices. In the case of trading this means making explicitly bad trading decisions some portion of the time, and teams that have done this at any scale are unlikely to share the data. In the case of trading, any paper not tackling these issues head on is not likely to be useful.
  - p1esk 6y ago
    I don't get it - if you have accurate historical data, how is this different from having access to current real-time data? Why can't you pretend you live 20 years in the past and use the data you have as if it were real-time?
    - apoorvkumar 6y ago
      For 1 because your trading existence in that universe would change the future which you can't account for. Your activity influences decisions of other HFTs in real time whereas with a static history you're claiming to be able to trade without perturbing the markets.
    - traK6Dcm 6y ago
      Data distribution shift. The market changes over time and your current data does not come from the same distribution as old data. That limits the amount of data you can use for training and testing. You need to be very careful not to overfit. That's especially true for something like daily or hourly data - there isn't much data to begin with and you won't have much left if you look at only a few weeks or months. Market data already has a low signal/noise ratio to begin with, so you need a good chunk of data to learn from. As you go to shorter time scales you get more usable data, but then you also need to deal with other issues such as latencies/jitter, market impact, complex order types, order book queues, etc. It becomes a different game.
scottlocklin 6y ago
>I'm particularly disappointed with arXiv/academia, because in other fields like biology and CS/ML/AI, published papers tend to be of higher quality than your average blog post. You should really google up something called the Gell-Mann amnesia effect. 99.9% of everything is shit. Including biology, CS, ML and especially "AI." Of course trading papers are even more universally shit, but once in a while someone publishes a non obvious to me risk factor.
- machinehermit 6y ago
  Thanks for the Gell-Mann amnesia effect.
smabie 6y ago
Have you tried quantpedia? https://quantpedia.com/ https://quantpedia.com/ It's expensive but I find it a really good source for ideas.

godelmachine 6y ago

Would you kindly give me a fair idea as to what’s a good amount of money to be made in this field?

imcoconut 6y ago
As with most industries, it depends. But junior people typically make in the 200-500K range. Then as you gain experience, develop your own ideas/strategies and are able to manage risk appropriately, the sky is the limit. The closer you are to managing money that's being invested the more you make. If you can run a 1.5 - 2 Sharpe strategy and never dip below a ~5% drawdown, i.e. probably have substantial positive skew in returns, you can make in the millions or tens of millions at the right fund. Note that as the OP correctly alluded to, this is much more difficult to do live than in a backtest.
- traK6Dcm 6y ago
  Is 200-500k still true? It used to be, but I think it has decreased significantly over the last decade. I'd say most junior people in this field are making about the same or less than software engineers these days. But like you said, the range here is incredibly wide and largely depends on how well your strategies do and if you have your own desk/fund.
  - derriz 6y ago
    I don’t think it is. I worked for one of the major HFTs and new graduates earned far less than that. In addition the churn rate was high - I’d say most new graduate hires didn’t last more than 2 years and what you learned in those 2 years was often not much use in terms of experience useful for other career paths. Bonus distribution was reverse exponential. Like traditional consultancy partnerships, a small few of the old hands made serious money but those at the “bottom” made ok money but after a few years their FAANG based contemporaries were doing better. Advancing up the ranks was not guaranteed even if you survived the frequent blood lettings.
- ganzuul 6y ago
  I wonder if this isn't the cause of lack of progress in science. - Why create wealth for all when you can acquire currency for yourself by managing other people's money? It seems to me like those tragic stories of genuises who died young. What could have been if their ideas had reached the world? But instead of dying the geniuses got sequestered into finance and secrecy, volunteering to make no mark at all on the world of their passing.