7 ms·

FinRL: The first open-source project for financial reinforcement learning

98 points by ghgr 5y ago

Galanwe 5y ago
As someone with 10+ years of exp in the industry, I had no clue what this was actually doing after reading the github page. Had to dive through the examples and videos. The alphas showcased seem weak: 1.5 max gross SR in sample, unresidualized, unconstrained and standalone. That would need to be blended with a hundred un correlated others to have a decent portfolio.
- nuclearnice1 5y ago
  What does unresidualized mean? What do the need to do to the software to “residualize” it? If that’s even the right expression.
  - Galanwe 5y ago
    It means to remove a set of factors from the signal - or the returns of the signal, which would impact its performance or exposure, and are not meant to be captured by the signal. Imagine a "I have no preference" signal which gives a weight of 1/n to each asset of the portfolio - you have 100 stocks in your basket, you give 1% to each. If you would compute its performance, you would see that this portfolio would roughly make 10% average returns over 20 years. That seems good right? Well no it's not, because these 10% don't come from anything meaningful that your signal did. It came because you gained exposure to the market, and that exposure "carried" your signal performance. > What do the need to do to the software to “residualize” it? Well residualizing overall just means that you want to regress your signals on a set of factors that would carry its performance for no justified reason. In practical terms, you could start by doing a linear regression of your signal against such factors (say beta, country, sector) and only keeping the remaining residuals (alpha).
    - artemonster 5y ago
      Can you recommend books on the topic to learn about this kind of stuff?
      - harveynick 5y ago
        https://www.amazon.co.uk/Quantitative-Equity-Portfolio-Management-Applications-ebook/dp/B005H7L85C https://www.amazon.co.uk/Quantitative-Equity-Portfolio-Manag... Caveat: it’s not exactly “easy reading”, and you might want to have the “three blue one brown” YouTube channel on standby.
        artemonster 5y ago
        Watched every video of them, I am okay with math. Thank you very much! Btw, since quant industry is full of secrets - is there any source on some actual money figures - how much firms make, what are the sums that are required to be shuffled, what are the frequencies of trades, profitability figures? Anything to get a slight glimpse into inner workings of this. Unfortunately googling turns out nothing or scams
        l337 5y ago
        Hedge funds are a very diverse group: from small shops managing tens of millions to the large funds managing tens of billions. The very good ones probably make around 20% a year (meaning that they earn around $40M per year per $1B that’s under management). HFT companies are a different story. The biggest ones make around $1B/year (e.g Virtu, Flow Traders which are public), with a few of them even bigger. There are many tier two companies making tens - few hundred million dollars per year. So none of them are really big (compared, to say, a big bank), but they do make a lot of money per employee, and the salaries reflect that.
        artemonster 5y ago
        Thank you!
        Galanwe 5y ago
  - caffeine 5y ago
    It comes from the “residual” of a linear regression. So you take some obvious variable like the market basket of stocks, do a regression, and then do your machine learning on the residuals. The reason is that you don’t get “credit” for doing whatever the market did, you only get credit for doing even better.
- igorkraw 5y ago
  What's the best way for someone doing and ML and optimisation PhD to get into quant research/Trading? I've been playing with paper trading, trying to use creative data sources to find alpha and algotrading micro amounts of Cryptos for a while now and am seriously considering doing this for my post PhD life if I can't find a nice tenure track (a complication is that I do not want to leave Switzerland until I have citizenship here, so any advice or introductions would be very appreciated!)
  - melony 5y ago
    Get an internship at a reputable prop shop or hedge fund (that specialize in quant finance). There are plenty in Switzerland.
    - bravura 5y ago
      Aren't they going to impose a strong non-compete agreement on you, so that once you've actually learned something you are handcuffed to them?
      - auntienomen 5y ago
        You can't really impose strong non competes on interns. You don't show them the good stuff either. Just enough to get/give an impression for later recruiting.
      - Galanwe 5y ago
        As mentioned in comments, it's pretty rare to give a strong non compete to interns. But interns aside, the whole industry is built around these competes, so mechanism are in place to circumvent them. Some HF will happily buyback the non compete, some will offer a welcome bonus to compensate, some will be ready to wait, some will make an offside deal with the departing company, etc.
    - igorkraw 5y ago
      Do you have a list/advice to find them? I only found a few with offices in Switzerland, and not being in the industry I don't fully trust my judgement of what is "reputable"
- torbTurret 5y ago
  Quants: So much fancy vocabulary just to consistently underperform your favorite mutual index fund :)
  - Galanwe 5y ago
    Do you really believe the world is that simple? It poses you no problem to live in a world where your outsider opinion on a topic you probably just read about on the internet, is right versus a whole industry of very smart people managing hundreds of billions? At no point do you wonder that if thousands of smart, successful people and companies do something, then maybe that something is not as absurd as some internet blog posts may suggest?
    - torbTurret 5y ago
      I have done research in financial modeling and indeed have a family-member that manages his own successful hedge fund, so the comment was somewhat tongue-in-cheek. I do consider myself an outsider, though, as I’ve been out of the field for sometime. Still, if you’ve worked in hedge funds you’ll know two things: connections and luck account for almost everything. What remains are mostly linear models that almost never outperform your basic mutual index funds. Those very smart people moving billions do get things very wrong, though often at an imbalanced risk to themselves (an imbalance in their favor, mind you), so I don’t put much stock in such sweeping appraisals.
jasfi 5y ago
I might be able to use these algorithms in my automated trading system: https://tradecast.one https://tradecast.one. I'm not sure how well they perform, but something to consider.
ghgr 5y ago
I'm curious how efficient is RL for trading. Given that RL is extremely data-hungry I'd say that this approach is likely to result in an overfit model. Does anybody have positive experiences with this?
- deleted 5y ago
  [deleted]
nautilus12 5y ago
This is self defeating if you open source an approach like this and it is effective, then the laws of arbitrage says that the market will remove any additional benefits of using it.
- hungryforcodes 5y ago
  It would have to be the DOMINANT approach for that to happen.
  - deleted 5y ago
    [deleted]
- vageli 5y ago
  > This is self defeating if you open source an approach like this and it is effective, then the laws of arbitrage says that the market will remove any additional benefits of using it. This assumes perfect information. The effects of law of arbitrage are not instantaneous.
- deleted 5y ago
  [deleted]
mumblemumble 5y ago
Quick hot take: when the project has a Venmo QR code for soliciting donations, you can safely assume they have not actually figured out how to make lots of money on the securities markets. Which is not me looking to trash it. I'd just encourage people to be aware of the context. This is an academic project which has apparently only been tested on hindcast data. Real life trading may present challenges that were well outside their scope of research.
- nightski 5y ago
  I'd agree with the skepticism. That said, it takes a minimum of 25k to get started using something like this in the real world due to the PDT rule (honestly more than that so you don't dip below 25k). It's possible they just aren't ready to throw that kind of capital at it yet. Also even if they aren't using real money that does not mean they are limited to historical data. You can run algos on live data even without actively trading. What you don't get is how your actions affect the market itself.
  - pid-1 5y ago
    > Also even if they aren't using real money that does not mean they are limited to historical data. You can run algos on live data even without actively trading. What you don't get is how your actions affect the market itself. That equals to useless toy software in my book. But it's just my opinion. Edit: Look, I don't want to trash the devs for no reason, but the amount of "finance" academic stuff that claims to produce successful strategies is just ridiculous. At some point people just get tired.
    - nightski 5y ago
      You may well be right that it's a toy (and judging from their docs it seems like it), but it's basically the first step in using an algo for trading. You always test on live market feeds first.
  - deleted 5y ago
    [deleted]
Lapsa 5y ago
```Technical indicators: 'macd', 'boll_ub', 'boll_lb', 'rsi_30', 'dx_30', 'close_30_sma', 'close_60_sma'.``` lulz
- yeahwhatever10 5y ago
  Why lulz? Bad indicators, too few?
  - ploika 5y ago
    Technical analysis (Bollinger bands and the like) would be treated similarly to horoscopes by many people who know how to work a Bloomberg terminal.
    - saeranv 5y ago
      I've never understood how someone had the nerve to trademark "Bollinger bands", when all they are is just a measure of 2 standard deviations from a rolling mean of share price. It's like renaming 1+1=2 after yourself and and trademarking it's use.
    - mellavora 5y ago
      So then why does Bloomberg build in so many technical indicators into its product? Look, I share your skepticism of technical analysis, but there are many traders who do not. And when managing traders, you might want to let them trade on TA because it can give them a reason to take action. Otherwise they can suffer from analysis paralysis.
  - Galanwe 5y ago
    These are naive and unrefined technical indicators. - there's only price/volume, it's very poor in terms of information. Smells like an academic with only access to a free database. - it's really the classical technical analysis 101 indicators, studied 10000 times in research articles. - you would need wayyyy more indicators to get a meaningful model.
t_mann 5y ago
I really can't help but wonder - say I want to create a trading strategy that I actually want to use. Is there really any point in learning an open source framework, where I can assume that the most profitable strategies it can produce are already being used by someone? I think I'd rather start by learning the underlying frameworks like TensorFlow, that this one seems to wrap, and produce something that has at least a minimal chance of being somewhat original. That being said, I have more fundamental doubts that as a private individual, with only off-the-shelf hardware, one could produce any meaningful signals with DL that actually beat the market.
- Galanwe 5y ago
  > Is there really any point in learning an open source framework, where I can assume that the most profitable strategies it can produce are already being used by someone? In practice, yes, because the methodology itself is only as good as the data you feed it. There's only so much you can do with e.g. ML. The framework gives you that methodology. You can fiddle with it, fine tune it, etc, but that's often just marginal gain. The real juice is in the features that you prepared, and the data you have available for these features.
  - t_mann 5y ago
    Hmm, but seen under this angle, it seems even worse, no? The tool lists about a dozen well-known public data sources that it's compatible to work with, you can't expect to gain any edge from those. Sure, you could learn how to fit your proprietary data into this tool, but that goes back to my point, why not learn how to work with TensorFlow or whatever ML framework directly? They're not that difficult that you'd need a wrapper.
Stampo00 5y ago
Based on the name, I was hoping it was a roguelike.
ausbah 5y ago
smells like a solution in search of a problem

98 points by ghgr 5y ago

As someone with 10+ years of exp in the industry, I had no clue what this was actually doing after reading the github page. Had to dive through the examples and videos. The alphas showcased seem weak: 1.5 max gross SR in sample, unresidualized, unconstrained and standalone. That would need to be blended with a hundred un correlated others to have a decent portfolio.

nuclearnice1 5y ago
What does unresidualized mean? What do the need to do to the software to “residualize” it? If that’s even the right expression.
- Galanwe 5y ago
  It means to remove a set of factors from the signal - or the returns of the signal, which would impact its performance or exposure, and are not meant to be captured by the signal. Imagine a "I have no preference" signal which gives a weight of 1/n to each asset of the portfolio - you have 100 stocks in your basket, you give 1% to each. If you would compute its performance, you would see that this portfolio would roughly make 10% average returns over 20 years. That seems good right? Well no it's not, because these 10% don't come from anything meaningful that your signal did. It came because you gained exposure to the market, and that exposure "carried" your signal performance. > What do the need to do to the software to “residualize” it? Well residualizing overall just means that you want to regress your signals on a set of factors that would carry its performance for no justified reason. In practical terms, you could start by doing a linear regression of your signal against such factors (say beta, country, sector) and only keeping the remaining residuals (alpha).
  - artemonster 5y ago
    Can you recommend books on the topic to learn about this kind of stuff?
    - harveynick 5y ago
      https://www.amazon.co.uk/Quantitative-Equity-Portfolio-Management-Applications-ebook/dp/B005H7L85C https://www.amazon.co.uk/Quantitative-Equity-Portfolio-Manag... Caveat: it’s not exactly “easy reading”, and you might want to have the “three blue one brown” YouTube channel on standby.
      - artemonster 5y ago
        Watched every video of them, I am okay with math. Thank you very much! Btw, since quant industry is full of secrets - is there any source on some actual money figures - how much firms make, what are the sums that are required to be shuffled, what are the frequencies of trades, profitability figures? Anything to get a slight glimpse into inner workings of this. Unfortunately googling turns out nothing or scams
        l337 5y ago
        Hedge funds are a very diverse group: from small shops managing tens of millions to the large funds managing tens of billions. The very good ones probably make around 20% a year (meaning that they earn around $40M per year per $1B that’s under management). HFT companies are a different story. The biggest ones make around $1B/year (e.g Virtu, Flow Traders which are public), with a few of them even bigger. There are many tier two companies making tens - few hundred million dollars per year. So none of them are really big (compared, to say, a big bank), but they do make a lot of money per employee, and the salaries reflect that.
        artemonster 5y ago
        Thank you!
        Galanwe 5y ago
- caffeine 5y ago
  It comes from the “residual” of a linear regression. So you take some obvious variable like the market basket of stocks, do a regression, and then do your machine learning on the residuals. The reason is that you don’t get “credit” for doing whatever the market did, you only get credit for doing even better.
igorkraw 5y ago
What's the best way for someone doing and ML and optimisation PhD to get into quant research/Trading? I've been playing with paper trading, trying to use creative data sources to find alpha and algotrading micro amounts of Cryptos for a while now and am seriously considering doing this for my post PhD life if I can't find a nice tenure track (a complication is that I do not want to leave Switzerland until I have citizenship here, so any advice or introductions would be very appreciated!)
- melony 5y ago
  Get an internship at a reputable prop shop or hedge fund (that specialize in quant finance). There are plenty in Switzerland.
  - bravura 5y ago
    Aren't they going to impose a strong non-compete agreement on you, so that once you've actually learned something you are handcuffed to them?
    - auntienomen 5y ago
      You can't really impose strong non competes on interns. You don't show them the good stuff either. Just enough to get/give an impression for later recruiting.
    - Galanwe 5y ago
      As mentioned in comments, it's pretty rare to give a strong non compete to interns. But interns aside, the whole industry is built around these competes, so mechanism are in place to circumvent them. Some HF will happily buyback the non compete, some will offer a welcome bonus to compensate, some will be ready to wait, some will make an offside deal with the departing company, etc.
  - igorkraw 5y ago
    Do you have a list/advice to find them? I only found a few with offices in Switzerland, and not being in the industry I don't fully trust my judgement of what is "reputable"
torbTurret 5y ago
Quants: So much fancy vocabulary just to consistently underperform your favorite mutual index fund :)
- Galanwe 5y ago
  Do you really believe the world is that simple? It poses you no problem to live in a world where your outsider opinion on a topic you probably just read about on the internet, is right versus a whole industry of very smart people managing hundreds of billions? At no point do you wonder that if thousands of smart, successful people and companies do something, then maybe that something is not as absurd as some internet blog posts may suggest?
  - torbTurret 5y ago
    I have done research in financial modeling and indeed have a family-member that manages his own successful hedge fund, so the comment was somewhat tongue-in-cheek. I do consider myself an outsider, though, as I’ve been out of the field for sometime. Still, if you’ve worked in hedge funds you’ll know two things: connections and luck account for almost everything. What remains are mostly linear models that almost never outperform your basic mutual index funds. Those very smart people moving billions do get things very wrong, though often at an imbalanced risk to themselves (an imbalance in their favor, mind you), so I don’t put much stock in such sweeping appraisals.

jasfi 5y ago

I might be able to use these algorithms in my automated trading system: https://tradecast.one https://tradecast.one. I'm not sure how well they perform, but something to consider.

ghgr 5y ago

I'm curious how efficient is RL for trading. Given that RL is extremely data-hungry I'd say that this approach is likely to result in an overfit model. Does anybody have positive experiences with this?

deleted 5y ago
[deleted]

nautilus12 5y ago

This is self defeating if you open source an approach like this and it is effective, then the laws of arbitrage says that the market will remove any additional benefits of using it.

hungryforcodes 5y ago
It would have to be the DOMINANT approach for that to happen.
- deleted 5y ago
  [deleted]
vageli 5y ago
> This is self defeating if you open source an approach like this and it is effective, then the laws of arbitrage says that the market will remove any additional benefits of using it. This assumes perfect information. The effects of law of arbitrage are not instantaneous.
deleted 5y ago
[deleted]

mumblemumble 5y ago

Quick hot take: when the project has a Venmo QR code for soliciting donations, you can safely assume they have not actually figured out how to make lots of money on the securities markets. Which is not me looking to trash it. I'd just encourage people to be aware of the context. This is an academic project which has apparently only been tested on hindcast data. Real life trading may present challenges that were well outside their scope of research.

nightski 5y ago
I'd agree with the skepticism. That said, it takes a minimum of 25k to get started using something like this in the real world due to the PDT rule (honestly more than that so you don't dip below 25k). It's possible they just aren't ready to throw that kind of capital at it yet. Also even if they aren't using real money that does not mean they are limited to historical data. You can run algos on live data even without actively trading. What you don't get is how your actions affect the market itself.
- pid-1 5y ago
  > Also even if they aren't using real money that does not mean they are limited to historical data. You can run algos on live data even without actively trading. What you don't get is how your actions affect the market itself. That equals to useless toy software in my book. But it's just my opinion. Edit: Look, I don't want to trash the devs for no reason, but the amount of "finance" academic stuff that claims to produce successful strategies is just ridiculous. At some point people just get tired.
  - nightski 5y ago
    You may well be right that it's a toy (and judging from their docs it seems like it), but it's basically the first step in using an algo for trading. You always test on live market feeds first.
- deleted 5y ago
  [deleted]

Lapsa 5y ago

```Technical indicators: 'macd', 'boll_ub', 'boll_lb', 'rsi_30', 'dx_30', 'close_30_sma', 'close_60_sma'.``` lulz

yeahwhatever10 5y ago
Why lulz? Bad indicators, too few?
- ploika 5y ago
  Technical analysis (Bollinger bands and the like) would be treated similarly to horoscopes by many people who know how to work a Bloomberg terminal.
  - saeranv 5y ago
    I've never understood how someone had the nerve to trademark "Bollinger bands", when all they are is just a measure of 2 standard deviations from a rolling mean of share price. It's like renaming 1+1=2 after yourself and and trademarking it's use.
  - mellavora 5y ago
    So then why does Bloomberg build in so many technical indicators into its product? Look, I share your skepticism of technical analysis, but there are many traders who do not. And when managing traders, you might want to let them trade on TA because it can give them a reason to take action. Otherwise they can suffer from analysis paralysis.
- Galanwe 5y ago
  These are naive and unrefined technical indicators. - there's only price/volume, it's very poor in terms of information. Smells like an academic with only access to a free database. - it's really the classical technical analysis 101 indicators, studied 10000 times in research articles. - you would need wayyyy more indicators to get a meaningful model.

t_mann 5y ago

I really can't help but wonder - say I want to create a trading strategy that I actually want to use. Is there really any point in learning an open source framework, where I can assume that the most profitable strategies it can produce are already being used by someone? I think I'd rather start by learning the underlying frameworks like TensorFlow, that this one seems to wrap, and produce something that has at least a minimal chance of being somewhat original. That being said, I have more fundamental doubts that as a private individual, with only off-the-shelf hardware, one could produce any meaningful signals with DL that actually beat the market.

Galanwe 5y ago
> Is there really any point in learning an open source framework, where I can assume that the most profitable strategies it can produce are already being used by someone? In practice, yes, because the methodology itself is only as good as the data you feed it. There's only so much you can do with e.g. ML. The framework gives you that methodology. You can fiddle with it, fine tune it, etc, but that's often just marginal gain. The real juice is in the features that you prepared, and the data you have available for these features.
- t_mann 5y ago
  Hmm, but seen under this angle, it seems even worse, no? The tool lists about a dozen well-known public data sources that it's compatible to work with, you can't expect to gain any edge from those. Sure, you could learn how to fit your proprietary data into this tool, but that goes back to my point, why not learn how to work with TensorFlow or whatever ML framework directly? They're not that difficult that you'd need a wrapper.

Stampo00 5y ago

Based on the name, I was hoping it was a roguelike.

ausbah 5y ago

smells like a solution in search of a problem