3 ms·

Firstly, nice effort. Secondly, some features you could add: 1. Constrained optimization - including budget constraints, sector selection constraints etc. A t

Firstly, nice effort. Secondly, some features you could add: 1. Constrained optimization - including budget constraints, sector selection constraints etc. A tough one would be cardinal constraints e.g. I am limited to 4 stocks etc. 2. Return attribution - whether the returns your portfolio earned were due to stock selection or asset allocation or both (Brinson Model: http://www.mscibarra.com/research/articles/2002/PerfBrinson.pdf http://www.mscibarra.com/research/articles/2002/PerfBrinson....). 3. Performance and compression - how would this deal with huge covariance matrices? 10000 x 10000? Matrix operarions on these wouldn't be trivial. In-memory serialization/deserialization issues also come to mind. (edit: then again, Excel can't do 10k x 10k :) ) 4. I'm not conversant with SciPy - does this use BFGS/similar for optimization? 5. Compute as a service? Host a grid? Let calculation requests come to you via Excel? (Nobody would want a 10000 asset timeseries to be processed on their CPU for two hours).

wtvanhest 14y ago
I second everything photon has said and would add that anyone using this model should acknowledge the risk they are taking by using past stock correlations to predict future optimal portfolios. While this approach is central to finance, recently (past 18 months) we have seen bizarrely correlated asset price changes due to macro issues (especially in Europe). These macro issues may change the way stocks correlate in future markets. I would back test any portfolio with data from at least 18 months ago to make sure it still looks pretty good.
- bravura 14y ago
  I'm interested in talking about macro variable prediction. [edit: A colleague of mine has a model that predicts macro variables, and wants beta users.] Could you email me? (Email in profile.)
- wiredfool 14y ago
  I'd strongly consider not using 25% of the data for the correlations. Then backcheck the generated model based on that last 25%. I'd be really tempted to make sure that the 25% included things like the Global Financial Cluster..ck, just to see how well things go when something unexpected happens. Either that or run separate correlations for each $time_period and see how the results change over time. If they're stable, then you've got better confidence than if they're wildly changing at each interval. Also, realize that people with way more data and processor power have been doing this for a long time, with very mixed results. /me remembers my uncle slaving over the Apple ][e looking at charts and numbers.
- photon137 14y ago
  >we have seen bizarrely correlated asset price changes due to macro issues Actually, this is quite expected given event-risk in the market. For example, even in a non-crisis situation, correlation increases in intra-day trading when market-moving news is due - e.g. payroll numbers are due or ISM numbers are released. Coming back to the present, headline-risk hasn't been as high as it has been since May, 2010 as we keep having one EU summit after another, one central bank announcement after another and each having a huge impact on how asset prices behave. So the correlation is not entirely bizarre - people have been making correlation trades as well - for example JP Morgan's huge loss while betting on credit index tranches was a bet on implied vs. realized correlation (it didn't play out as expected, of course). I'd be interested to know how people are playing this out in the equity space - via variance swaps?
karamazov 14y ago
Thanks for the suggestions! I might implement a couple in a future post. Numpy can handle a 10k by 10k matrix - in this case, you wouldn't want to print it to the spreadsheet, but you could still run the optimization. I've tested to 100 tickers and there isn't a noticeable increase in computation time. (I'm sure there would be at 10,000, but it's probably doable; the constraint here might be fetching that much data.) The optimization is done with a Nelder-Mead simplex algorithm [1], but SciPy has a lot of different functionality built in. There's a different function, fmin_bgfs[2], that does use BFGS optimization. Computations as a service is something we're definitely looking into; if there's enough demand we'll do it. [1] http://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method http://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method [2] http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_bfgs.html#scipy.optimize.fmin_bfgs http://docs.scipy.org/doc/scipy/reference/generated/scipy.op...

wtvanhest 14y ago

I second everything photon has said and would add that anyone using this model should acknowledge the risk they are taking by using past stock correlations to predict future optimal portfolios. While this approach is central to finance, recently (past 18 months) we have seen bizarrely correlated asset price changes due to macro issues (especially in Europe). These macro issues may change the way stocks correlate in future markets. I would back test any portfolio with data from at least 18 months ago to make sure it still looks pretty good.

bravura 14y ago
I'm interested in talking about macro variable prediction. [edit: A colleague of mine has a model that predicts macro variables, and wants beta users.] Could you email me? (Email in profile.)
wiredfool 14y ago
I'd strongly consider not using 25% of the data for the correlations. Then backcheck the generated model based on that last 25%. I'd be really tempted to make sure that the 25% included things like the Global Financial Cluster..ck, just to see how well things go when something unexpected happens. Either that or run separate correlations for each $time_period and see how the results change over time. If they're stable, then you've got better confidence than if they're wildly changing at each interval. Also, realize that people with way more data and processor power have been doing this for a long time, with very mixed results. /me remembers my uncle slaving over the Apple ][e looking at charts and numbers.
photon137 14y ago
>we have seen bizarrely correlated asset price changes due to macro issues Actually, this is quite expected given event-risk in the market. For example, even in a non-crisis situation, correlation increases in intra-day trading when market-moving news is due - e.g. payroll numbers are due or ISM numbers are released. Coming back to the present, headline-risk hasn't been as high as it has been since May, 2010 as we keep having one EU summit after another, one central bank announcement after another and each having a huge impact on how asset prices behave. So the correlation is not entirely bizarre - people have been making correlation trades as well - for example JP Morgan's huge loss while betting on credit index tranches was a bet on implied vs. realized correlation (it didn't play out as expected, of course). I'd be interested to know how people are playing this out in the equity space - via variance swaps?

karamazov 14y ago

Thanks for the suggestions! I might implement a couple in a future post. Numpy can handle a 10k by 10k matrix - in this case, you wouldn't want to print it to the spreadsheet, but you could still run the optimization. I've tested to 100 tickers and there isn't a noticeable increase in computation time. (I'm sure there would be at 10,000, but it's probably doable; the constraint here might be fetching that much data.) The optimization is done with a Nelder-Mead simplex algorithm [1], but SciPy has a lot of different functionality built in. There's a different function, fmin_bgfs[2], that does use BFGS optimization. Computations as a service is something we're definitely looking into; if there's enough demand we'll do it. [1] http://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method http://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method [2] http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin_bfgs.html#scipy.optimize.fmin_bfgs http://docs.scipy.org/doc/scipy/reference/generated/scipy.op...