4 ms·

From what I can see, that doesn't look too good. There are likely issues with the data, which is not easily solvable given that complete databases are expensive

From what I can see, that doesn't look too good. There are likely issues with the data, which is not easily solvable given that complete databases are expensive. And there are likely issues with the modelling, it looks like the model is just scenario analysis. This is better than a misspecified parametric model (most advisers don't do scenario analysis either, so it is definitely valuable...although requires market knowledge to parse fully) but relies on good data...the data is not good. The correct way to do it is a Monte Carlo simulation using either: a good parametric model (this is very tricky, something like a T distribution is least worst...but is still bad, the ideal is some kind of regime model, proper volatility modelling, and something that models expectation accurately between regimes...tricky), or using block bootstrap on good historical data (more robust quantitatively...but again, good long-term data costs $100k+). So...it is straightforward if you have good data and understand how to build a proper event-based backtesting system (or code something good enough)...but, as said, 99.99% of advisers do not have the resources for this. In most cases, they are unable to buy software implementing this or even know they need it (I have seen this software but it is usually at advisers that require liquid wealth of $10m+...it isn't only the cost of the software but most advisers below this level have no quantitative skills/staff so don't understand why it is necessary). Again, I would be cautious about any numbers that come out, and focus on the 5th percentile (and understand that it is a wild over-estimate if the data isn't good).

sokoloff 6y ago
I think there's a reasonable question of whether a Monte-Carlo simulation is better or worse than applying actual sequential returns from all 100+ starting years (which is what cFIREsim does and the data of which is available to all). For me, I'd rather backtest against all the actuals than to look to a Monte-Carlo sim with a distribution which is the same as the per-year returns, because I don't believe that sequence of returns year to year are entirely independent, but rather have a relationship to each other because of the longer-term business cycles. This is naturally accomplished by using sequential year actuals. I agree with you on focusing on the low-end/worst-case outcomes. I'm planning our family's retirement based on a 98% success rate as a minimum (and where the 2% cases can be managed by scaling down lifestyle rather than by starving).
- hogFeast 6y ago
  I am not sure what you mean. Using sequential returns from every starting year in your database is, if I have understood you correctly, the same as using block bootstrap (at the limit, sequential returns from every starting year is the same as block bootstraps of lifespan length taken an infinite number of times). You don't look at yearly returns. There are many reasons for not doing this but the basic one is volatility clustering. Yearly returns are mostly independent, but volatility is not...again, that is why you use a block bootstrap (you can vary the length of block but for this kind of analysis more than once decade makes sense).
  - sokoloff 6y ago
    > You don't look like at pear returns I don't understand what you mean here, but I'd like to. Could you explain? I'm using cFIREsim to simulate 100+ different retirement trajectories based on historical results and ensure that 98% of them exceed my desired spending in all years.
bhelkey 6y ago
> There are likely issues with the data, which is not easily solvable given that complete databases are expensive. >> cFIREsim uses historical stock/bond/gold/inflation data from 1871 to present[1] I am not an expert here buy my impress is that this data is publicly available. What data is this model missing? [1] cfiresim.com/about/
- hogFeast 6y ago
  It isn't, it is expensive and time-consuming to collect this data, and the edge is huge (I know a fund manager who has been spending several million a year for the best part of a decade collecting historical financial statement data). Also, a lot of the large databases are known to be inaccurate (for example, CRSP). I think the best long-term data is Credit Suisse DSM and Global Financial Data (I can't attest to the accuracy of the latter but have heard it is good). The website says it is using Schiller's data...so the model is missing almost everything. Stock markets have existed in multiple countries for centuries, using one country makes no sense. It also looks like the bond data is totally wrong (prices rather than total returns).
  - modeless 6y ago
    I was thinking about making a monte carlo simulation for my own use. Obviously I'm not going to be spending millions for the data. What's the best freely or cheaply (i.e. not "call us for pricing") available historical data for stocks and bonds? Or is the free data just so bad it's not worth doing at all? And what are the typical problems with the free data?

sokoloff 6y ago

I think there's a reasonable question of whether a Monte-Carlo simulation is better or worse than applying actual sequential returns from all 100+ starting years (which is what cFIREsim does and the data of which is available to all). For me, I'd rather backtest against all the actuals than to look to a Monte-Carlo sim with a distribution which is the same as the per-year returns, because I don't believe that sequence of returns year to year are entirely independent, but rather have a relationship to each other because of the longer-term business cycles. This is naturally accomplished by using sequential year actuals. I agree with you on focusing on the low-end/worst-case outcomes. I'm planning our family's retirement based on a 98% success rate as a minimum (and where the 2% cases can be managed by scaling down lifestyle rather than by starving).

hogFeast 6y ago
I am not sure what you mean. Using sequential returns from every starting year in your database is, if I have understood you correctly, the same as using block bootstrap (at the limit, sequential returns from every starting year is the same as block bootstraps of lifespan length taken an infinite number of times). You don't look at yearly returns. There are many reasons for not doing this but the basic one is volatility clustering. Yearly returns are mostly independent, but volatility is not...again, that is why you use a block bootstrap (you can vary the length of block but for this kind of analysis more than once decade makes sense).
- sokoloff 6y ago
  > You don't look like at pear returns I don't understand what you mean here, but I'd like to. Could you explain? I'm using cFIREsim to simulate 100+ different retirement trajectories based on historical results and ensure that 98% of them exceed my desired spending in all years.

bhelkey 6y ago

> There are likely issues with the data, which is not easily solvable given that complete databases are expensive. >> cFIREsim uses historical stock/bond/gold/inflation data from 1871 to present[1] I am not an expert here buy my impress is that this data is publicly available. What data is this model missing? [1] cfiresim.com/about/

hogFeast 6y ago
It isn't, it is expensive and time-consuming to collect this data, and the edge is huge (I know a fund manager who has been spending several million a year for the best part of a decade collecting historical financial statement data). Also, a lot of the large databases are known to be inaccurate (for example, CRSP). I think the best long-term data is Credit Suisse DSM and Global Financial Data (I can't attest to the accuracy of the latter but have heard it is good). The website says it is using Schiller's data...so the model is missing almost everything. Stock markets have existed in multiple countries for centuries, using one country makes no sense. It also looks like the bond data is totally wrong (prices rather than total returns).
- modeless 6y ago
  I was thinking about making a monte carlo simulation for my own use. Obviously I'm not going to be spending millions for the data. What's the best freely or cheaply (i.e. not "call us for pricing") available historical data for stocks and bonds? Or is the free data just so bad it's not worth doing at all? And what are the typical problems with the free data?