I'd strongly consider not using 25% of the data for the correlations. Then backcheck the generated model based on that last 25%. I'd be really tempted to make s

I'd strongly consider not using 25% of the data for the correlations. Then backcheck the generated model based on that last 25%. I'd be really tempted to make sure that the 25% included things like the Global Financial Cluster..ck, just to see how well things go when something unexpected happens. Either that or run separate correlations for each $time_period and see how the results change over time. If they're stable, then you've got better confidence than if they're wildly changing at each interval. Also, realize that people with way more data and processor power have been doing this for a long time, with very mixed results. /me remembers my uncle slaving over the Apple ][e looking at charts and numbers.