4 ms·
They're attempting something fantastically difficult to do effectively (arguably beyond human capability) and arguing that their slightly-better-than-chance res
by maxander 8y ago
They're attempting something fantastically difficult to do effectively (arguably beyond human capability) and arguing that their slightly-better-than-chance results indicate that their approach is promising. More likely, the system is picking up on some "low hanging-fruit" indicators in the data (perhaps that corporate filings with certain buzzwords are likely to fail, or that reference projects beyond a certain age, or similar) which would already be easily picked up on by human analysts, and that anything much beyond that would be impossible with anything remotely resembling the present method.
I want to posit a general hypothesis (perhaps it's already been said); better-than-chance performance by a classifier on some dataset is not evidence that a similar (or any) classifier can perform better on the same data.
- pasta 8y agoI was thinking about weather prediction. Now climate is changing you see that all the models are becoming less and less acurate. So yeah some models might be performing better but overall they are all performing worse.
- nl 8y agoNow climate is changing you see that all the models are becoming less and less acurate. That's not true. "Last year, the National Weather Service 5-day forecasts were within 4 degrees of the high temperature. That's as accurate as 2005's 3-day forecasts and a full degree better than the 5-day forecasts of 11 years ago"[1] [1] http://www.mcall.com/business/tech/mc-weather-forecasts-improvement-20170602-story.html http://www.mcall.com/business/tech/mc-weather-forecasts-impr... https://www.washingtonpost.com/news/capital-weather-gang/wp/2017/02/03/the-amazing-improvements-in-winter-weather-forecasting-since-the-1970s/ https://www.washingtonpost.com/news/capital-weather-gang/wp/...
- tephra 8y agoHave there been any research into the decline of weather prediction models? Sounds really interesting.
- starpilot 8y agoWeather models are constantly being validated and improved. They are getting better as computation power and grid resolution have increased. The climate is changing but the laws of physics are not. Also, climate is different from weather.
- ataturk 8y agoYes, like "gender" climate is fungible and able to be described in whatever politically expedient manner is most advantageous.
- wisty 8y agoWeather prediction is constantly getting more accurate. And since it only really cares about what happens in 7 (now maybe 10) days climate change is simply not a factor.
- kokey 8y agoYup, it might be picking up on a few clear negative signals like profit warnings and positive signals like mergers. It doesn't seem to be doing much better than the old Bayesian classifiers on Reuters feeds do, though since it's more advanced it may potentially be better, but that's a big maybe.
- soVeryTired 8y agoA fuller analysis would show the time-series of total stock returns and mark the corresponding prediction. I've built these models professionally, and these are some questions I would expect to be asked: Did all the true 'low return' predictions happen during a bear market, or is there a good temporal mix between predictions? Did they all happen for the same company? If so, it it possible the report was revised after publication? What horizon are they measuring performance over? What happens if the change that horizon? What happens if they lag the prediction by a day? What happens if they use a finer granularity than high / medium / low?
- odonnellryan 8y ago> already be easily picked up on by human analysts How?