2 ms·
This is very interesting. I wonder if someone could create a future-sight benchmark for these models? Like, if given a set of newspaper articles for the past N
by gravypod 8mo ago
This is very interesting. I wonder if someone could create a future-sight benchmark for these models? Like, if given a set of newspaper articles for the past N months can it predict if certain world events would happen? We could backtest against results that have happened since the training cutoff.
- houtanb 8mo agoFYI, ForecastBench [1] tests LLMs' out-of-sample forecasting accuracy. The ForecastBench Tournament Leaderboard [2] allows external participants to submit models, most of whom provide some sort of web search / news scaffolding to improve model forecasting accuracy. [1] https://www.forecastbench.org/ https://www.forecastbench.org/ [2] https://www.forecastbench.org/tournament/ https://www.forecastbench.org/tournament/
- kqr 8mo agoThese days computers compete along with humans in forecasting tournaments on Metaculus. They don't quite beat the top humans yet, but they're up there. https://www.metaculus.com/futureeval/ https://www.metaculus.com/futureeval/