4 ms·

Is there any already resolved question in which metaculus forecasters made a surprisingly accurate prediction about an AI advance?

• yunohn 3y ago
  Anyone can signup and vote, so I assume it tends towards the majority option tbh. Seems like obviously noisy data.
  • tersh 3y ago
    The Metaculus prediction weights predictors by how accurate they've been.
    • yunohn 3y ago
      Yes, but if voters are voting the majority (and probable) decision, that just continues the trend of predicting obvious outcomes. It’s not something to do with understanding domain knowledge and voting based on scientific research.
      • avionical 3y ago
        I wouldn't say AGI in 10 years is predicting an obvious outcome. And the same goes for many other questions on the site.
        • yunohn 3y ago
          It will trend towards the major opinion as it gets closer to 10yrs. They haven’t been around long enough.
    • deleted 3y ago
      [deleted]
• usaar333 3y ago
  https://www.metaculus.com/questions/track-record/ https://www.metaculus.com/questions/track-record/ Only 13 resolved binary questions where you had longer prediction horizons (1+ year), the accuracy is zilch in AI category - Brier score of 0.25 which is akin to just guessing out 50% for all questions. Generally overconfident. (Other categories much better - Brier of 0.14 1 year out)
  • avionical 3y ago
    There are several AI categories on the track record page; make sure you're not just selecting the one or you'll miss a lot. There's a careful analysis of the overall track record on AI questions here: https://www.metaculus.com/notebooks/16708/exploring-metaculuss-ai-track-record/ https://www.metaculus.com/notebooks/16708/exploring-metaculu... The short version is that the Brier score is much better than .25 for AI questions, and the weighted Metaculus Prediction is more accurate still.
    • usaar333 3y ago
      Good call on the categories. > The short version is that the Brier score is much better than .25 for AI questions, and the weighted Metaculus Prediction is more accurate still. Added more categories. 1 year out is 0.217. I agree that's better than chance, though "much better"? That said, this is dominated by bad community predictions pre-2020 and there's not much data recently for binary questions. I agree that CRPS is better - but it's not clear to me from that link how early they are looking at questions - accuracy gets better closer to resolve date -- I'm claiming that longer-term predictions are shakier.
      • avionical 3y ago
        And you may want to check the weighted Metaculus Prediction if you haven't already.
    • Imnimo 3y ago
      Can I see the list of questions used in this analysis somewhere? Is it literally just the set of questions I see when I filter for "Resolved" and "Artificial Intelligence"? My impression from browsing that set of questions is that it's a mix of pretty trivial things like "how expensive will chatGPT be?" or "when will Google release Bard?". There are very few questions in the bunch I'd even consider interesting, let alone ones where the metaculus prediction appears to have offered any meaningful insight.