5 ms·

Wisdom of the crowds.

johnsimer 3y ago
but also skin in the game and supply/demand Should be much more accurate than a simple poll, because people with the belief that the odds are undervalued will put money in the game to hopefully profit off their belief It's more like the Efficient Market Hypothesis
- samatman 3y ago
  There's no skin in the game with Metaculus, it's a prediction aggregator, not a prediction market. Nothing is at stake. This is why you see a somewhat absurd skew on AI doom/safety questions, using Metaculus is a hobby of the cult^H^H^H^H movement, and no one's swooping in to steal their money because there isn't any.
  - gbear605 3y ago
    It’s somewhat more complex than that because Metaculus weights its aggregates by how accurate predictors have been in the past. Definitely not perfect though, and the efficient market hypothesis isn’t relevant at all.
bumby 3y ago
How well does this work when there is fundamentally little information? It works in stock trading because so much information is standardized and regulated. I am skeptical of it being as useful here simply because the crowds don't have much relevant information, as far as I know. For example, do the crowds know about the design redundancy? What about the supplier quality control? Or the reliability data on the check valves used? etc. etc.
- avmich 3y ago
  Engineering, physical principles, I guess. Those who place bets or making predictions may know a thing or two on mechanics, rocket science and history of building launchers.
  - bumby 3y ago
    To put a finer point on it, those seem like rather blunt instruments for prediction compared to how risk is typically measured by aerospace reliability engineers.
    - avmich 3y ago
      I'm sure some aerospace professionals are wondering how it's possible to predict stock behavior on an exchange, but have no problem with educated predictions of launcher performance.
      - bumby 3y ago
        That right. The point being, they are very different systems with different information inputs. Meaning what works for one may not be a good method the the other.
    - persolb 3y ago
      Think of it like top down budget va bottom up budget. RAMS engineers do a bottom up analysis. However, it is fairly typical for the to be subtlety biased in a way that changes the final answer. (Usually by ‘rounding up’ on individual hazards). Top down takes an outside view and says ‘this looks like these other past projects. It should land in around the same place.’ I’ve found top down tends to be more accurate project wise. Bottom up more accurate for specific subsystems.
      - bumby 3y ago
        Related to this discussion, the results found by Feynman in the Challenger investigation found that the top-down risk assessment was off by orders of magnitude.

but also skin in the game and supply/demand Should be much more accurate than a simple poll, because people with the belief that the odds are undervalued will put money in the game to hopefully profit off their belief It's more like the Efficient Market Hypothesis

samatman 3y ago
There's no skin in the game with Metaculus, it's a prediction aggregator, not a prediction market. Nothing is at stake. This is why you see a somewhat absurd skew on AI doom/safety questions, using Metaculus is a hobby of the cult^H^H^H^H movement, and no one's swooping in to steal their money because there isn't any.
- gbear605 3y ago
  It’s somewhat more complex than that because Metaculus weights its aggregates by how accurate predictors have been in the past. Definitely not perfect though, and the efficient market hypothesis isn’t relevant at all.

bumby 3y ago

How well does this work when there is fundamentally little information? It works in stock trading because so much information is standardized and regulated. I am skeptical of it being as useful here simply because the crowds don't have much relevant information, as far as I know. For example, do the crowds know about the design redundancy? What about the supplier quality control? Or the reliability data on the check valves used? etc. etc.

avmich 3y ago
Engineering, physical principles, I guess. Those who place bets or making predictions may know a thing or two on mechanics, rocket science and history of building launchers.
- bumby 3y ago
  To put a finer point on it, those seem like rather blunt instruments for prediction compared to how risk is typically measured by aerospace reliability engineers.
  - avmich 3y ago
    I'm sure some aerospace professionals are wondering how it's possible to predict stock behavior on an exchange, but have no problem with educated predictions of launcher performance.
    - bumby 3y ago
      That right. The point being, they are very different systems with different information inputs. Meaning what works for one may not be a good method the the other.
  - persolb 3y ago
    Think of it like top down budget va bottom up budget. RAMS engineers do a bottom up analysis. However, it is fairly typical for the to be subtlety biased in a way that changes the final answer. (Usually by ‘rounding up’ on individual hazards). Top down takes an outside view and says ‘this looks like these other past projects. It should land in around the same place.’ I’ve found top down tends to be more accurate project wise. Bottom up more accurate for specific subsystems.
    - bumby 3y ago
      Related to this discussion, the results found by Feynman in the Challenger investigation found that the top-down risk assessment was off by orders of magnitude.