5 ms·

It's strange to map language to probability ranges. The guidance should be to just say the probability range. No ambiguity. Clear. Actionable and also measurabl

by mempko 1y ago

It's strange to map language to probability ranges. The guidance should be to just say the probability range. No ambiguity. Clear. Actionable and also measurable.

Muromec 1y ago
That's the other way around -- there was no probability range to begin with.
throwaway81523 1y ago
It's still a subjective estimate, but Samosvety (predictor group) does seem to work that way, and HPMOR suggested something similar. Basically assign probabiltiies to less complex unknowns using numbers pulled out of your butt if that's all you can do. Then you can compute conditional probabilities of various more complicated events using those priors. At least then, a consistent set of numbers has carried through the calculation, even if those numbers were wrong at the outset. It's suppose to help your mental clarity. I guess you can also perturb the initial numbers to guess something like a hyperdistribution at the other end. I haven't tried this myself and haven't run across a situation to apply it to lately, but I thought it was interesting.
- andrewflnr 1y ago
  > a consistent set of numbers has carried through the calculation, even if those numbers were wrong at the outset I kind of see how this might be useful, but what I've actually seen is an illusion of certainty from looking at numbers and thinking that means you're being quantitative instead of, you know, pulling things out of your butt. Garbage in, garbage out still applies.
  - throwaway81523 1y ago
    Yes, the potential illusion is most dangerous if you show someone else the numbers and they take them seriously. If they're only for your own calculations then you can remember what they are made of.
    - plorkyeran 1y ago
      In practice people seem to be very bad at remembering that. Pretty universally people act as though doing math on made up numbers makes them less erroneous rather than more.
      - widforss 1y ago
        That's the whole point of Fermi estimates. Find a plausible number given uncertain inputs.
      - chrisweekly 1y ago
        Yeah, mistaking precision for accuracy is a common fallacy.
        JadeNB 1y ago
        > Yeah, mistaking precision for accuracy is a common fallacy. I remember an appealing description of the difference being that a precise archer might sink all their arrows at the same spot on the edge of the target, whereas an accurate archer might sink all their arrows near the bull's eye without always hitting the same spot.
  - mempko 1y ago
    Yes, despite pulling the numbers out of your butt, having them and then seeing how accurate you are helps you calibrate your butt. That way over time, the numbers you pull out of your but become more accurate (and even possibly more precise)
- Jach 1y ago
  If you don't perturb the initial numbers to see what changes to help construct a richer model (as well as one with tighter bounds on what it predicts or forbids), you're leaving out a lot of the benefits of the exercise. Sometimes the prior doesn't matter that much because you find you already have or can easily collect sufficient data to overcome various arbitrary priors. Sometimes different priors will result in surprisingly different conclusions, some of them you can even be more confident in ruling out because of absence of data in their predictions. (Mathematically, absence of evidence is evidence of absence, though of course it's just an inequality so the proof says nothing on whether it's weak or strong evidence.) And of course some priors are straight from the butt but others are more reasonably estimated even if still quite uncertain; in any case much like unit testing of boundary conditions you can still work through a handful of choices to see the effects. Someone mentioned fermi calculations, a related fun exercise in this sort of logic is the work on grabby aliens: https://grabbyaliens.com/ https://grabbyaliens.com/
csours 1y ago
Or use a histogram.
layer8 1y ago
How would you possibly measure the “Probability of an Invasion of Yugoslavia in 1951”, in March 1951?
- konstantinua00 1y ago
  9/12 3 months have passed, 9 to go :)

by mempko 1y ago

It's strange to map language to probability ranges. The guidance should be to just say the probability range. No ambiguity. Clear. Actionable and also measurable.

Muromec 1y ago

That's the other way around -- there was no probability range to begin with.

throwaway81523 1y ago

It's still a subjective estimate, but Samosvety (predictor group) does seem to work that way, and HPMOR suggested something similar. Basically assign probabiltiies to less complex unknowns using numbers pulled out of your butt if that's all you can do. Then you can compute conditional probabilities of various more complicated events using those priors. At least then, a consistent set of numbers has carried through the calculation, even if those numbers were wrong at the outset. It's suppose to help your mental clarity. I guess you can also perturb the initial numbers to guess something like a hyperdistribution at the other end. I haven't tried this myself and haven't run across a situation to apply it to lately, but I thought it was interesting.

andrewflnr 1y ago
> a consistent set of numbers has carried through the calculation, even if those numbers were wrong at the outset I kind of see how this might be useful, but what I've actually seen is an illusion of certainty from looking at numbers and thinking that means you're being quantitative instead of, you know, pulling things out of your butt. Garbage in, garbage out still applies.
- throwaway81523 1y ago
  Yes, the potential illusion is most dangerous if you show someone else the numbers and they take them seriously. If they're only for your own calculations then you can remember what they are made of.
  - plorkyeran 1y ago
    In practice people seem to be very bad at remembering that. Pretty universally people act as though doing math on made up numbers makes them less erroneous rather than more.
    - widforss 1y ago
      That's the whole point of Fermi estimates. Find a plausible number given uncertain inputs.
    - chrisweekly 1y ago
      Yeah, mistaking precision for accuracy is a common fallacy.
      - JadeNB 1y ago
        > Yeah, mistaking precision for accuracy is a common fallacy. I remember an appealing description of the difference being that a precise archer might sink all their arrows at the same spot on the edge of the target, whereas an accurate archer might sink all their arrows near the bull's eye without always hitting the same spot.
- mempko 1y ago
  Yes, despite pulling the numbers out of your butt, having them and then seeing how accurate you are helps you calibrate your butt. That way over time, the numbers you pull out of your but become more accurate (and even possibly more precise)
Jach 1y ago
If you don't perturb the initial numbers to see what changes to help construct a richer model (as well as one with tighter bounds on what it predicts or forbids), you're leaving out a lot of the benefits of the exercise. Sometimes the prior doesn't matter that much because you find you already have or can easily collect sufficient data to overcome various arbitrary priors. Sometimes different priors will result in surprisingly different conclusions, some of them you can even be more confident in ruling out because of absence of data in their predictions. (Mathematically, absence of evidence is evidence of absence, though of course it's just an inequality so the proof says nothing on whether it's weak or strong evidence.) And of course some priors are straight from the butt but others are more reasonably estimated even if still quite uncertain; in any case much like unit testing of boundary conditions you can still work through a handful of choices to see the effects. Someone mentioned fermi calculations, a related fun exercise in this sort of logic is the work on grabby aliens: https://grabbyaliens.com/ https://grabbyaliens.com/

csours 1y ago

Or use a histogram.

layer8 1y ago

How would you possibly measure the “Probability of an Invasion of Yugoslavia in 1951”, in March 1951?

konstantinua00 1y ago
9/12 3 months have passed, 9 to go :)