Great insight, this is a general issue with making decisions based on predictions. It comes up in Hanson's "Futarchy" / decision markets too. For example, if y

Great insight, this is a general issue with making decisions based on predictions. It comes up in Hanson's "Futarchy" / decision markets too. For example, if you have a huge bias against recruiters and never take their calls because your predictions are < 3%, but the true probability is 50%, you'll never find out those predictions are wrong. One way to mitigate this is to add random exploration. Whenever you decide not to do something, with probability say 1/20, just do it anyway.