5 ms·

Thank you for your comments - there's a lot of issues about this problem that I'm not entirely comfortable with. With that said, I'm not sure that I see the co

by ced 15y ago

Thank you for your comments - there's a lot of issues about this problem that I'm not entirely comfortable with. With that said, I'm not sure that I see the connection between what you're arguing and the significance problem in the original. What do you think of the example with the two doctors? http://lesswrong.com/lw/mt/beautiful_probability/ http://lesswrong.com/lw/mt/beautiful_probability/ Changing stopping rules after seeing the data creates bias/distortions We're talking about a fixed stopping rule, which depends on the data.

xenophanes 15y ago
Oh, I see what you mean about fixed. It's hard to generalize about all data-dependent stopping rules. I do think the one that says "If I get one conclusion, stop. If I get the other, keep trying," has got to be a bad idea! It prevents you from possibly finishing an experiment that concludes you're wrong. The possibility of not terminating seems especially problematic. But if the rule was more like "Roll 10 dice. If at least one came up 6, roll one more die." then it might be harmless and just complicate analyzing the results. I think that even if you get the same data set as your results, as in the linked doctor example, the stopping rule does matter. A reason for this is that it affects the repeatability of experiments. Experiments ought to be repeatable (within margins of error, and using all the same procedures including stopping rules). Let's consider what could happen with repeat trials. Suppose the medicine almost but not quite works according to the standard you're trying to test for that will make the medicine considered a success (cures at least 60 out of 100 people, on average -- not the most realistic standard, but that's not important). Also suppose the data set had 60 cures exactly, not 70 (the following is technically possible, but unlikely, with 70). So, the first experiment with a fixed N=100 will not be repeatable. Slightly too few people will be cured in future trials. The success on the first attempt was good luck within the margin for error (the real average cure rate is 59 out of 100). The second experiment, however, will eventually report that the medicine works on at least 60% of patients on average in all 10 (or whatever) repeat trials, even though this is false. (I think. Maybe they won't make that false "on average" claim but will conclude something different instead? What?) The reason this will happen is basically the same reason that if you flip a coin enough you random walk away from the average (and eventually visit both sides of the average). And because the real cure proportion is so near the goal, there's a pretty good chance you could do a lot of repeat trials without any stalling out for years (which might clue some people in to the problem, though if they strictly ignore data from studies that haven't stopped yet, then maybe it wouldn't). Similarly, imagine a study of coins which had a stopping rule to stop whenever you have at least 60% heads. You'll always be able to get that result and conclude the coin is biased, even if all coins used are fair. You'll often be able to get that result pretty quickly (and actually if you don't get it quickly, but hover around average, the expected time to get it will keep getting worse. But I bet we could come up with an example that doesn't have that property. Or we could consider 20 research groups, 15 of which report coins are biased and 5 of which never publish.). But the point is their result, claiming coins are biased, may be wrong. Even the possibility of the method getting a wrong answer, without anyone having made a mistake in doing it, is a major problem! If someone said, "Never mind their stopping rules, I want to salvage their coin flipping data and use it for my other project" I think they would have a serious problem because it's not a proper random set of coin flip data but is instead limited to various possible sequences of flips and not others. Now it could always be that trials with bad stopping rules get lucky and are correct, and using or believing their data won't work out badly. Their data set could happen to be identical to one that is properly collected. But I think one always has to fear the possibility that they didn't get lucky and their stopping conditions have spoiled the data (especially when you don't have a properly done trial with identical results to compare with) just as the people trying to prove coins are biased could easily spoil their data using fixed but unreasonable stopping conditions.
- ced 15y ago
  It's getting too late for me to think about statistics :) A few points: Similarly, imagine a study of coins which had a stopping rule to stop whenever you have at least 60% heads. You'll always be able to get that result and conclude the coin is biased, even if all coins used are fair. This is not true. Because of the law of large numbers, the probability of ever reaching the 60% decreases with time. I do think the one that says "If I get one conclusion, stop. If I get the other, keep trying," has got to be a bad idea! I understand what you're talking about. I see the potential for a problem. But my understanding is that Bayesian statistics isn't subject to that. Proper Bayesian result reporting doesn't say "We believe that the coin is biased". We would rather say "The probability that this coin is biased is 60%, subject to our assumptions and model". My feeling is that this statement is true: If the model and assumptions are correct, then the Bayesian outcome will be true regardless of the stopping rule. In this case: 60% of coins for which the Bayesian analyst proclaims P(biased)=0.6 WILL be biased (barring sampling variations). The stopping rule doesn't matter. I'll try and figure out a solid explanation by tomorrow.
  - xenophanes 15y ago
    FYI I edited my post to mention the issue about the coins (your first point) shortly after submitting it. I'm guessing you read the non-edited version. > Proper Bayesian result reporting doesn't say "We believe that the coin is biased". We would rather say "The probability that this coin is biased is 60%, subject to our assumptions and model". I'm not really sure what you're getting at here. None of the coins are biased, by premise, so they shouldn't be concluding either thing. If you throw in "if our model and assumptions are right" then you can shift the blame (if they assumed their stopping rule was OK, or came up with a model that says it's OK). But I'm not sure how that substantively helps. Will check back tomorrow for further comments from you.
    - ced 15y ago
      xenophanes, I've been thinking about the problem a lot today. I'm pretty sure that my point is basically right, if the model is correct, but my ideas are not clear enough to explain it properly. Model correctness in Bayesian statistics is a complicated problem, and as far as I can tell, it's not a completely solved one. Bayesians usually agree about their calculations, but there's heavy debate about the "philosophy". In any case, maybe you'll find Eliezer's other post insightful: http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ... I really hope to figure out model correctness, and this optional stopping problem looks a good vector of attack. Thank you for the discussion, and sorry for leaving you hanging! Cedric (if there's any Bayesian out there willing to continue the discussion, my email is in my profile)

by ced 15y ago

xenophanes 15y ago

Oh, I see what you mean about fixed. It's hard to generalize about all data-dependent stopping rules. I do think the one that says "If I get one conclusion, stop. If I get the other, keep trying," has got to be a bad idea! It prevents you from possibly finishing an experiment that concludes you're wrong. The possibility of not terminating seems especially problematic. But if the rule was more like "Roll 10 dice. If at least one came up 6, roll one more die." then it might be harmless and just complicate analyzing the results. I think that even if you get the same data set as your results, as in the linked doctor example, the stopping rule does matter. A reason for this is that it affects the repeatability of experiments. Experiments ought to be repeatable (within margins of error, and using all the same procedures including stopping rules). Let's consider what could happen with repeat trials. Suppose the medicine almost but not quite works according to the standard you're trying to test for that will make the medicine considered a success (cures at least 60 out of 100 people, on average -- not the most realistic standard, but that's not important). Also suppose the data set had 60 cures exactly, not 70 (the following is technically possible, but unlikely, with 70). So, the first experiment with a fixed N=100 will not be repeatable. Slightly too few people will be cured in future trials. The success on the first attempt was good luck within the margin for error (the real average cure rate is 59 out of 100). The second experiment, however, will eventually report that the medicine works on at least 60% of patients on average in all 10 (or whatever) repeat trials, even though this is false. (I think. Maybe they won't make that false "on average" claim but will conclude something different instead? What?) The reason this will happen is basically the same reason that if you flip a coin enough you random walk away from the average (and eventually visit both sides of the average). And because the real cure proportion is so near the goal, there's a pretty good chance you could do a lot of repeat trials without any stalling out for years (which might clue some people in to the problem, though if they strictly ignore data from studies that haven't stopped yet, then maybe it wouldn't). Similarly, imagine a study of coins which had a stopping rule to stop whenever you have at least 60% heads. You'll always be able to get that result and conclude the coin is biased, even if all coins used are fair. You'll often be able to get that result pretty quickly (and actually if you don't get it quickly, but hover around average, the expected time to get it will keep getting worse. But I bet we could come up with an example that doesn't have that property. Or we could consider 20 research groups, 15 of which report coins are biased and 5 of which never publish.). But the point is their result, claiming coins are biased, may be wrong. Even the possibility of the method getting a wrong answer, without anyone having made a mistake in doing it, is a major problem! If someone said, "Never mind their stopping rules, I want to salvage their coin flipping data and use it for my other project" I think they would have a serious problem because it's not a proper random set of coin flip data but is instead limited to various possible sequences of flips and not others. Now it could always be that trials with bad stopping rules get lucky and are correct, and using or believing their data won't work out badly. Their data set could happen to be identical to one that is properly collected. But I think one always has to fear the possibility that they didn't get lucky and their stopping conditions have spoiled the data (especially when you don't have a properly done trial with identical results to compare with) just as the people trying to prove coins are biased could easily spoil their data using fixed but unreasonable stopping conditions.

ced 15y ago
It's getting too late for me to think about statistics :) A few points: Similarly, imagine a study of coins which had a stopping rule to stop whenever you have at least 60% heads. You'll always be able to get that result and conclude the coin is biased, even if all coins used are fair. This is not true. Because of the law of large numbers, the probability of ever reaching the 60% decreases with time. I do think the one that says "If I get one conclusion, stop. If I get the other, keep trying," has got to be a bad idea! I understand what you're talking about. I see the potential for a problem. But my understanding is that Bayesian statistics isn't subject to that. Proper Bayesian result reporting doesn't say "We believe that the coin is biased". We would rather say "The probability that this coin is biased is 60%, subject to our assumptions and model". My feeling is that this statement is true: If the model and assumptions are correct, then the Bayesian outcome will be true regardless of the stopping rule. In this case: 60% of coins for which the Bayesian analyst proclaims P(biased)=0.6 WILL be biased (barring sampling variations). The stopping rule doesn't matter. I'll try and figure out a solid explanation by tomorrow.
- xenophanes 15y ago
  FYI I edited my post to mention the issue about the coins (your first point) shortly after submitting it. I'm guessing you read the non-edited version. > Proper Bayesian result reporting doesn't say "We believe that the coin is biased". We would rather say "The probability that this coin is biased is 60%, subject to our assumptions and model". I'm not really sure what you're getting at here. None of the coins are biased, by premise, so they shouldn't be concluding either thing. If you throw in "if our model and assumptions are right" then you can shift the blame (if they assumed their stopping rule was OK, or came up with a model that says it's OK). But I'm not sure how that substantively helps. Will check back tomorrow for further comments from you.
  - ced 15y ago
    xenophanes, I've been thinking about the problem a lot today. I'm pretty sure that my point is basically right, if the model is correct, but my ideas are not clear enough to explain it properly. Model correctness in Bayesian statistics is a complicated problem, and as far as I can tell, it's not a completely solved one. Bayesians usually agree about their calculations, but there's heavy debate about the "philosophy". In any case, maybe you'll find Eliezer's other post insightful: http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequently_subjective/ http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ... I really hope to figure out model correctness, and this optional stopping problem looks a good vector of attack. Thank you for the discussion, and sorry for leaving you hanging! Cedric (if there's any Bayesian out there willing to continue the discussion, my email is in my profile)