4 ms·

The likelihood ratio is just true no matter how you use it. The meaning of an experimental result should not depend on the experimenter's private state of mind

by Eliezer 16y ago

The likelihood ratio is just true no matter how you use it. The meaning of an experimental result should not depend on the experimenter's private state of mind (such as the stopping rule they followed) if that state of mind does not affect the experimental apparatus. The likelihood ratio is objective, "statistical significance" is not. See link.

carbocation 16y ago
Totally agreed. But assuming that you wanted to calculate a posterior, what would you have chosen for a prior here?
- Eliezer 16y ago
  (Note: Second paragraph in grandparent was edit, original reply in parent may have been to first paragraph only.) Well, keeping in mind that experimenters should just report likelihood ratios, I'd have estimated a pretty low prior probability that one of two identical pages produced more signups than the other. But I suppose that after seeing a likelihood ratio of 1000/1 favoring the hypothesis "each of these pages has a separate fixed conversion rate" over "both of these pages have the same fixed conversion rate", using Laplace's Rule of Succession (i.e. a flat prior between 0 and 1 for the conversion rate) in all cases, I'd think that something was more likely than not going on.
  - carbocation 16y ago
    For academic journals, I think that what you propose is totally reasonable. I'll try to give a small taste for those not familiar: For a new HIV test, I want to know the likelihood ratios, not other parameters which tend to be driven by the background HIV prevalence in the researcher's specific catchment area. My catchment area's HIV prevalence will differ, so other aspects of the test that are dominated by prevalence will not be the same in my population. However, the likelihood ratio will be. The point that I didn't finish making before getting kicked out of the coffee shop was that, at some point, people will want to compute a posterior, so they'll need to figure out a prior. Your suggestion above is helpful. Edit - Also, you're right in that I saw just the 1st paragraph when I initially replied.

by Eliezer 16y ago

carbocation 16y ago

Totally agreed. But assuming that you wanted to calculate a posterior, what would you have chosen for a prior here?

Eliezer 16y ago
(Note: Second paragraph in grandparent was edit, original reply in parent may have been to first paragraph only.) Well, keeping in mind that experimenters should just report likelihood ratios, I'd have estimated a pretty low prior probability that one of two identical pages produced more signups than the other. But I suppose that after seeing a likelihood ratio of 1000/1 favoring the hypothesis "each of these pages has a separate fixed conversion rate" over "both of these pages have the same fixed conversion rate", using Laplace's Rule of Succession (i.e. a flat prior between 0 and 1 for the conversion rate) in all cases, I'd think that something was more likely than not going on.
- carbocation 16y ago
  For academic journals, I think that what you propose is totally reasonable. I'll try to give a small taste for those not familiar: For a new HIV test, I want to know the likelihood ratios, not other parameters which tend to be driven by the background HIV prevalence in the researcher's specific catchment area. My catchment area's HIV prevalence will differ, so other aspects of the test that are dominated by prevalence will not be the same in my population. However, the likelihood ratio will be. The point that I didn't finish making before getting kicked out of the coffee shop was that, at some point, people will want to compute a posterior, so they'll need to figure out a prior. Your suggestion above is helpful. Edit - Also, you're right in that I saw just the 1st paragraph when I initially replied.