4 ms·
<strike>This article confuses more than it informs.</strike> The answer to the question whether one can look at experimental data in A/B tests (or by analogy c
by croemer 2y ago
<strike>This article confuses more than it informs.</strike>
The answer to the question whether one can look at experimental data in A/B tests (or by analogy clinical trials) can be answered with yes even though the article doesn't make this very clear. Instead, it gets lost in superficial Frequentist/Bayesian (keyword) name dropping.
The concept of adjusting for early checking is interesting but this article is less useful than just looking at the the original alpha spending paper:
https://pubmed.ncbi.nlm.nih.gov/7973215/ https://pubmed.ncbi.nlm.nih.gov/7973215/
For a Bayesian approach:
https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.4780040412 https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.47800404...
- Paul-Craft 2y agoUnfortunately, your comment does neither, which is the far worse offense. See also: HN guidelines regarding shallow dismissals.
- rhymer 2y agoMy takeaway is to avoid mixing the frequentist and Bayesian approaches. Choose one method: either follow the frequentist approach and avoid early data analysis, or use the Bayesian approach to compute posterior probabilities once data are available. Mixing the two without expertise can lead to errors.
- croemer 2y agoMixing seems like a straw man, why would one? I don't see what's wrong with Frequentist approach with alpha spending. The downside is one needs to understand alpha spending. But doing Bayesian without understanding it can be just as bad.