3 ms·
Hi Have you looked into these two? - Trustworthy Online Controlled Experiments by Kohavi, Tang, and Xu - Statistical Methods in Online A/B Testing by Georgi
by sebg 2y ago
Hi
Have you looked into these two?
- Trustworthy Online Controlled Experiments by Kohavi, Tang, and Xu
- Statistical Methods in Online A/B Testing by Georgi Georgiev
Recommended by stats stackexchange
(https://stats.stackexchange.com/questions/546617/how-can-i-learn-statistical-a-b-testing https://stats.stackexchange.com/questions/546617/how-can-i-l...)
There's a bunch of other books/courses/videos on o'reilly.
Another potential way to approach this learning goal is to look at Evan's tools (https://www.evanmiller.org/ab-testing/ https://www.evanmiller.org/ab-testing/) and go into each one and then look at the JS code for running the tools online.
See if you can go through and comment/write out your thoughts on why it's written that way. of course, you'll have to know some JS for that, but it might be helpful to go through a file like (https://www.evanmiller.org/ab-testing/sample-size.js https://www.evanmiller.org/ab-testing/sample-size.js) and figure out what math is being done.
- sebg 2y agoPS - if you are looking for more of the academic side (cutting edge, much harder statistics), you can start to look at recent work people are doing with A/B tests like this paper -> https://arxiv.org/abs/2002.05670 https://arxiv.org/abs/2002.05670
- sebg 2y agoEven more! Have you seen this video - https://www.nber.org/lecture/2024-methods-lecture-susan-athey-analysis-and-design-multi-armed-bandit-experiments-and-policy https://www.nber.org/lecture/2024-methods-lecture-susan-athe... Might be interesting to you.
- iamacyborg 2y agoI’ll second Trustworthy Online Controlled Experiments. Fantastic read and Ron Kohavi is worth a follow on LinkedIn as he’s quite active there and usually sharing some interesting insights (or politely pointing out poor practices).
- mef 2y agospeaking of Georgi Georgiev, I can’t recommend enough his AB testing tools at https://www.analytics-toolkit.com https://www.analytics-toolkit.com being able to tell when an experiment has entered the Zone of Futility has been super valuable.