4 ms·
Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test? Do websites usually just use t
by anthony_doan 7y ago
Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test?
Do websites usually just use t-test only? Like adding one feature at a time?
- joker3 7y agoPeople should be using more than one design, but that's not as commonly taught as it should be. I'm going to give a talk about that to my company's ML group in about a month, and hopefully that'll improve things where I am.
- jj12345 7y agoNot sure if you'd be willing, but I'd love a quick rundown of the high level takeaways if you'd be willing to drop them here. Are you talking about more than one experimental design in terms of comparing the exp/control distributions or something else?
- joker3 7y agoI'm just going to present the absolute basics of the topic. It'll be a high-level overview of something along the lines of chapters 1-4 of https://www.amazon.com/Analysis-Experiments-Chapman-Statistical-Science/dp/1439868131/ https://www.amazon.com/Analysis-Experiments-Chapman-Statisti....
- jj12345 7y agoThanks for the reference!
- benmaraschino 7y agoNot the OP, but I work on similar problems, albeit in a different setting (healthcare, millions+ of patients). The gist is that you have to bake experimental design into the deployment of your ML model, but in many cases a simple RCT or A/B test just won't cut it. This is largely because when you deploy a model, no matter how sophisticated or accurate, there's no guarantee that it'll actually move the needle in terms of the outcomes you care about—hence you need to run some kind of trial. At the same time, you want to maximize overall utility by not having to allocate more subjects to your control arm (or harmful, or resource-intensive and ineffective treatment arms) than you need to. This latter point is much more of a problem in medicine than it is in other settings, as you can imagine. These considerations point to adaptive designs that balance exploration/exploitation, e.g. those based on multi-armed bandits. Currently working on some cool (in my opinion) variations of MABs that incorporate domain-specific knowledge, so I could talk about this all day!
- joker3 7y agoDo you know a good introduction to adaptive/sequential designs? I'm looking for something along the lines of a textbook aimed at a graduate level seminar.
- tixocloud 7y agoWould be really keen to hear you speak about the subject in greater detail actually. Love having to balance the practicalities of implementing a model in production and validating the outcome while not missing out on utility.
- amirmasoudabdol 7y agoIt's even worst than that. Most of the time the validity of the t-test that they are running is questionable. They are technically running an online t-test and as soon as they find significance they stop. This is fundamentally wrong, and not conclusive at all.
- smueller1234 7y agoA few years ago when I was still working there, involved in the experimentation tooling among other things, we largely excised that particular behavior. What are you basing your assertion on?
- anthony_doan 7y agoCheating on a hypothesis test... that's terrible.
- huac 7y agosay you want to try adding two features, which don't you think interact with each other, e.g. a change to the "pick this room button" and a change to the checkout flow. then, you can randomly assign users to two experiments, independently. your t-test results should then be valid if the two features are independent.
- anthony_doan 7y ago> which don't you think interact with each other, > your t-test results should then be valid if the two features are independent. Assuming that your assumption are correct on interaction effect. You can do a hypothesis test on that assumption while including both factors (the two features). Which will clear away any doubt with a 95% confidence or hire a statistician =).