3 ms·
Yea, but what are they using to analyze your results? You literally have no idea how robust (or not) their statistical methods are. It's been my experience th
by tjpaudio 7y ago
Yea, but what are they using to analyze your results? You literally have no idea how robust (or not) their statistical methods are. It's been my experience that 3rd party solutions are notoriously shoddy in this department. They have no incentive to be robust.
- disgruntledphd2 7y agoAFAIK (it's been over a year), they use intent to treat approach, and only remove users from the experiment after they have already been matched to an ad. This allows them to ensure that the users are matched in terms of expected conversion rate, which allows for valid results. I believe that there were rumours of using bayesian analysis in the future, but I believe it's a two-sided p-value based on the differences in your chosen outcome. Reasonably simple, and mostly effective.