4 ms·
Facebook are 100% not going to lie to you through their experimentation platform. The whole point of it is to help people accurately measure the results of thei
by disgruntledphd2 7y ago
Facebook are 100% not going to lie to you through their experimentation platform. The whole point of it is to help people accurately measure the results of their ads.
In general, it will be extremely difficult to run accurate tests on FB (or indeed Google) ads without using some of their infrastructure, as these platforms can balance users appropriately in terms of likely response to your ads (for instance, ensuring that both ads are shown to people with approximately the same click probability).
Without this, you run the risk of making bad decisions because one ad got served to users who were far more likely to convert.
Of course you should definitely run your own tests, but I wouldn't completely discount the platform's tools as they do have advantages which are impossible for you to replicate (i.e. balancing users in terms of response rates in each condition).
- tjpaudio 7y agoYea, but what are they using to analyze your results? You literally have no idea how robust (or not) their statistical methods are. It's been my experience that 3rd party solutions are notoriously shoddy in this department. They have no incentive to be robust.
- disgruntledphd2 7y agoAFAIK (it's been over a year), they use intent to treat approach, and only remove users from the experiment after they have already been matched to an ad. This allows them to ensure that the users are matched in terms of expected conversion rate, which allows for valid results. I believe that there were rumours of using bayesian analysis in the future, but I believe it's a two-sided p-value based on the differences in your chosen outcome. Reasonably simple, and mostly effective.