3 ms·
ML messing with you is an important consideration when you are trying to run this sort of experiment on a platform that you don't understand very well. The mos
by jefftk 5y ago
ML messing with you is an important consideration when you are trying to run this sort of experiment on a platform that you don't understand very well. The most simplest reliable way to run this experiment is to give the ad network an HTML creative that looks like:
<script>
var treatment = readTreatmentFromCookie();
if (!treatment) {
treatment = Math.random() < 0.5 ? CONTROL : EXPERIMENT;
writeTreatmentToCookie(treatment);
}
if (treatment === EXPERIMENT) {
showAd(PRODUCT);
} else {
showAd(CHARITY);
}
</script>
The creative is opaque to the network, which means it's not going to be able to do anything fancy like you're describing.
Many networks offer the ability to run a fully supported incrementality study, where they effectively use one company's ads as a control for another's, but this is a version of an experiment that you can run even if you don't trust the ad network at all.