3 ms·
Thank you! > I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results. That i
by maximedupre 4y ago
Thank you!
> I think the only downside to this approach is that there might be some effect based on time of day or the interval affecting your results.
That is definitely true. I'm about to start alternating the versions every 5m to mitigate this. The closer I can get to 0m, the more accurate the results are. This way even if you get a followers spike (let's say you get a viral tweet), the followers will be properly distributed between each version.
- yeahboats 4y agoI think what you really want to do is randomize A or B within each 30m (or 5m) interval. This is basically switchback testing. Door dash has a nice write-up here: https://doordash.engineering/2018/02/13/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash/ https://doordash.engineering/2018/02/13/switchback-tests-and...
- maximedupre 4y agoOh right, that's actually a better technique! Thanks for the idea.
- shay_ker 4y agoYep - we've run many switchback tests so I'm happy to chat more about it. It's a lot more akin to what you're building here, from a stats point of view.
- kqr 4y agoThen you get the opposite problem: users are more likely to see multiple versions of the same profile.
- maximedupre 4y agoI feel like that's less of a problem because the user is more likely to convert on the version that he likes more, which would still provide accurate data. But yes, there is no perfect solution with the limitations of the API :D