4 ms·
Don't want to rain on your parade but this is in no way a scientific A/B test. If I understand this correctly, Birdy periodically switches profile information b
by sreejithr 4y ago
Don't want to rain on your parade but this is in no way a scientific A/B test. If I understand this correctly, Birdy periodically switches profile information b/n A and B.
* How do you make sure the same user who was on A variant yesterday doesn't get the B variant today?
If the same user gets both variants over different periods of time, how can you say treatment group engaged more with your content. People shuffle in and out of the treatment group periodically. Thoughts?
- maximedupre 4y agoYou understand Birdy's mechanism correctly :D > How do you make sure the same user who was on A variant yesterday doesn't get the B variant today? You can't. You make the point that the stats generated by Birdy are not 100% accurate, but they're the best we've got given the limitations of the Twitter API :D I make the point that the data is good enough, and statistically significant enough (especially if you let the test run enough) to help you find the better profile version. I agree that this subset of visitors that may come twice or more can skew the data. Birdy's stats are not to be taken as the absolute truth, but as solid clues as to which profile version it is alternating between is providing the best results. I'm looking forward to improving Birdy as Twitter adds more capabilities to their API!