5 ms·
> In conclusion: A/B testing is conceptually simple, but can be difficult to execute if your product involves anything you'd consider "social interaction". A/B
by fragsworth 9y ago
> In conclusion: A/B testing is conceptually simple, but can be difficult to execute if your product involves anything you'd consider "social interaction".
A/B testing is really hard, even when there is no social interaction internal to your application (like a game). The easiest thing to test is marketing for new users. But even that can be tricky, where one group of people might click fewer ads but are higher quality. This is especially troublesome when you can't tell how many friends they invite through word of mouth. And then if you want to implement other features or fix bugs over the duration of the test, that also poisons the results.
And then the number of incoming users you need is massive.
To do a good A/B test that doesn't mislead you is extremely difficult. Almost to the point that for most companies, I don't think it is worth doing.
> So... I guess my final recommendation is that you should hire some data scientists that like doing experiments.
Like this guy says, you basically need a data science team to do it effectively. If you've got a small team (like I work with), don't waste your time with it. You're better off just adding new features and fixing bugs.
- andreasklinger 9y agoImo comes down two truths for me: - it's easy to push the wrong people down the funnel - use quantitive observation to train your qualified decisions
- autokad 9y agodo the test, but dont get married to the results. every test, even those done best, are evidence in the decision making process, not proof.
- jerednel 9y ago> But even that can be tricky, where one group of people might click fewer ads but are higher quality This is why I always pull performance metrics in addition to impressions/clicks when running A/B tests. Not being in the media/creative department means I often don't see the creatives i'm asked to analyze the performance of so there are occasionally different creatives that have a stronger resonance with people with a higher propensity to convert. This is itself a learning and could lead to using that sort of creative more for acquisition focused campaigns, rotating it out of the upper funnel ad rotation and coming up with a new upper funnel creative to test for the purposes of building large audience pools. > This is especially troublesome when you can't tell how many friends they invite through word of mouth. Could you provide an incentive (gold/credit/whatever) for each new active user referred which could feed into the evaluation criteria of the test as far as the "value" that a particular conversion brought with it? Then I suppose it turns more into a LTV study.