4 ms·
Ask HN: startups and A/B testing
tl;dr: do you have any example of scientifically sound A/B test that works for an early stage startup? Although I'm a fan of A/B testing, I've found myself confused/concerned about timings.
I was reading Google analytics docs on multi-armed bandit experiments for A/B testing.
https://support.google.com/analytics/answer/2844870?hl=en&ref_topic=2844866
In summary, assuming 100 visits/day and 4% conversion rate (CvR) with page A, a scientifically significant experiment for page B to say that the new version actually generates CvR of 5% will require 66 days with the multi-armed approach, or 223 days with the naif 50%-50% approach.
To me, even 66 days seem an enormous amount of time for an early stage startup (say, 1k-to-100k users, 10% active users, growing at 5%/week), where decisions need to be made in a few days. Assuming you can pay for ads, you can probably increase the visits/day, but then the CvR drops down.
So, unless you're testing a really bad A page vs a really good B page, I can't figure out an example where the time required is acceptable (say, max 1 week).
I was wonder if anyone has real numbers that also satisfy a more theoretical appetite ;)
- ig1 13y agoIf you're a typical early stage startup with 100 visits a day then you shouldn't be caring about the difference between 4% and 5% - it's not big enough to matter. At that stage you should be testing big changes not little ones and increasing traffic.
- ecesena 13y agoI agree on focusing on big changes, but these hardly need to be A/B tested. A/B should be more for optimization. So this seems to confirm my idea that (statistically sound) A/B testing is not for startups...
- ig1 13y agoYou should definitely be A/B testing big changes on things like landing pages.
- karolisd 13y agoMeasure first. Setup analytics correctly so as you get traffic, you can analyze your funnel and decide where to focus on testing. Multi-armed bandit testing is cool, but at this point is it worth consuming brain cycles on that vs getting more traffic?
- ecesena 13y agoI totally agree with you guys, traffic first. But if you look around everybody is just talking about A/B testing. My point is that that doesn't look to me statistically meaningful... it's kind like you're taking the A/B testing approach, but applying it the wrong way... What I was asking for was a real example of A/B testing for a startup, but you're basically confirming me that this doesn't exist. :)
- karolisd 13y agoHow many steps are there in your sign-up process? What's the progression rate for each step? What area needs the biggest improvement? Focus on that.