4 ms·
There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscrip
by goodside 4y ago
There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made the statistics more complicated.
- tsimionescu 4y agoSure, but in many cases, such as the example given by GP, long-term AB testing is hard or almost impossible. For the testing to have validity, you need the A and B cohorts to be stable, and have little or no overlap, and that is hard for long time spans for anything that is not account based (and somewhat dangerous even for account-based things, as people will almost certainly start to notice that they are getting a different experience than their peers, which may upset them).
- goodside 4y agoIn online dating, at least, this is a non-issue. Using an online dating app is, ironically, a solitary enough activity that people don’t go around comparing whether their UI is different from their friends’ UI. You of course can’t let the same user see two versions, but that just means doing permanent group assignment on signup. We used to A/B test subscription prices over enormous ranges (e.g., randomly giving some people 90% discounts) and approximately nobody noticed outside of obscure Reddit threads.
- avisser 4y agoI wonder if you two are talking past each other a little. I'm thinking that A/B testing for content is a different beast than A/B testing for experience.
- goodside 4y agoI’m not disagreeing — My point is really, “not all AB testing is bad, even if the kind you’re most familiar with leads to shitty content.” My second comment was just more of side note.
- tsimionescu 4y agoSure, this is a niche with some very nice properties for this type of thing.
- Philadelphia 4y agoOkCupid has completely destroyed its interface and utility, so whatever they’re doing doesn’t seem to be working anymore.
- goodside 4y agoI left in 2015, as soon as it became apparent the party was over. OkCupid went downhill for a lot of reasons, but overly aggressive A/B testing wasn't one of them.
- dan_mctree 4y agoWhat do you think the reasons were?
- im3w1l 4y agoDid you A/B test the matching algorithms?
- goodside 4y agoDon’t recall specifically. There maybe may have been randomized A/B tests on the “special blend” at some point, I think — it was never spelled out on the site, but I think that was the experimental mix du jour and we tended to use that instead of forced randomization.