3 ms·
I had a PM who pushed us to A/B test _everything_. We hired a new graphic designer who suggested that we change our product links from ALL CAPS to Title Case (a
by happimess 4y ago
I had a PM who pushed us to A/B test _everything_. We hired a new graphic designer who suggested that we change our product links from ALL CAPS to Title Case (a very popular idea on the team, and his first real suggestion after a few weeks with us), and she insisted that we A/B test it first. It felt like an insult to him, and a dumb test since title case looked way better.
The three key outcomes I observed from the relentless A/B testing were UI antipatterns, team burnout, and a well-attended conference talk about "how we ran 105 A/B tests in a year, and what we learned".
- gorbachev 4y agoWas one of the learnings "Everyone hated the product manager"?
- edmundsauto 4y agoI've had a similar experience, although my learning was "people, even experts, are really terrible at understanding which treatment will perform better". We always run >=3 variants, surveyed the dozen team members on which one they thought would run. Over the years, there was no clear pattern over who could make that prediction. IE, it's not possible to predict which is the most effective treatment, even when you include a really bad idea in the treatments!