3 ms·
I don’t dislike them inherently, but as people tend to believe they’re a one-stop solution it’s worth pointing out what they can’t do: - On their own they don’
by robin_reala 1y ago
I don’t dislike them inherently, but as people tend to believe they’re a one-stop solution it’s worth pointing out what they can’t do:
- On their own they don’t provide a hypothesis. You need to do user research to discover those.
- On their own they don’t tell you the value in what you’re testing. You need further analysis to understand that.
- They’re typically a short-term solution to longer-term problems. It’s easy to use them to achieve local maxima without understanding the wider picture.
- They’ll only tell you what the majority of your users find effective, without highlighting what hurts the journey for a minority of users. For example you could A/B test something that improves the journey for every group apart from visually impaired users, but block them completely, and still implement it because it’s the winning bucket.
Yes, there are guardrails you can put in place for all of these, and good teams do, but that requires a degree of maturity that is hard for a lot of organisations.