3 ms·
> The "objective" approach was never actually objective. It had just as many biases and issues as everything after it, but everyone sort of pretended it didn't.
by hn_throwaway_99 7y ago
> The "objective" approach was never actually objective. It had just as many biases and issues as everything after it, but everyone sort of pretended it didn't.
On a related note, I often see similar issues with the A/B testing programs in many startups and large companies. It's not that the objective approach with A/B testing is in any way wrong, it's that the constraints of the tests and the business environment mean it often isn't testing what you think it's testing.
For example, I've seen more than one A/B test where the business goal was to reduce the number of users taking some action (e.g. calling customer service, for example). So instead of fixing the underlying problem, or making online help easier or better to use, the A/B test basically just tests making the customer service link as hard to find/obscure as possible. Run the test for a few weeks, and voila, see how it's a big A/B test winner because support calls are way down!
Not all examples are egregious as that, but fundamentally virtually all of the A/B tests I've seen are limited to a relatively short time period, but aren't really able to test the longer term, follow on effects to brand reputation or customer sentiment.