4 ms·
It is interesting to see the math behind why it did not works and probably it is a good approach to never discard an idea just because it sounds stupid. But I s
by LukaAl 11y ago
It is interesting to see the math behind why it did not works and probably it is a good approach to never discard an idea just because it sounds stupid. But I see really a little value in this line of thinking.
One of the big issues with experiments is how much value they really drive, which is the real effects. In many situations we do A/B testing for changes whose real impact is a fraction of a percent of the tracked metric. If anyone followed the discussion about p-hacking in science [1] should really start wondering if the way we run experiments, considering the little changes we see, is not nearing p-hacking, especially when I see this ideas floating, and if we don't need to find more significant way of running our experiments.
[1] One of the first article I found about the issue http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106 http://journals.plos.org/plosbiology/article?id=10.1371/jour...
- squarecog 11y agoco-author of blog post here. We covered some of this, at a high level, here: https://blog.twitter.com/2015/the-what-and-why-of-product-experimentation-at-twitter-0 https://blog.twitter.com/2015/the-what-and-why-of-product-ex... Certainly not everything needs to be A/B tested -- depending on the nature of the change and the kind of insight one is looking to get, MAB may be better, if it applies, or a number of other approaches can be more beneficial, including not running an experiment at all. After all, Twitter itself came from a side project at the podcasting company Odeo. If Twitter was a/b tested, I'm sure it wouldn't move any podcasting metrics in a meaningful way :). I'm obviously biased -- but I've seen really important insights gained from A/B tests, and wouldn't dismiss them too easily.