38 ms·
I'm a bit troubled with my results being used to support that conclusion, in the matter of a pharmaceutical researcher who published "Penicillin is not an effec
by patio11 16y ago
I'm a bit troubled with my results being used to support that conclusion, in the matter of a pharmaceutical researcher who published "Penicillin is not an effective treatment for brain cancer" would be troubled to be cited in support of "Well, that tears it, medical research is a sham."
It seems to me that the issue isn't A/B testing, it is blind adoption of other people's recommendations. You certainly don't need A/B testing to give people suboptimal advice -- designers, developers, and marketers do it every day without any evidence whatsoever! ("Eat more bananas, they're good for brain cancer!" <-- the state of web design best practices and, ahem, not far off from the state of nutrition research)
- Tichy 16y agoOf course, I have no doubts that A/B Testing is useful, I take more issues with the way it is being marketed. Likewise a lot of SEO is probably useful, but there are also a lot of black sheep.
- Tichy 16y agoBtw. I don't like your analogy, I did not conclude from your example that A/B Testing never works. But one example is enough to show that it does not always work. Specifically, the claim "use A/B Testing and your signup rate will improve x%" is apparently not true.
- aristus 16y agoThe same thing happens with optimizing code for speed/efficiency/etc. People want rules of thumb and they don't want to hear the truth, which is "Rules of thumb are usually bullshit. You need to measure, change something, then measure again. Your system may or may not be the same as mine."