4 ms·
This reminded me of information theory. The idea of entropy in information - that if you can predict the next 0 or 1 with better than 50% certainty, you can com
by will_critchlow 16y ago
This reminded me of information theory. The idea of entropy in information - that if you can predict the next 0 or 1 with better than 50% certainty, you can compress that information (or in this case, you could have designed a better sequence of experiments).
I do think it's worth differentiating between types of failure, however. You don't want 50% of your experiments failing in the sense of failing to give you an answer in either direction (i.e. being inconclusive).
And this leaves aside the topic (covered in the article) of the more serious kinds of failure - of live code etc. You probably don't want that happening 50% of the time.
- mturmon 16y agoThat's precisely where Norvig is coming from. Given their infrastructure, it's entirely possible that some portion of their A/B testing is run this way (i.e., test thresholds chosen to maximize information gain).
- dkarl 16y agoI had a boss who always pushed me away from low-probability experiments. Whenever I suggested trying something that was unlikely to work, he probed to see if the single low-probability experiment could be restructured as a series of 50/50 experiments. It turned a lot of failures into partial successes that could be applied to future problems.