3 ms·

It doesn't apply in the A/B testing case because you're randomly assigning a person to one group or another. Simpson's paradox can't occur if the two treatment

by jfarmer 17y ago

It doesn't apply in the A/B testing case because you're randomly assigning a person to one group or another. Simpson's paradox can't occur if the two treatment groups are the same size because the distribution of any external variable is constant across the two treatments (within some known, maximal error). For example, if women are 10% of your population then there will be ~10% women in all your treatments.

swombat 17y ago
On the contrary, it does apply if you're doing any kind of splitting of your users along various axes. For example, if you're comparing users from Slashdot to users from Hacker News, you might get the following data set for, say, the average click-through rate over 1 week: Week 1: Slashdot: 500 hits, 50% click-through rate (250 click-throughs) Hacker News: 10 hits, 90% click-through rate (9 click-throughs) Based on this data, you might decide to tweak the page to be more Hacker-News friendly and less Slashdot-friendly. But, hey, maybe you want to run the experiment for one more week, just to be sure. Week 2: Slashdot: 10 hits, 10% click-through rate (9 click-throughs) Hacker News: 500 hits, 20% click-through rate (100 click-throughs) Wow, we have confirmation! Hacker News is clearly a better source of traffic than Slashdot. So you forget about Slashdot, since HN is consistently giving you a better click-through rate than Slashdot. Except, of course, when you look at the combined numbers... Weeks 1 and 2 combined: Slashdot: 50.7% (259 click-throughs for 510 hits) Hacker News: 21.3% (109 click-throughs for 510 hits) So here, the paradox applies in reverse - looking at the weekly data gives you a false impression about how to tweak things to achieve your overall goals. Now, you might say, this is not A/B testing, just monitoring click-through rates of various sources. However, if you simply double this test (so that you're testing two versions of a page), you might once again reach erroneous conclusions about how each version works for each referrer, because of the Simpson effect. So while this is not immediately relevant to the most basic A/B testing, it can easily become relevant to more complex examples. And, of course, if you're doing your A/B testing without doing any splitting of the incoming users, you might miss out some other subtleties about your users, so there's a strong temptation to do so.
- jfarmer 17y ago
  You're right, that isn't A/B testing. :) Everything you said is right, though, and are exactly the types of situations where Simpson's paradox can occur.

by jfarmer 17y ago

swombat 17y ago

On the contrary, it does apply if you're doing any kind of splitting of your users along various axes. For example, if you're comparing users from Slashdot to users from Hacker News, you might get the following data set for, say, the average click-through rate over 1 week: Week 1: Slashdot: 500 hits, 50% click-through rate (250 click-throughs) Hacker News: 10 hits, 90% click-through rate (9 click-throughs) Based on this data, you might decide to tweak the page to be more Hacker-News friendly and less Slashdot-friendly. But, hey, maybe you want to run the experiment for one more week, just to be sure. Week 2: Slashdot: 10 hits, 10% click-through rate (9 click-throughs) Hacker News: 500 hits, 20% click-through rate (100 click-throughs) Wow, we have confirmation! Hacker News is clearly a better source of traffic than Slashdot. So you forget about Slashdot, since HN is consistently giving you a better click-through rate than Slashdot. Except, of course, when you look at the combined numbers... Weeks 1 and 2 combined: Slashdot: 50.7% (259 click-throughs for 510 hits) Hacker News: 21.3% (109 click-throughs for 510 hits) So here, the paradox applies in reverse - looking at the weekly data gives you a false impression about how to tweak things to achieve your overall goals. Now, you might say, this is not A/B testing, just monitoring click-through rates of various sources. However, if you simply double this test (so that you're testing two versions of a page), you might once again reach erroneous conclusions about how each version works for each referrer, because of the Simpson effect. So while this is not immediately relevant to the most basic A/B testing, it can easily become relevant to more complex examples. And, of course, if you're doing your A/B testing without doing any splitting of the incoming users, you might miss out some other subtleties about your users, so there's a strong temptation to do so.

jfarmer 17y ago
You're right, that isn't A/B testing. :) Everything you said is right, though, and are exactly the types of situations where Simpson's paradox can occur.