3 ms·

Unless I misread this it sounds like the assignment is on 2 day chunks rather than some unique user ID. Just doing this at impression rather than user level mak

Unless I misread this it sounds like the assignment is on 2 day chunks rather than some unique user ID. Just doing this at impression rather than user level makes me question the applicability of the results, but really interesting read nonetheless.

gwern 9y ago
Per-user level doesn't capture the effects I am interested in. Imagine you click on a website and the banner ad makes you subconciously less happy and so you don't submit it to HN for thousands of other users to read; how would randomizing on you and thousands of others' user IDs help? There's no way to link your user ID's randomization to the subsequent visits (or non-visits) of thousands of other people; it just averages everything out to 0 ('500 visitors were exposed to ads, 500 were not; conclusion: ???'), when in fact the real answer is something like '-1000 hits because user yellow_postit was exposed to ads and didn't like them'. But if you did it on a per-day or per-week or per-month basis, then you could see effects like that: "on ad-free weeks, there are twice as many submissions to Reddit which drives +100 hits per day" etc. Then you don't need to try to track each individual user's activities or reshares - whatever the effects are, they'll show up at the population level.
- portman 9y ago
  Isn't this particular test susceptible to confounding effects? Traffic fluctuates day-to-day, week-to-week, and month-to-month, so how can you be sure it was the presence-or-absence of ads and not something else? If you randomize at the visitor level, you are sampling from both high-and-low traffic days, and control for any external fluctuations.
  - gwern 9y ago
    Because you're randomizing at the 2-day level, on average there will be just as many advertising/high-traffic days as advertising/low-traffic days, and as many no-advertising/high-traffic days as no-advertising/low-traffic days. The randomization is unaffected by traffic and uncorrelated with it. The unit of analysis is each day, not each visitor. This is why it has to be run for several months, otherwise you don't wind up with a decent n=50 pairs. That's the tradeoff here: it lets you look at the totals, but it takes a lot longer than if you randomize per visitor in which case you could finish the test in a few days, often.
    - jackgolding 9y ago
      The stats went well over my head but as a web analyst I thought maybe you could have asked a simpler question. Pick a segment of your site such as visits from Google Search who landed on your homepage (most likely people who searched "gwern") which should reduce a lot of those spikes.
      - gwern 9y ago
        Subsetting will also increase the variance of each datapoint (consider the extreme case of picking a subset which was 0 or 1 visits per day), so is probably not a win. It's also hard to imagine what subset properly reflects all sources of traffic and so is informative about the total effect of advertising. Search queries definitely is not it.

gwern 9y ago

Per-user level doesn't capture the effects I am interested in. Imagine you click on a website and the banner ad makes you subconciously less happy and so you don't submit it to HN for thousands of other users to read; how would randomizing on you and thousands of others' user IDs help? There's no way to link your user ID's randomization to the subsequent visits (or non-visits) of thousands of other people; it just averages everything out to 0 ('500 visitors were exposed to ads, 500 were not; conclusion: ???'), when in fact the real answer is something like '-1000 hits because user yellow_postit was exposed to ads and didn't like them'. But if you did it on a per-day or per-week or per-month basis, then you could see effects like that: "on ad-free weeks, there are twice as many submissions to Reddit which drives +100 hits per day" etc. Then you don't need to try to track each individual user's activities or reshares - whatever the effects are, they'll show up at the population level.

portman 9y ago
Isn't this particular test susceptible to confounding effects? Traffic fluctuates day-to-day, week-to-week, and month-to-month, so how can you be sure it was the presence-or-absence of ads and not something else? If you randomize at the visitor level, you are sampling from both high-and-low traffic days, and control for any external fluctuations.
- gwern 9y ago
  Because you're randomizing at the 2-day level, on average there will be just as many advertising/high-traffic days as advertising/low-traffic days, and as many no-advertising/high-traffic days as no-advertising/low-traffic days. The randomization is unaffected by traffic and uncorrelated with it. The unit of analysis is each day, not each visitor. This is why it has to be run for several months, otherwise you don't wind up with a decent n=50 pairs. That's the tradeoff here: it lets you look at the totals, but it takes a lot longer than if you randomize per visitor in which case you could finish the test in a few days, often.
  - jackgolding 9y ago
    The stats went well over my head but as a web analyst I thought maybe you could have asked a simpler question. Pick a segment of your site such as visits from Google Search who landed on your homepage (most likely people who searched "gwern") which should reduce a lot of those spikes.
    - gwern 9y ago
      Subsetting will also increase the variance of each datapoint (consider the extreme case of picking a subset which was 0 or 1 visits per day), so is probably not a win. It's also hard to imagine what subset properly reflects all sources of traffic and so is informative about the total effect of advertising. Search queries definitely is not it.