3 ms·

So their argument is that the tests are not providing causal inference, because the platforms can target the two A/B test groups differently. With that in mind

So their argument is that the tests are not providing causal inference, because the platforms can target the two A/B test groups differently. With that in mind, my reading of this is: if you're a researcher trying to say "Advertisement A is more appealing than Y", you need causal inference and these tests won't give them to you. If you're a marketer just trying to determine what ad spend is more efficient, you don't need that causal inference. In other words, they are debunking using these tools for research studies, and not as marketing tools.

tsimionescu 2y ago
I think it's somewhat more important than that. Basically the point is that the A/B tests tell you "which ad is more effective spending on this particular platform with the particular arrangement of users you selected". If you try to expand the user group after the A/B test is over, or if you take the ads to another platform, you shouldn't expect the results to hold. So if you're an ad exec, you should make sure you run separate A/B tests for each specific ad channel you intend to spend money on. An A/B test on Facebook does not tell you anything about how well the same ads would do on Google, and even less so on TV (even assuming you are reaching roughly the same type of audience). This happens because Facebook's targeting mechanisms are not the same as Google's, and so they may optimize the ads differently than Google within the same population groups you selected and give you different results. And TV ads are not targeted at all, even if the audience you are reaching is the same.
- Ozzie_osman 2y ago
  True, but any ad exec worth their salt already knows this, if not because of different targeting algorithms, then at least different user and intent profiles (eg social users are generally younger and lower intent).
  - tsimionescu 2y ago
    True, but any researcher worth their salt should also have already known this. And yet, lots of scientists didn't, and I'd bet the same is true for lots of ad execs.
mathteddybear 2y ago
Yeah it's kind of a different standard There is a section 2.1.3 "Online platform studies versus lift tests" in the article. For the marketing tools purpose, you can use either (or some mixture of both). There are pros and cons to the choice.
aaron695 2y ago
[dead]
ano-ther 2y ago
The issue seems to be that the platforms optimize before showing — presumably because they get paid for click-throughs. Couldn’t they offer an unbiased randomization option with a different payment model (eg based on showings, not clicks)? Would preserve their revenue and researchers get a good tool.

tsimionescu 2y ago

I think it's somewhat more important than that. Basically the point is that the A/B tests tell you "which ad is more effective spending on this particular platform with the particular arrangement of users you selected". If you try to expand the user group after the A/B test is over, or if you take the ads to another platform, you shouldn't expect the results to hold. So if you're an ad exec, you should make sure you run separate A/B tests for each specific ad channel you intend to spend money on. An A/B test on Facebook does not tell you anything about how well the same ads would do on Google, and even less so on TV (even assuming you are reaching roughly the same type of audience). This happens because Facebook's targeting mechanisms are not the same as Google's, and so they may optimize the ads differently than Google within the same population groups you selected and give you different results. And TV ads are not targeted at all, even if the audience you are reaching is the same.

Ozzie_osman 2y ago
True, but any ad exec worth their salt already knows this, if not because of different targeting algorithms, then at least different user and intent profiles (eg social users are generally younger and lower intent).
- tsimionescu 2y ago
  True, but any researcher worth their salt should also have already known this. And yet, lots of scientists didn't, and I'd bet the same is true for lots of ad execs.

mathteddybear 2y ago

Yeah it's kind of a different standard There is a section 2.1.3 "Online platform studies versus lift tests" in the article. For the marketing tools purpose, you can use either (or some mixture of both). There are pros and cons to the choice.

aaron695 2y ago

[dead]

ano-ther 2y ago

The issue seems to be that the platforms optimize before showing — presumably because they get paid for click-throughs. Couldn’t they offer an unbiased randomization option with a different payment model (eg based on showings, not clicks)? Would preserve their revenue and researchers get a good tool.