3 ms·
You definitely don’t understand causal inference of ad attribution if this is what you think. I’ll give an example from a real experience I worked on before. A
by mlthoughts2018 6y ago
You definitely don’t understand causal inference of ad attribution if this is what you think.
I’ll give an example from a real experience I worked on before. A client was a large haircut chain and they gave us a data set of individuals that had received a specific ad campaign from them via a loyalty program (so it had a variety of demographic and customer history data).
When we used the transactions data on this cohort to measure specific lift in the ad campaign, it was hugely negative - we thought there must be some kind of bug in the code, stressed out late nights coding and sanity checking everything and finding no issues that could explain it.
Our internal managers said we cannot go back to the client and ask for details or anything until we fix the negative lift because “negative lift isn’t actually possible in real advertising.” (Literal quote from an experienced manager in a well-known advertising analytics firm - I even showed him plenty of research out of Facebook, Google and academia disproving it, but it was totally taboo to mention negative lift).
Finally after some agonizing weeks we had no choice but to go back to the client and ask them to explain the full mechanics of their ad delivery, how people were targeted, to see if there was some bias or something we had no way to account for.
And of course - they told us all these individuals had been given a physical coupon as they were leaving from a haircut. OMG! The physical mechanics of a haircut means you are obviously not coming back for another one for say 3-4 weeks on average, and you also won’t have perfect customer retention, so basically there is no way this kind of campaign can produce positive lift unless you build a statistical model that successfully controls for the built in customer churn and the mechanical delay period before the ad treatment could possibly causally effect people’s next haircut choice.
This is just a drop in the bucket of strange things advertisers do. Often they won’t tell you targeting details like this, because they believe their method of targeting (even as simple as a coupon for loyalty rewards members) is some type of proprietary technique. This gets really really bad with firms that run their own digital advertising. If they are targeting “upper income people who just bought a BMW” via The Trade Desk, that’ll hugely skew any attribution modeling if they refuse to tell you they did that. A ton of the quant work of ad attribution is trying to automatically control for as many of these glaring omissions from the client as you can, since they often lie, refuse to tell you, or are actually wrong about the targeting they believed they used.
In short, measuring whether it was the ad’s effect that led to that macro increase or decrease in revenue is no easier than the general problem of causal inference from non-random trial data lacking clean splits, and often is much harder.
- mlthoughts2018 6y agoPerhaps for a simpler example: imagine you start an ad campaign on the same exact day the government issues a huge stimulus check. You see a big increase of sales. Is it because the ad did a good job of causally impacting people to choose your brand? Or is it because the world changed underneath your feet and suddenly lots of people with pre-existing demand had more money? Maybe your ad even had a negative effect because more people were willing to spend the stimulus check on your product, but they were turned off by the ad - so your revenue went way up but the ad actually lost you money. This is about the simplest type of confounding example you’ll see - real world ad attribution is far messier, with 10 or 15 of these issues all intermixing, some of which you have measured data to possibly use as a control, some of which you don’t. Your client might not even have done any control group at all. They might just say, “we sent this ad to absolutely everyone, how did it perform compared to a hypothetical where we didn’t send it at all?”
- tareqak 6y agoI see what you are saying: that there are too many confounding variables involved and it is not as simple as A implies B. I’d then argue the same thing about attribution then though: a company like Google or Facebook might be measuring many variables for a given user across however many surfaces they can possibly reach, but some set of offline variables that are invisible / unknowable from their vantage point might be the actual cause of a given conversion. If this is indeed the case, then measuring aggregate advertising metrics should be sufficient enough to inform ad spend in the long run.