3 ms·

> Show ads to your test group, don't show to control Which is the wrong experiment to run. The old adage is that 50% of advertising works, you just don't know

> Show ads to your test group, don't show to control Which is the wrong experiment to run. The old adage is that 50% of advertising works, you just don't know which part. You need an experiment to prove that ad spending scales with ROI, as ad agencies claim. That may well be completely false.

kposehn 7y ago
That’s the thing - you run these test with individual campaigns in many channels. Digital gives you the ability to test this at scale as you grow your advertising, thus you can gauge with high confidence exactly what advertising is scaling with positive ROI. Incrementality testing isn’t limited to your advertising as a whole - you can get down to very fine detail as long as your experiment is correctly designed and you wait for stat-sig results.
dredmorbius 7y ago
The basic experiment can be varied to allow for that. Specify a set of zones. Target different levels of advertising at these. Look at sales trends. Rinse, wash, repeat. Note that major corporations often designate specific areas for product testing. The Chicago area, and Australia, see trial runs of McDonalds products and store concepts, as an example. The fundamental concept of testing advertising effectiveness is not new. Here's an account from 1909 (from one of my faviourite sources on advertising and media, Hamilton Holt's Commercialism and Journalism): https://archive.org/details/commercialismjou00holtuoft/page/22 https://archive.org/details/commercialismjou00holtuoft/page/...
Supermancho 7y ago
> You need an experiment to prove that ad spending scales with ROI, as ad agencies claim There is a limited audience that can be influenced. Just like house ads on a site (where you advertise to sell your own goods on your own site), there's only so much that is affected because you have finite traffic. There is a curve for every permutation of context (demographic, geographic, etc).

kposehn 7y ago

That’s the thing - you run these test with individual campaigns in many channels. Digital gives you the ability to test this at scale as you grow your advertising, thus you can gauge with high confidence exactly what advertising is scaling with positive ROI. Incrementality testing isn’t limited to your advertising as a whole - you can get down to very fine detail as long as your experiment is correctly designed and you wait for stat-sig results.

dredmorbius 7y ago

The basic experiment can be varied to allow for that. Specify a set of zones. Target different levels of advertising at these. Look at sales trends. Rinse, wash, repeat. Note that major corporations often designate specific areas for product testing. The Chicago area, and Australia, see trial runs of McDonalds products and store concepts, as an example. The fundamental concept of testing advertising effectiveness is not new. Here's an account from 1909 (from one of my faviourite sources on advertising and media, Hamilton Holt's Commercialism and Journalism): https://archive.org/details/commercialismjou00holtuoft/page/22 https://archive.org/details/commercialismjou00holtuoft/page/...

Supermancho 7y ago

> You need an experiment to prove that ad spending scales with ROI, as ad agencies claim There is a limited audience that can be influenced. Just like house ads on a site (where you advertise to sell your own goods on your own site), there's only so much that is affected because you have finite traffic. There is a curve for every permutation of context (demographic, geographic, etc).