6 ms·

Reliably someone comes along every few months to question digital ads. I always come back to analyses of incrementality as the real proof. Take an audience of

Reliably someone comes along every few months to question digital ads. I always come back to analyses of incrementality as the real proof. Take an audience of X people. Divide them in two. Show ads to your test group, don't show to control. Watch your business grow and gauge the lift between the two audiences. The companies that know how to advertise at scale do this constantly and can gauge the real effect of their ad dollars. Facebook, Google and others make these tests possible in their platforms, while other software suites such as Impact Altitude and VisualIQ allow you to do this kind of analysis and testing as well. In the end, most of it proves out to be incremental. There are notable exceptions of course, but when are there not?

matheusmoreira 7y ago
> Take an audience of X people. Divide them in two. Show ads to your test group, don't show to control. Watch your business grow and gauge the lift between the two audiences. So advertising industries use people as guinea pigs to study the "real effect of their ad dollars". No ethics committee, no informed consent, nothing. Yet another reason to block all ads.
- ryan93 7y ago
  Hyperbolic.
- jefftk 7y ago
  If you can ethically do something (show ads) or not do something (not show ads) it's equally ethical to run an experiment where you do the thing some of the time and measure the effect. Internet companies run A/B tests all the time, and it's not controversial.
switzer 7y ago
Agree. I have used the term "incrementality" to communicate the incremental benefit of a given ad compared to a control whereas the article uses "advertising effect". Measurement is also difficult because every ad platform grades their own homework (they each have analytics that attributes sales to their platforms liberally). It is important for a brand to have their own attribution platform that measures incrementality as well as the relative contribution each ad exposure makes toward a given sale. Both of these things are possible to measure. The problem is that the band of users that display positive ROI (when incrementality is measured) is so small that people don't believe the data. Additionally, brand exposure is hard to measure and almost completely ignored in digital advertising. Over time I imagine that there will be more research in this area (beyond just measuring an in-view ad impression)
viraptor 7y ago
How do you separate "people who haven't seen an ad" into a group in Google analytics? I don't believe "organic" source does that, because that means they weren't targeted by the ad in the first place, creating a bias.
- kevinventullo 7y ago
  Off the top of my head, you could do geographic bucketing. Take say 100 cities, and only run ads in 50 of them.
  - kposehn 7y ago
    Geo tests are useful for sure, but also require a lot of data to get statistically significant results. Any noise in your data (seasonality, PR, etc) has to be taken into account as well. They’re harder to execute in general but can work well.
- MichaelBurjack 7y ago
  Advertisers have many tools to measure "ghost ads" aka "people who haven't seen an ad but would otherwise have qualified for all your target criteria". See https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2620078 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2620078 as an example.
  - viraptor 7y ago
    That's of course possible to measure in theory. I meant it as a response to "Facebook, Google and others make these tests possible in their platforms" since I haven't seen it exposed as an explicit option on either.
    - jefftk 7y ago
      You can run this yourself without any help from them: show one group ads for your site, and show another group white rectangles. Compare performance.
    - 7y ago
buboard 7y ago
> Show ads to your test group, don't show to control Which is the wrong experiment to run. The old adage is that 50% of advertising works, you just don't know which part. You need an experiment to prove that ad spending scales with ROI, as ad agencies claim. That may well be completely false.
- kposehn 7y ago
  That’s the thing - you run these test with individual campaigns in many channels. Digital gives you the ability to test this at scale as you grow your advertising, thus you can gauge with high confidence exactly what advertising is scaling with positive ROI. Incrementality testing isn’t limited to your advertising as a whole - you can get down to very fine detail as long as your experiment is correctly designed and you wait for stat-sig results.
- dredmorbius 7y ago
  The basic experiment can be varied to allow for that. Specify a set of zones. Target different levels of advertising at these. Look at sales trends. Rinse, wash, repeat. Note that major corporations often designate specific areas for product testing. The Chicago area, and Australia, see trial runs of McDonalds products and store concepts, as an example. The fundamental concept of testing advertising effectiveness is not new. Here's an account from 1909 (from one of my faviourite sources on advertising and media, Hamilton Holt's Commercialism and Journalism): https://archive.org/details/commercialismjou00holtuoft/page/22 https://archive.org/details/commercialismjou00holtuoft/page/...
- Supermancho 7y ago
  > You need an experiment to prove that ad spending scales with ROI, as ad agencies claim There is a limited audience that can be influenced. Just like house ads on a site (where you advertise to sell your own goods on your own site), there's only so much that is affected because you have finite traffic. There is a curve for every permutation of context (demographic, geographic, etc).
mobjack 7y ago
Acquiring an audiences to divide into a split test is the hard part. It is something that could work for retargeting existing users but doesn't work for acquiring new visitors.
- MichaelBurjack 7y ago
  What makes you believe this to be true? You define an audience. You split it. You test and measure the result. What component of that strategy doesn't work for new visitors? Many eyeball vendors (Google, Facebook, heck traditional TV and other media!) are set up precisely to allow for such cases.
- kposehn 7y ago
  Not quite. One tool with FB lift tests is to make a lookalike audience. FB will then split that audience in two for you so you have a test and control group. The original test required two ad accounts and keeping your overall advertising static (not changing anything) for the duration. Today it is much easier and has a lot more features. Google does a search lift test with YouTube as well. You can run a large set of ads against an audience in YT and google will either suppress them for a subset. Then you can see your overall organic lift based on your video ads.
  - mobjack 7y ago
    FB can track ads using clicks and view through. With a control group, you can only use view through. I am just skeptical of how it is a fair test especially when FB has an interest to prove that ads work. YouTube brand lift is an indirect measure that is hard to translate to actual sales to see if the ads were cost effective. The math is fuzzy when measuring marketing ads and it is real hard to be truly scientific.
    - jefftk 7y ago
      The proper test is to use viewthrough for both the experiment and control
- 7y ago
MiroF 7y ago
Isn't that literally the test described by FB economists in the article that showed that selection effect was 10x the impact of advertising?
jsonne 7y ago
I'm running a campaign that spends about $10,000 a day for a brand on Facebook. In fact at that level they will run said brand lift study for you. So yes, despite this articles implication the data does exist and we do know the difference between advertising to an audience and not.
- mac01021 7y ago
  That's millions per year. What general class of industry are you in?
  - jsonne 7y ago
    1. In the grand scheme of advertising it's actually still a fairly small budget. 2. For this advertiser they really only advertise q4 so it isn't quite that much. 3. It's a CPG in a grocery store is all I can really say.
  - djhn 7y ago
    The average "target" client on the lead list of your average big name advertising agency in a European country with a population of 50 million will have a yearly spend of 50-100 million across all channels, with 1-10 million going on agency fees excluding 3rd party costs. 10k per day on Facebook is a small consumer start-up levels of adspend.

matheusmoreira 7y ago

> Take an audience of X people. Divide them in two. Show ads to your test group, don't show to control. Watch your business grow and gauge the lift between the two audiences. So advertising industries use people as guinea pigs to study the "real effect of their ad dollars". No ethics committee, no informed consent, nothing. Yet another reason to block all ads.

ryan93 7y ago
Hyperbolic.
jefftk 7y ago
If you can ethically do something (show ads) or not do something (not show ads) it's equally ethical to run an experiment where you do the thing some of the time and measure the effect. Internet companies run A/B tests all the time, and it's not controversial.

switzer 7y ago

Agree. I have used the term "incrementality" to communicate the incremental benefit of a given ad compared to a control whereas the article uses "advertising effect". Measurement is also difficult because every ad platform grades their own homework (they each have analytics that attributes sales to their platforms liberally). It is important for a brand to have their own attribution platform that measures incrementality as well as the relative contribution each ad exposure makes toward a given sale. Both of these things are possible to measure. The problem is that the band of users that display positive ROI (when incrementality is measured) is so small that people don't believe the data. Additionally, brand exposure is hard to measure and almost completely ignored in digital advertising. Over time I imagine that there will be more research in this area (beyond just measuring an in-view ad impression)

viraptor 7y ago

How do you separate "people who haven't seen an ad" into a group in Google analytics? I don't believe "organic" source does that, because that means they weren't targeted by the ad in the first place, creating a bias.

kevinventullo 7y ago
Off the top of my head, you could do geographic bucketing. Take say 100 cities, and only run ads in 50 of them.
- kposehn 7y ago
  Geo tests are useful for sure, but also require a lot of data to get statistically significant results. Any noise in your data (seasonality, PR, etc) has to be taken into account as well. They’re harder to execute in general but can work well.
MichaelBurjack 7y ago
Advertisers have many tools to measure "ghost ads" aka "people who haven't seen an ad but would otherwise have qualified for all your target criteria". See https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2620078 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2620078 as an example.
- viraptor 7y ago
  That's of course possible to measure in theory. I meant it as a response to "Facebook, Google and others make these tests possible in their platforms" since I haven't seen it exposed as an explicit option on either.
  - jefftk 7y ago
    You can run this yourself without any help from them: show one group ads for your site, and show another group white rectangles. Compare performance.
  - 7y ago

buboard 7y ago

> Show ads to your test group, don't show to control Which is the wrong experiment to run. The old adage is that 50% of advertising works, you just don't know which part. You need an experiment to prove that ad spending scales with ROI, as ad agencies claim. That may well be completely false.

kposehn 7y ago
That’s the thing - you run these test with individual campaigns in many channels. Digital gives you the ability to test this at scale as you grow your advertising, thus you can gauge with high confidence exactly what advertising is scaling with positive ROI. Incrementality testing isn’t limited to your advertising as a whole - you can get down to very fine detail as long as your experiment is correctly designed and you wait for stat-sig results.
dredmorbius 7y ago
The basic experiment can be varied to allow for that. Specify a set of zones. Target different levels of advertising at these. Look at sales trends. Rinse, wash, repeat. Note that major corporations often designate specific areas for product testing. The Chicago area, and Australia, see trial runs of McDonalds products and store concepts, as an example. The fundamental concept of testing advertising effectiveness is not new. Here's an account from 1909 (from one of my faviourite sources on advertising and media, Hamilton Holt's Commercialism and Journalism): https://archive.org/details/commercialismjou00holtuoft/page/22 https://archive.org/details/commercialismjou00holtuoft/page/...
Supermancho 7y ago
> You need an experiment to prove that ad spending scales with ROI, as ad agencies claim There is a limited audience that can be influenced. Just like house ads on a site (where you advertise to sell your own goods on your own site), there's only so much that is affected because you have finite traffic. There is a curve for every permutation of context (demographic, geographic, etc).

mobjack 7y ago

Acquiring an audiences to divide into a split test is the hard part. It is something that could work for retargeting existing users but doesn't work for acquiring new visitors.

MichaelBurjack 7y ago
What makes you believe this to be true? You define an audience. You split it. You test and measure the result. What component of that strategy doesn't work for new visitors? Many eyeball vendors (Google, Facebook, heck traditional TV and other media!) are set up precisely to allow for such cases.
kposehn 7y ago
Not quite. One tool with FB lift tests is to make a lookalike audience. FB will then split that audience in two for you so you have a test and control group. The original test required two ad accounts and keeping your overall advertising static (not changing anything) for the duration. Today it is much easier and has a lot more features. Google does a search lift test with YouTube as well. You can run a large set of ads against an audience in YT and google will either suppress them for a subset. Then you can see your overall organic lift based on your video ads.
- mobjack 7y ago
  FB can track ads using clicks and view through. With a control group, you can only use view through. I am just skeptical of how it is a fair test especially when FB has an interest to prove that ads work. YouTube brand lift is an indirect measure that is hard to translate to actual sales to see if the ads were cost effective. The math is fuzzy when measuring marketing ads and it is real hard to be truly scientific.
  - jefftk 7y ago
    The proper test is to use viewthrough for both the experiment and control
7y ago

MiroF 7y ago

Isn't that literally the test described by FB economists in the article that showed that selection effect was 10x the impact of advertising?

jsonne 7y ago

I'm running a campaign that spends about $10,000 a day for a brand on Facebook. In fact at that level they will run said brand lift study for you. So yes, despite this articles implication the data does exist and we do know the difference between advertising to an audience and not.

mac01021 7y ago
That's millions per year. What general class of industry are you in?
- jsonne 7y ago
  1. In the grand scheme of advertising it's actually still a fairly small budget. 2. For this advertiser they really only advertise q4 so it isn't quite that much. 3. It's a CPG in a grocery store is all I can really say.
- djhn 7y ago
  The average "target" client on the lead list of your average big name advertising agency in a European country with a population of 50 million will have a yearly spend of 50-100 million across all channels, with 1-10 million going on agency fees excluding 3rd party costs. 10k per day on Facebook is a small consumer start-up levels of adspend.