4 ms·

There is absolutely no doubt that all those sites, and many others, are continuously running such experiments. There are also new financing methods, paywall im

There is absolutely no doubt that all those sites, and many others, are continuously running such experiments. There are also new financing methods, paywall implementations, prices, cooperations, etc. being tested all the time, with some successes and many failures. While people here and in general often seem to consider publishers to be rather stupid, I think there's enough evidence at this point to conclude that it has become a particularly difficult business, what with almost half of revenue almost instantly disappearing when classifieds move to the net, and people generally considering to appear out of thin air because the costs of copying are zero.

gwern 8y ago
> There is absolutely no doubt that all those sites, and many others, are continuously running such experiments. They are running randomized experiments, but they are not necessarily running the right experiments. If the bottom line is revenue, then attrition may not be directly measured; if they are testing on per page-view or per session instead of bucketing users permanently, then they can miss effects too; and if they are testing by bucketing users simultaneously, they can miss non-additive site-wide effects. If you look at my comment, I link two publicly-documented randomized experiments, and both set up/analyze their tests in an unusual way. After I published my results, I heard about some parallel results, but not that many. I heard whispers that Google might (or might not) have internally run equivalent tests; I heard from Chris Stucchio that an unspecified company he worked for had run an A/B test with similar results; I heard from a mobile developer that they had run a similar experiment but found no effect of ads; Pandora subsequently published their A/B test; and Mozilla has published this. I also read through most of the 'advertising avoidance' literature in academia going back 30 years, and found nothing relevant. And that's it. So, if everyone is always continuously running obviously relevant experiments... where are they?
- manigandham 8y ago
  Do you mean where are the published results? That's not something publishers do, so most of the intel will be internal. It's also unlikely that are running experiments constantly as the basic stuff has been figured out. Also most of the innovation happens with adtech vendors who try out new ways to get attention and payments. TrueX and Sourcepoint have been working on some new stuff, along with the various formats that keep being invented.
  - gwern 8y ago
    > Do you mean where are the published results? That's not something publishers do Publishers like Google, Microsoft, and Yahoo do publish many research papers going in depth on A/B tests - such as the effects of page load time - and I've benefited from reading them over the years. So yes, it is something they do.
    - manigandham 8y ago
      None of those companies are considered publishers, and that seems to be more generic UX research rather than ad focused. For example, Washington Post runs experiments specifically with ads as a digital media/news publisher, but you won't see that research published anywhere.
      - gwern 8y ago
        They publish a ton of stuff and are some of the biggest media properties on the Internet. Other publishers like the NYT or the BBC also do writeups of A/B tests and lessons, but they're such small fry I didn't want to mention them. And if 'page loads harm user retention' is worth publishing, 'ads harm user retention' definitely is and they are exactly the companies best placed to find that out.
        manigandham 8y ago
        We're just going in circles here. You asked about relevant experiments as a reply to comments about media publishers and ads. I'm telling you why you don't see that research as official papers. Google, Microsoft (Bing), and Yahoo are not considered media publishers because they are ad networks first. If you just dismiss actual media publishers like NYT and BBC as small fry (when they are some of the biggest) then I'm not sure what you were expecting as an answer. Even if you don't discount their sizes, they do not reveal their research because there's no time or incentive to reveal things like yield management, which is complex, audience specific, and contains competitive intel. That's why it can't be officially found, but you can get details if you talk to their dev teams.

gwern 8y ago

> There is absolutely no doubt that all those sites, and many others, are continuously running such experiments. They are running randomized experiments, but they are not necessarily running the right experiments. If the bottom line is revenue, then attrition may not be directly measured; if they are testing on per page-view or per session instead of bucketing users permanently, then they can miss effects too; and if they are testing by bucketing users simultaneously, they can miss non-additive site-wide effects. If you look at my comment, I link two publicly-documented randomized experiments, and both set up/analyze their tests in an unusual way. After I published my results, I heard about some parallel results, but not that many. I heard whispers that Google might (or might not) have internally run equivalent tests; I heard from Chris Stucchio that an unspecified company he worked for had run an A/B test with similar results; I heard from a mobile developer that they had run a similar experiment but found no effect of ads; Pandora subsequently published their A/B test; and Mozilla has published this. I also read through most of the 'advertising avoidance' literature in academia going back 30 years, and found nothing relevant. And that's it. So, if everyone is always continuously running obviously relevant experiments... where are they?

manigandham 8y ago
Do you mean where are the published results? That's not something publishers do, so most of the intel will be internal. It's also unlikely that are running experiments constantly as the basic stuff has been figured out. Also most of the innovation happens with adtech vendors who try out new ways to get attention and payments. TrueX and Sourcepoint have been working on some new stuff, along with the various formats that keep being invented.
- gwern 8y ago
  > Do you mean where are the published results? That's not something publishers do Publishers like Google, Microsoft, and Yahoo do publish many research papers going in depth on A/B tests - such as the effects of page load time - and I've benefited from reading them over the years. So yes, it is something they do.
  - manigandham 8y ago
    None of those companies are considered publishers, and that seems to be more generic UX research rather than ad focused. For example, Washington Post runs experiments specifically with ads as a digital media/news publisher, but you won't see that research published anywhere.
    - gwern 8y ago
      They publish a ton of stuff and are some of the biggest media properties on the Internet. Other publishers like the NYT or the BBC also do writeups of A/B tests and lessons, but they're such small fry I didn't want to mention them. And if 'page loads harm user retention' is worth publishing, 'ads harm user retention' definitely is and they are exactly the companies best placed to find that out.
      - manigandham 8y ago
        We're just going in circles here. You asked about relevant experiments as a reply to comments about media publishers and ads. I'm telling you why you don't see that research as official papers. Google, Microsoft (Bing), and Yahoo are not considered media publishers because they are ad networks first. If you just dismiss actual media publishers like NYT and BBC as small fry (when they are some of the biggest) then I'm not sure what you were expecting as an answer. Even if you don't discount their sizes, they do not reveal their research because there's no time or incentive to reveal things like yield management, which is complex, audience specific, and contains competitive intel. That's why it can't be officially found, but you can get details if you talk to their dev teams.