13 ms·

A/B test of banner ads vs. traffic

320 points by luu 9y ago

deleted 9y ago
[deleted]
katzgrau 9y ago
As someone who has founded and runs an adserver for a living (we deal only with legitimate publications, nothing sketchy), I found this thought provoking. Since I deal mostly in niche/local advertising, I might have a unique perspective. It turns out that the more relatable a banner ad is, the less offended the reader is. That may sound obvious, and it probably is. But the general effect is that users engage with and are sometimes genuinely interested in banner ads for their favorite restaurants, venues, and other spots. Local and niche advertisers are part of the community. A Ford or Chevy ad is generally much less relatable. So my overall point is that the nature of the publication and its advertising clients is going to impact whether some readers are turned off - so it's difficult to derive a rule of thumb for banner advertising as a whole.
- Terr_ 9y ago
  I imagine the worst advertising is where it feels both unfeelingly-impersonal yet also spyingly-specific.
- stemuk 9y ago
  Maybe a site like gwern.net is generally less suited for A/B experiments like that because of the unique look and speed 'HTML only' sites have. Sites like cnn.com or yahoo.com are generally much more crowded and colorful, so banner ads blend in much better. This might in return result in a higher user acceptance rate for banner ads and probably a different engagement rate too.
- HugThem 9y ago
  Where do your ads appear? On your site I do not see a publisher signup page.
- hanbura 9y ago
  I think a most of the problems with ads are just caused by how Google and other big players run online advertising. Somewhere along the way "let's add an image of some company our readers might be interested in" turned into "let's serve a huge Javascript that perceivably slows the browser in order to serve an ad from the highest bidder who might or might not run a criminal enterprise. Also make sure the ads show off the vast amounts of data we collect, we have to appear as creepy as possible". Small ad networks or sites that organize their adverts themselves sometimes have great ads. But I am not sure if the reputation of online ads isn't already damaged beyond repair. Add to that the small revenue from ads (the author makes $30/month from ads and $700/month from patreon)
  - bartread 9y ago
    ^^^This. I run Adsense on https://arcade.ly https://arcade.ly. The pages are heavy anyway because they implement games, and much of the payload is music and, to a lesser extent, sound effects. As an experiment I tried loading the pages without music and was nothing short of appalled by the amount of crap Adsense downloads. Thing is, it's not Adsense itself that's the problem: it's the adverts themselves and the lack of control/moderation Google chooses to exert over ad providers. They serve up tons - I mean really a lot - of JavaScript, which tries to track you in N different ways, all so they can display a hyperlinked image with the most basic of animations and mouse over effects. This payload, often running to well over 1MB, measurably slows load and execution performance on the page. The creepiness infuriates me, but it's the overall shittiness of the development practice that produces so much bloat for so little value I find most offensive. I mean these people go in to work, half-ass throwing together a bunch of code and assets in the worst way imaginable, and then carelessly spooge their effluent all over the web... and then they have the audacity to start whingeing about people running ad-blockers. What do they expect? My site gets next to no traffic at the moment but if/when it's grown a bit I'm going to start looking at non-ad ways to earn money from it. I'm even considering ditching them at the moment to see if that might help grown traffic in itself.
    - prawn 9y ago
      I don't remember it being like this before they tied it in with DoubleClick?
- AJ007 9y ago
  The other side to this is users see the ads, relate to the ads, click the ads, and thus leave your site. Your pageviews will drop. Whether or not one agrees with the outcome, if you have a large site you definitely should be doing independent testing. Google's ad reps, much less reps from other networks, will tell you to do things that harm your site.
  - katzgrau 9y ago
    Well, most ads open in a new window. And most serious publishers barely have the resources to survive let alone do A/B testing. The publishers I work with thankfully shy away from network advertising and only deal with clients they have a direct relationship with. Regardless, it really isn't easy or very lucrative running a digital content based business. There are outliers, sure, but the long tail paints a much different story in my experience.
- majani 9y ago
  This is the biggest problem I have with Adsense. Most of the time it seems like they couldn't find an ad that fits me but their system decided to just throw me a random ad for Forex Trading or whatever. This turns their supposedly sophisticated ad system into a random generator as far as I'm concerned. They could fix this by: 1. Just showing blanks if they can't get something targeted to my preferences 2. Not allowing advertisers to be too general in their targeting if they are doing performance campaigns(CPC and CPA). Only allow CPM advertisers to target broad categories.
  - init-as 9y ago
    I just wanted to say that it’s actually nice not to be the only one who is harassed by forex trading ads around the web. Them and Mobile Strike are pretty much all of the ads I see. (I do have Adblock but can’t block for YouTube mobile etc.)
    - def_true_false 9y ago
      There are other apps for youtube. E.g. newpipe.
  - huac 9y ago
    Your concerns are typically addressed by: 1. Reserve pricing - if no ad is 'good enough', e.g. no advertiser is willing to bid above a certain threshold for you, then no ad is shown 2. ePCM pricing - Adsense predicts what the likelihood of clickthrough/conversion is for a given ad and user, and includes that in its auction algorithm. Roughly, eCPM bid = cost per click / probability of click. This addresses the typical problem of high CPC/CPA bid but low engagement probability ads crowding out otherwise good CPM-based ads. In general responsible advertisers have aligned incentives with viewers. Intrusive or obnoxious (or malicious) advertisements hurt viewers, who are less likely to return.
    - emn13 9y ago
      I think a high reserve price is actually a really neat idea - you'd think that might have the knock-on consequence of reducing the attractiveness of adblockers, and furthermore of reducing viewer fatigue (so that the ads you do show have more impact). If you're smart enough, and take into account the "damage" to your relationship with the viewer when choosing an ad (i.e. aiming primarily for the ad the viewer is most likely to appreciate, and only secondarily other effects such as how likely they are to click and how much the advertiser is willing to pay), you might even be able to make ads the viewer actually wants to see; which in the long run is probably a very good place to be. I wonder if part of the reason that never happened is simply that ads are in a sense "reserved" layout space that need to be filled consistently and cannot be filled later on (due to layout jumpiness).
- deleted 9y ago
  [deleted]
- badrabbit 9y ago
  As a user,I am most offended by ads that target me. I would rather see a completely unrelatable ad. It feels like learning new information that way,as opposed to being stalked and preyed upon.
  - kzrdude 9y ago
    I would love to see more ads that instead target the content of the page. When I'm reading about machine learning, it should be books close to that topic, and other related things. I can't believe little that happens.
    - majewsky 9y ago
      Considering the sprawl of sponsored content, maybe because it's cheaper/easier to just buy the author?
    - pault 9y ago
      It has the added benefit of having a high probability that you'll be interested in the product, since you wouldn't be on the page if you didn't find the topic interesting. Instead we get tracking networks showing you ads for TVs because you bought a TV last week.
      - human 9y ago
        I think the problem with that is that most browsing is done on pages that do not target a valuable topic: news, weather, memes, etc. For that reason, it's better to remind you about that Amazon product still in your cart while you're browsing for random stuff.
    - spc476 9y ago
      I used to to that with Amazon. About a decade ago, I added an Amazon banner, and the link included a search term. So I would add an appropriate search term (related to the top blog entry on the page, which is where the Amazon banner appeared). Sometime in the past few months, Amazon removed that and only allows generic sales banners to be presented.
- smsm42 9y ago
  > It turns out that the more relatable a banner ad is, the less offended the reader is. This is of course true, and kinda obvious, but I am not sure this is workable as business model. I mean, here are not a lot of ads that are immediately relevant for me. Maybe once or twice per month there's something that I buy that ad could sway my decision at least potentially (i.e. expose me to some provider I wasn't aware of, or some brand I haven't considered). The rest is basically wasted on me. How much 2 clicks per month (with unknown conversion rate) are worth to advertisers? I have no idea, but I wonder whether it is possible to support a site on that revenue? I, of course, could be wildly atypical case (for one, I use adblockers and rarely impulse-buy) and maybe typical visitor would be much more engaged with ads. Or it could be that ad buyer industry (not ones that make the ads, but ones that order them - Fords and Chevys of the world) do not have proper tools to see how effective their ads are and just throw a chunk of budget to the wind, get some traffic from it and feel good about it.
  - katzgrau 9y ago
    It's true — but advertising has two sides. You can of course be running a campaign that you're hoping for immediate ROI on, like a flash sale. In that case, you really want clicks and some sort of immediate indicator that the campaign is working, like conversions. But sometimes as a business owner you're also campaigning for visibility / mindshare. At the minimum you want to distribute a message and let people know you exist. That's why people still buy billboards. It's a type of advertising that will probably never go away - especially because it's relatively low cost to implement for all sides - just need a designer to create the ad and you're good to go. It doesn't seem like the most dependable business model but it's pretty timeless even in the digital age. The advertiser has an ego and instinct and is ultimately going to buy what they feel is going to work for their business.
- lactinfo 9y ago
  What the hell is a relatable advertisement?
yellow_postit 9y ago
Unless I misread this it sounds like the assignment is on 2 day chunks rather than some unique user ID. Just doing this at impression rather than user level makes me question the applicability of the results, but really interesting read nonetheless.
- gwern 9y ago
  Per-user level doesn't capture the effects I am interested in. Imagine you click on a website and the banner ad makes you subconciously less happy and so you don't submit it to HN for thousands of other users to read; how would randomizing on you and thousands of others' user IDs help? There's no way to link your user ID's randomization to the subsequent visits (or non-visits) of thousands of other people; it just averages everything out to 0 ('500 visitors were exposed to ads, 500 were not; conclusion: ???'), when in fact the real answer is something like '-1000 hits because user yellow_postit was exposed to ads and didn't like them'. But if you did it on a per-day or per-week or per-month basis, then you could see effects like that: "on ad-free weeks, there are twice as many submissions to Reddit which drives +100 hits per day" etc. Then you don't need to try to track each individual user's activities or reshares - whatever the effects are, they'll show up at the population level.
  - portman 9y ago
    Isn't this particular test susceptible to confounding effects? Traffic fluctuates day-to-day, week-to-week, and month-to-month, so how can you be sure it was the presence-or-absence of ads and not something else? If you randomize at the visitor level, you are sampling from both high-and-low traffic days, and control for any external fluctuations.
    - gwern 9y ago
      Because you're randomizing at the 2-day level, on average there will be just as many advertising/high-traffic days as advertising/low-traffic days, and as many no-advertising/high-traffic days as no-advertising/low-traffic days. The randomization is unaffected by traffic and uncorrelated with it. The unit of analysis is each day, not each visitor. This is why it has to be run for several months, otherwise you don't wind up with a decent n=50 pairs. That's the tradeoff here: it lets you look at the totals, but it takes a lot longer than if you randomize per visitor in which case you could finish the test in a few days, often.
HugThem 9y ago
He seems to not show how the ads looked like on his page. He seems to only attribute it to load time. But maybe the ads were visually annoying? Why no screenshots?
- gwern 9y ago
  I have adblock so I don't usually see them and didn't make a point of screenshotting them. When I did (usually checking in Chromium that some CSS change was working), they were fairly ordinary AdSense ads, IMO. Like on DNB FAQ, they would usually be Lumosity or college ads. They weren't flashing or visually offensive, so I never saw them as a problem.
  - HugThem 9y ago
    I looked at some of your pages on archive.org but saw no ads either. Can you give me a hint how to find the pages with the ads?
    - gwern 9y ago
      It was on almost all of the HTML pages (except the index & Modafinil) up to September. Perhaps the JS doesn't run when loaded from IA or something?
      - HugThem 9y ago
        Good question. Even if it does not load, it should still show the adsense code in the html. But looking at some random post, I don't see it: http://web.archive.org/web/20150304235943/http://www.gwern.net/Haskell%20Summer%20of%20Code http://web.archive.org/web/20150304235943/http://www.gwern.n... Adsense is usually a script loaded from googlesyndication.com which is not in the source here. Maybe archive.org rewrote the html so much that it is not there anymore?
        figgis 9y ago
        After a few moments of searching I'm going to assume that whoever specifically scraped and uploaded the page may have removed the adsense javascript. Doesn't look to be a normal archive.org thing.
magicmikexxl 9y ago
There are ways to have comments that are a lot less awful than Disqus.
- spiderfarmer 9y ago
  I scanned the article to see if it was interesting enough to dive into the details, but when I saw the Disqus plugin I closed the tab. What's the point of dropping Adsense if you keep the Disqus plugin?
  - burntwater 9y ago
    Simple, they value the discussion capabilities provided by Disqus. Disqus was worth keeping, Adsense was not.
    - spiderfarmer 9y ago
      That’s my point exactly. Did they do the same exercise for Disqus? Or did they get all scientific on evaluating Adsense, and go with a “hunch” for Disqus? That’s like saying “Marlboro cigarettes are harmful, here’s proof, let’s grab a pack of Lucky Strikes and discuss”.
      - gwern 9y ago
        It's more like smoking two brands of cigarettes and then deciding to stop half your smoking; that is surely an improvement... This is just whataboutism. AdSense should be judged on its own merits, and it fails badly: it does nothing I want (aside from delivering quite small amounts of money) at an unacceptable cost. Maybe Disqus is also bad but I would have to ponder how much I want commenting and how to deal with the extant corpus of comments and in any case, at the moment I am gearing up for another ad test to try to figure out what about AdSense is so harmful (and help convince the doubters that there is an effect at all), so I can't test Disqus without complicating things a lot.
- gwern 9y ago
  Which are and which work with a static Hakyll site?
soared 9y ago
I honestly couldn't follow most of the math but found this really interesting. I used to do UX analyses at a marketing agency and would constantly link clients to [1]. Page speed is huge, especially on desktop where its easier to navigate away. OP mentions this, but I imagine the effect is massive on a site where users are actively researching a topic and have tens of tabs open and so can easily switch to a similar website. I do think this test could have been done without banner ads involved. Looking at page load speed per user and how that effects traffic would likely have the same results. From my link below: > A 1 second delay in page response can result in a 7% reduction in conversions. [1] https://blog.kissmetrics.com/loading-time/ https://blog.kissmetrics.com/loading-time/
- gwern 9y ago
  I'm aware of those results, but I assumed that they weren't an issue because I was using the best-practices 'async'/background loading AdSense JS snippets. The ads never block the rendering or loading of the rest of the page, they pop in a second or two later.
- mattmanser 9y ago
  In my experience that kissmetrics blog is a load of nonsense in practice. We massively reduced load time on the all the key pages from 4 secs to roughly <400 ms avg. No 25% bump in sales for us. At the time the client sold about 50-80 things a day on thousands of daily visits. A month later, they sold about 50-80 things a day. This was a fairly standard website similar to booking.com or any search/book site, basically location aware searches with a landing page for each venue and a "book" CTA. I did a similar thing with their mobile site a year or two later, massively reducing the page-weight and reducing loading times on the devices I tested from seconds to again sub-second load times. Again no discernible bump.
  - danieltillett 9y ago
    Did you try to see what happened if you increase the load time above 4 seconds?
  - icelancer 9y ago
username223 9y ago
> A decision analysis of revenue vs readers yields an acceptable damage of ~3% total traffic loss. Power analysis of historical ... traffic data demonstrates that the high autocorrelation yields very low statistical power with standard tests & regressions but acceptable power with ARIMA models. Was all that math worth your time, or would you have made more money selling tamales on the corner?
soared 9y ago
I'm curious if any ad networks or SSPs are differentiating based on load times. I wouldn't think so because big sites that drive lots of impressions tend to load slower and aren't as concerned with page speed as someone like OP. I'd also be curious to see his user to patreon donator conversion rate, and if removed ads and 'gaining' 10% of traffic would be worth it from a revenue stand point.
imh 9y ago
What do you think the mechanism is on the connection between ads and pageviews? Or the mechanism for the autocorrelation of pageviews? Have you been able connect the dots on something like sharing to sites like HN?
- gwern 9y ago
  > What do you think the mechanism is on the connection between ads and pageviews? The mechanism that makes the most sense to me is via influence in social media resharing. This requires affecting only a few people (power users/the '1%') but can plausibly deliver the necessary effect size. The logical next step in the analysis would be to go back into Google Analytics and pull out another set of time-series and try to split between search engine, direct, and referrals; the decrease should be concentrated as search engine < direct < referrals. I'm not looking forward to figuring out how to code that up as a time-series model, though... > Or the mechanism for the autocorrelation of pageviews? Well, some of the mechanisms there are obvious just from the referring URLs. Many people don't read HN via the front page, they use a third-party interface or a daily newsletter; so while OP is getting substantial traffic today, it'll also get traffic tomorrow from people catching up with today's 'best of HN' etc. And from HN it gets auto-posted to Twiter, and of course many people will only see the tweets later when they log into Twitter the next morning or evening. Then you have timezones - perhaps someone is asleep now even if they do read the HN frontpage directly. It might get included in blogger link roundups or sites like The Browser weeks or months from now - more autocorrelation. What about search engines? They'll notice, and will update. No one has mentioned any comparable experiments so right now my A/B test appears to be _sui generis_, so people debating the costs and benefits of advertising might link it over the next few months. And so on. Lots of possible ways.
HoyaSaxa 9y ago
As always, thanks for the very thorough write up. My immediate thoughts were highlighted in the "Discussion" section. Namely, the demographic of your website might respond more negatively to the presence of ads than the general population. This seems to be substantiated by the unusually high use of ad blockers (~60% vs ~18% [1]). I also wonder if some of the drop off could be explained by the use of Patreon and ads. For instance, I would never pay for Hulu because they still show ads. Did you receive any Patreon cancellations or negative feedback from pledged users? [1] US ad block usage as of Feb 2017 https://pagefair.com/blog/2017/adblockreport/ https://pagefair.com/blog/2017/adblockreport/
- gwern 9y ago
  Patreon-wise, I had ads long predating Patreon, so presumably anyone signing up was already OK with that? I don't remember anyone ever complaining about both ads & Patreon, unless there's a complaint form on Patreon.com I haven't been checking. (Looking at the signup numbers, it goes from $619 for August to $645 for September/October, which at least is in the predicted direction, but too few datapoints to tell since the usual trend is upwards.)
badrabbit 9y ago
I don't know how much research has been done on the subject,but personally I don't mind ads in general,I mind ads that target (stalk) me and intrusive ads. I can tolerate ads that target the page content or random ads just fine.
- 0xcde4c3db 9y ago
  > I don't know how much research has been done on the subject,but personally I don't mind ads in general,I mind ads that target (stalk) me and intrusive ads. That used to be my position. I think I still don't mind honest-to-goodness advertising: this product/vendor exists and has these features. I actually do still see this kind of ad on sites connected to electrical engineering; presumably other competitive technical product areas like photography or scientific/industrial equipment have it too. But the more I've experienced the dissonance of being hit with bursts of mainstream advertising after moving to a generally lower-advertising lifestyle (ad blockers/NoScript, cord-cutting, more judicious selection of news sources, etc.), the more distinctly I've felt the reality that we're almost a century into the general trend of "advertising" being a euphemism for psychosocial manipulation. In other words, modern advertising is for-profit propaganda. In my view, it's not enough for the media of advertising to be more passive. The technology is amplifying it, but the overall strategy is what's fundamentally exploitative. Many people think that merely by being cynical about this, they're immunizing themselves. It's tempting to think that if you study the techniques and make snarky remarks like, say, "oh, look how cool this Lexus is; if I buy it that means I'm powerful and important", then you've somehow defeated it. Make no mistake: that's just another strand in the web. The idea that the advertiser wants to push is still being normalized in your mind, even through the filter of irony. I don't intend to be alarmist here. Humanity has just about survived a century of this crap, after all. Life goes on. Revolutions, both figurative and literal, haven't been rendered obsolete by the sheer marketing budgets of incumbents. But I think we have evolved a warped view of what qualifies as normal, honest communication.
gallerdude 9y ago
I used to roll my eyes every time I saw someone on HN talk about "signal to noise ratio," but I'm getting more and more swayed. Humans deal in abstractions: more clean an abstraction is, the more we can hold onto it. So when I read a ~10% difference in traffic, I'm not surprised. Art, in absolutely any form, is always about focusing on your theme. And ads, by their very nature, blur that focus.
Ruud-v-A 9y ago
I am curious, what constitutes “traffic”, and if it is page loads, why is it the right thing to measure? Users who load the page are already visiting, regardless of whether there is an ad on the page or not. (Or does it include a delay to filter visitors who immediately close the page?) And they did not know in advance whether there was going to be an ad, so their decision to visit the site is not based on the presence of ads. (Unless perhaps if they had a particularly bad experience a previous time — bad enough to remember the domain and not follow the link.) I don’t understand how turning ads on and off every two days can be used to understand long-term effects. There appears to be the assumption that there is a relation between a user seeing an ad or not, and subsequent traffic. There might be a small correlation because of visitors browsing the site, or likeliness to share, but if outside of the spikes, the majority of visitors arrive from an old external link or search engine, and they only read a single page, then how does turning ads on and off every two days measure impact on these users?
- eof 9y ago
  It is probably a combination of performance impact, lower page views per visit, and lower return rate.
47 9y ago
How does this experiment methodology account for cross device mixed experience? Because of the multiple devices use, There will be lot of users that have seen both variation.
- gwern 9y ago
  I think changes over time in device should be like any other unobserved variable like age or country or gender which might affect traffic: averaged out by randomization. The point of randomization is that on average, the two groups will be balanced. (If the intervention itself changes the mix and the changed mix then yields more traffic, then that changes what one might speculate about how it works but the simple result of ads->less-traffic is still true.)
buro9 9y ago
What I learned from running hundreds of forums is that: 1. Banners hurt unless they are relevant 2. Filtering for banners that are relevant tends to reduce revenue rather than increase as there is a smaller pool of highly relevant advertisers 3. Removing all banner adverts significantly increased engagement by every metric 4. Some revenue is possible using affiliate links within the content (re-writing existing links rather than inserting awful links on a word match basis) That last one is what I now do. Is it as profitable as running banners everywhere? No. But it is the highest possible engagement with some revenue, and if one focuses energy on reducing costs instead it can work well.
- badestrand 9y ago
  Thanks for sharing. I really wish there was a general web subscription with revenue-sharing between all visited websites. I hate seeing the ads on my own sites but have no viable way to replace them. And as a user I would like to support several sites, apart from disabling the adblocker.
  - pbhjpbhj 9y ago
    Have you tried Flattr, I've not used it but I see it on a lot of sites and gather it fits that niche. https://thenextweb.com/tech/2017/10/24/flattr-now-lets-pay-content-creators-without-actually-anything/ https://thenextweb.com/tech/2017/10/24/flattr-now-lets-pay-c... may be of interest too.
  - bertil 9y ago
    pbhjpbhj is right to mention Flattr. Patreon is also a good idea. Sponsoring is also possible if you can find something relevant and willing: a lot of the podcast and YouTube channels that I follow have audiobook platforms sponsor them.
  - Mz 9y ago
    I have Patreon and a Pay Pal tip jar. I still have ads on some sites. But I have fantasies that I will someday have enough revenue from other things to justify removing ads from most sites.
  - 9y ago
Mz 9y ago
I would love to remove ads from all but one of my sites. But I have not been very effective at monetizing my writing some other way. I removed them from a bunch of sites and saw a decrease in ad revenue, but did not see some noticeable difference in either traffic or tips. (I only later added Patreon.) There are myriad contributing factors that I can't isolate out. People don't like ads, but many of the people who loudly complain about ads will also very aggressively tell you that they will not tip, will not be a Patreon supporter and expect high quality content for free. They will aggressively say that your business model is your problem, not theirs, while actively shooting down any way to monetize content. These are often people who make good money as, for example, programmers. Another thing such people have told me is to get a real job, like they have, and write for free. It is a kind of classism in a sense: Writers who provide good quality web content aren't seen as valuable or serious contributors the way programmers are. Yet, at the same time, people decry both the lack of good content and the trend towards content done as content marketing. I get paid to write content done as content marketing. I rarely write anything I feel bad about. I work on projects I can believe in. But I find it frustrating because I would rather spend more of my time producing high quality, independent content for my own projects. These are projects intended to put out useful information and/or to be entertaining. I am doing some unique things that add value to the world. I run a homeless site with useful information for homeless Americans. Most homeless services have terrible websites. They are almost always donor facing, not client facing. It is very challenging for a homeless person to find useful information online. I had a college class on internet search and I had a hard time finding information I needed. The other thing is that a lot of homeless services are a very negative experience and can help keep you trapped. Services are often designed with the idea that homeless people are incompetent losers who can't be trusted to make good decisions. They often come with strings attached that can help keep a person trapped in poverty. I want to promote market based solutions that can work for homeless individuals, but can lead to a future because they continue to be viable after you get off the street. Programs to help the homeless or help poor people often actively encourage failure. I think I have worked out a couple of models that work, but I am not finding a path forward. I continually come up against the idea that I should just stop caring about marginalized people because there is no money in it. I should just go where the money is, and never mind that the world is going to hell. And I can't quite bring myself to do that because there is scary stuff happening and I feel like if I don't do something, no one will. Because rich people aren't going to solve it. The attitudes of so many of them that poor people just need a handout (UBI), not earning opportunities, is part of the problem. We have the technology to allow people with disabilities, like me and my sons, to work in a way that works for us. But I don't know how to overcome the attitudes that we shouldn't bother to try, we should just accept that we are losers unworthy of any respect. The inability to get taken seriously is a big barrier to getting engagement and traction and that is a big barrier to turning it into money. The whole thing makes me crazy. It is a case of the key is in the safe.
smsm42 9y ago
> ~60% of visitors have adblock This is much higher than I'd expect. Of course, it is a special kind of site, but still. Can we say a model is in serious trouble yet?
- bertil 9y ago
  I think the right metric for AdBlock (like every service) is retention. If most people who try it like it enough to remember how to install it on their following browser that model has a sigmoid-shaped problem.
- Feniks 9y ago
  I'm amazed they can even tell. A lot of people who use adblocking also browse as ghost traffic that can't even be analyzed.
  - Freak_NL 9y ago
    Ad blocking means that your browser does not send HTTP requests for items that match the profile of ads and tracking beacons. A website can detect this by creating a local asset linked to from the HTML that is never rendered, but which looks like an ad from the ad blocker's perspective. If a browser loads your page, but subsequently won't load that dummy asset, it is probably using an ad blocker.
dfabulich 9y ago
The author's approach to estimating the value of traffic, even by subjective criteria, strikes me as puzzling, somewhat arbitrarily valuing ad-free page views at $0.02, despite the fact that they generate no direct revenue. > My website is important to me because it is what I have accomplished & is my livelihood, and if people are not reading it, that is bad. How bad? In lieu of advertising it’s hard to directly quantify the value of a page-view, so I can instead ask myself hypothetically, would I trade ~1 week of traffic for $360 (~$0.02/view, or to put it the other way, $18720/year)? Probably; that’s about the right number - with my current parlous income, I cannot casually throw away hundreds or thousands of dollars for some additional traffic and so I feel comfortable valuing page-views at ~$0.02 or a bit less. If your income is as parlous as you say, then it seems bizarre to value any amount of unmonetized traffic over cash flow. People call page views a "vanity metric" with good reason. You can't put food on your plate with page views. By valuing a single page view at $0.02, you're literally doing work for free, "for exposure." We all make fun of businesses who ask professionals to work for $0 for exposure. It's no better when you're making yourself work for unfair terms! To put this another way, I think the author can probably arrange to buy relevant traffic via Google AdWords at or less than $0.02 per page view. If traffic ("exposure") is that valuable, then, yes, stop running ads on your site, and instead buy ads of your own. But if that sounds outrageous to you, if you would never reach into your pocket to pay Google $360 for a week's worth of website traffic, then you should rethink whether unmonetized page views are really as valuable as you think they are.
- gwern 9y ago
  One can have enough money to value traffic but not enough to spend thousands of dollars for small amounts of additional traffic. As you have less money, the remaining money becomes much more valuable... > it seems bizarre to value any amount of ad-free traffic over cash flow. Also, while I do not know for certain how much each page-view is worth, I do know for certain that it is not exactly $0! Errors made on the assumption it is a cent or two are surely less than errors made on the assumption it is literally worthless. (As it happens, the introspection was pointless because the effect came nowhere near the threshold, and if it had, I would've had to reintrospect anyway since my Patreon has been working better and I have more money since I began the A/B test in January. There's no reason to not revisit thresholds if circumstances have changed, after all.)
dooglius 9y ago
The content is characterized as "highly technical longform static content in a minimalist layout optimized for fast loading & rendering catering to Anglophone STEM-types in the USA", which is definitely true, but I think there's an important subset not mentioned. According to http://www.gwern.net/ http://www.gwern.net/ the most popular three articles on the site are about darknet drug markets, and #4 and #5 are reviews of specific controlled substances in the US. I don't know what the ads for these pages looked like, but here's an example from a Google search for "modafinil" in incognito mode with my adblocker turned off: https://i.imgur.com/3dRssOy.png https://i.imgur.com/3dRssOy.png ["the silk road" and "lsd microdosing" didn't turn up any ads, but it's possible Google does more filtering on its own site than on others... I couldn't easily find any LSD review sites that used AdSense for a fair comparison] If I saw an ad like that in a banner at the top of the page, it's going to cause an immediate, visceral reaction: the site is sketchy, maybe trying to scam me, and any content is going to have a big conflict of interest. To be fair, since I use an ad-blocker, I'm probably more ad-sensitive than the sample set... but who hasn't gotten their fair share of similar email spam before? The CSVs don't have page information, unfortunately, but I'd expect most of the harmful effect comes from the drug-related pages.
- gwern 9y ago
  The modafinil page didn't have AdSense during the experiment, and if it had, I think the content would have not involved modafinil; Google was burned badly years ago by mega-fines for allowing greymarket pharmacies to advertise and I assume it's totally filtered them out, so any served ads would be targeted at other personal characteristics, presumably. (I also vaguely recollect trying to buy some adwords once while experimenting with advertising gwern.net pages, and the drug categories being totally out of bounds.) During the experiment, roughly, the most popular pages were a mix: https://www.gwern.net/About#january-2017---july-2017 https://www.gwern.net/About#january-2017---july-2017 The top 5 were modafinil (excluded from AdSense), spaced repetition (psychology), LSD microdosing, DNB FAQ (also psychology), and "Story Of Your Life" (physics/literary criticism).

320 points by luu 9y ago

deleted 9y ago

[deleted]

As someone who has founded and runs an adserver for a living (we deal only with legitimate publications, nothing sketchy), I found this thought provoking. Since I deal mostly in niche/local advertising, I might have a unique perspective. It turns out that the more relatable a banner ad is, the less offended the reader is. That may sound obvious, and it probably is. But the general effect is that users engage with and are sometimes genuinely interested in banner ads for their favorite restaurants, venues, and other spots. Local and niche advertisers are part of the community. A Ford or Chevy ad is generally much less relatable. So my overall point is that the nature of the publication and its advertising clients is going to impact whether some readers are turned off - so it's difficult to derive a rule of thumb for banner advertising as a whole.

Terr_ 9y ago
I imagine the worst advertising is where it feels both unfeelingly-impersonal yet also spyingly-specific.
stemuk 9y ago
Maybe a site like gwern.net is generally less suited for A/B experiments like that because of the unique look and speed 'HTML only' sites have. Sites like cnn.com or yahoo.com are generally much more crowded and colorful, so banner ads blend in much better. This might in return result in a higher user acceptance rate for banner ads and probably a different engagement rate too.
HugThem 9y ago
Where do your ads appear? On your site I do not see a publisher signup page.
hanbura 9y ago
I think a most of the problems with ads are just caused by how Google and other big players run online advertising. Somewhere along the way "let's add an image of some company our readers might be interested in" turned into "let's serve a huge Javascript that perceivably slows the browser in order to serve an ad from the highest bidder who might or might not run a criminal enterprise. Also make sure the ads show off the vast amounts of data we collect, we have to appear as creepy as possible". Small ad networks or sites that organize their adverts themselves sometimes have great ads. But I am not sure if the reputation of online ads isn't already damaged beyond repair. Add to that the small revenue from ads (the author makes $30/month from ads and $700/month from patreon)
- bartread 9y ago
  ^^^This. I run Adsense on https://arcade.ly https://arcade.ly. The pages are heavy anyway because they implement games, and much of the payload is music and, to a lesser extent, sound effects. As an experiment I tried loading the pages without music and was nothing short of appalled by the amount of crap Adsense downloads. Thing is, it's not Adsense itself that's the problem: it's the adverts themselves and the lack of control/moderation Google chooses to exert over ad providers. They serve up tons - I mean really a lot - of JavaScript, which tries to track you in N different ways, all so they can display a hyperlinked image with the most basic of animations and mouse over effects. This payload, often running to well over 1MB, measurably slows load and execution performance on the page. The creepiness infuriates me, but it's the overall shittiness of the development practice that produces so much bloat for so little value I find most offensive. I mean these people go in to work, half-ass throwing together a bunch of code and assets in the worst way imaginable, and then carelessly spooge their effluent all over the web... and then they have the audacity to start whingeing about people running ad-blockers. What do they expect? My site gets next to no traffic at the moment but if/when it's grown a bit I'm going to start looking at non-ad ways to earn money from it. I'm even considering ditching them at the moment to see if that might help grown traffic in itself.
  - prawn 9y ago
    I don't remember it being like this before they tied it in with DoubleClick?
AJ007 9y ago
The other side to this is users see the ads, relate to the ads, click the ads, and thus leave your site. Your pageviews will drop. Whether or not one agrees with the outcome, if you have a large site you definitely should be doing independent testing. Google's ad reps, much less reps from other networks, will tell you to do things that harm your site.
- katzgrau 9y ago
  Well, most ads open in a new window. And most serious publishers barely have the resources to survive let alone do A/B testing. The publishers I work with thankfully shy away from network advertising and only deal with clients they have a direct relationship with. Regardless, it really isn't easy or very lucrative running a digital content based business. There are outliers, sure, but the long tail paints a much different story in my experience.
majani 9y ago
This is the biggest problem I have with Adsense. Most of the time it seems like they couldn't find an ad that fits me but their system decided to just throw me a random ad for Forex Trading or whatever. This turns their supposedly sophisticated ad system into a random generator as far as I'm concerned. They could fix this by: 1. Just showing blanks if they can't get something targeted to my preferences 2. Not allowing advertisers to be too general in their targeting if they are doing performance campaigns(CPC and CPA). Only allow CPM advertisers to target broad categories.
- init-as 9y ago
  I just wanted to say that it’s actually nice not to be the only one who is harassed by forex trading ads around the web. Them and Mobile Strike are pretty much all of the ads I see. (I do have Adblock but can’t block for YouTube mobile etc.)
  - def_true_false 9y ago
    There are other apps for youtube. E.g. newpipe.
- huac 9y ago
  Your concerns are typically addressed by: 1. Reserve pricing - if no ad is 'good enough', e.g. no advertiser is willing to bid above a certain threshold for you, then no ad is shown 2. ePCM pricing - Adsense predicts what the likelihood of clickthrough/conversion is for a given ad and user, and includes that in its auction algorithm. Roughly, eCPM bid = cost per click / probability of click. This addresses the typical problem of high CPC/CPA bid but low engagement probability ads crowding out otherwise good CPM-based ads. In general responsible advertisers have aligned incentives with viewers. Intrusive or obnoxious (or malicious) advertisements hurt viewers, who are less likely to return.
  - emn13 9y ago
    I think a high reserve price is actually a really neat idea - you'd think that might have the knock-on consequence of reducing the attractiveness of adblockers, and furthermore of reducing viewer fatigue (so that the ads you do show have more impact). If you're smart enough, and take into account the "damage" to your relationship with the viewer when choosing an ad (i.e. aiming primarily for the ad the viewer is most likely to appreciate, and only secondarily other effects such as how likely they are to click and how much the advertiser is willing to pay), you might even be able to make ads the viewer actually wants to see; which in the long run is probably a very good place to be. I wonder if part of the reason that never happened is simply that ads are in a sense "reserved" layout space that need to be filled consistently and cannot be filled later on (due to layout jumpiness).
deleted 9y ago
[deleted]
badrabbit 9y ago
As a user,I am most offended by ads that target me. I would rather see a completely unrelatable ad. It feels like learning new information that way,as opposed to being stalked and preyed upon.
- kzrdude 9y ago
  I would love to see more ads that instead target the content of the page. When I'm reading about machine learning, it should be books close to that topic, and other related things. I can't believe little that happens.
  - majewsky 9y ago
    Considering the sprawl of sponsored content, maybe because it's cheaper/easier to just buy the author?
  - pault 9y ago
    It has the added benefit of having a high probability that you'll be interested in the product, since you wouldn't be on the page if you didn't find the topic interesting. Instead we get tracking networks showing you ads for TVs because you bought a TV last week.
    - human 9y ago
      I think the problem with that is that most browsing is done on pages that do not target a valuable topic: news, weather, memes, etc. For that reason, it's better to remind you about that Amazon product still in your cart while you're browsing for random stuff.
  - spc476 9y ago
    I used to to that with Amazon. About a decade ago, I added an Amazon banner, and the link included a search term. So I would add an appropriate search term (related to the top blog entry on the page, which is where the Amazon banner appeared). Sometime in the past few months, Amazon removed that and only allows generic sales banners to be presented.
smsm42 9y ago
> It turns out that the more relatable a banner ad is, the less offended the reader is. This is of course true, and kinda obvious, but I am not sure this is workable as business model. I mean, here are not a lot of ads that are immediately relevant for me. Maybe once or twice per month there's something that I buy that ad could sway my decision at least potentially (i.e. expose me to some provider I wasn't aware of, or some brand I haven't considered). The rest is basically wasted on me. How much 2 clicks per month (with unknown conversion rate) are worth to advertisers? I have no idea, but I wonder whether it is possible to support a site on that revenue? I, of course, could be wildly atypical case (for one, I use adblockers and rarely impulse-buy) and maybe typical visitor would be much more engaged with ads. Or it could be that ad buyer industry (not ones that make the ads, but ones that order them - Fords and Chevys of the world) do not have proper tools to see how effective their ads are and just throw a chunk of budget to the wind, get some traffic from it and feel good about it.
- katzgrau 9y ago
  It's true — but advertising has two sides. You can of course be running a campaign that you're hoping for immediate ROI on, like a flash sale. In that case, you really want clicks and some sort of immediate indicator that the campaign is working, like conversions. But sometimes as a business owner you're also campaigning for visibility / mindshare. At the minimum you want to distribute a message and let people know you exist. That's why people still buy billboards. It's a type of advertising that will probably never go away - especially because it's relatively low cost to implement for all sides - just need a designer to create the ad and you're good to go. It doesn't seem like the most dependable business model but it's pretty timeless even in the digital age. The advertiser has an ego and instinct and is ultimately going to buy what they feel is going to work for their business.
lactinfo 9y ago
What the hell is a relatable advertisement?

yellow_postit 9y ago

Unless I misread this it sounds like the assignment is on 2 day chunks rather than some unique user ID. Just doing this at impression rather than user level makes me question the applicability of the results, but really interesting read nonetheless.

gwern 9y ago
Per-user level doesn't capture the effects I am interested in. Imagine you click on a website and the banner ad makes you subconciously less happy and so you don't submit it to HN for thousands of other users to read; how would randomizing on you and thousands of others' user IDs help? There's no way to link your user ID's randomization to the subsequent visits (or non-visits) of thousands of other people; it just averages everything out to 0 ('500 visitors were exposed to ads, 500 were not; conclusion: ???'), when in fact the real answer is something like '-1000 hits because user yellow_postit was exposed to ads and didn't like them'. But if you did it on a per-day or per-week or per-month basis, then you could see effects like that: "on ad-free weeks, there are twice as many submissions to Reddit which drives +100 hits per day" etc. Then you don't need to try to track each individual user's activities or reshares - whatever the effects are, they'll show up at the population level.
- portman 9y ago
  Isn't this particular test susceptible to confounding effects? Traffic fluctuates day-to-day, week-to-week, and month-to-month, so how can you be sure it was the presence-or-absence of ads and not something else? If you randomize at the visitor level, you are sampling from both high-and-low traffic days, and control for any external fluctuations.
  - gwern 9y ago
    Because you're randomizing at the 2-day level, on average there will be just as many advertising/high-traffic days as advertising/low-traffic days, and as many no-advertising/high-traffic days as no-advertising/low-traffic days. The randomization is unaffected by traffic and uncorrelated with it. The unit of analysis is each day, not each visitor. This is why it has to be run for several months, otherwise you don't wind up with a decent n=50 pairs. That's the tradeoff here: it lets you look at the totals, but it takes a lot longer than if you randomize per visitor in which case you could finish the test in a few days, often.

HugThem 9y ago

He seems to not show how the ads looked like on his page. He seems to only attribute it to load time. But maybe the ads were visually annoying? Why no screenshots?

gwern 9y ago
I have adblock so I don't usually see them and didn't make a point of screenshotting them. When I did (usually checking in Chromium that some CSS change was working), they were fairly ordinary AdSense ads, IMO. Like on DNB FAQ, they would usually be Lumosity or college ads. They weren't flashing or visually offensive, so I never saw them as a problem.
- HugThem 9y ago
  I looked at some of your pages on archive.org but saw no ads either. Can you give me a hint how to find the pages with the ads?
  - gwern 9y ago
    It was on almost all of the HTML pages (except the index & Modafinil) up to September. Perhaps the JS doesn't run when loaded from IA or something?
    - HugThem 9y ago
      Good question. Even if it does not load, it should still show the adsense code in the html. But looking at some random post, I don't see it: http://web.archive.org/web/20150304235943/http://www.gwern.net/Haskell%20Summer%20of%20Code http://web.archive.org/web/20150304235943/http://www.gwern.n... Adsense is usually a script loaded from googlesyndication.com which is not in the source here. Maybe archive.org rewrote the html so much that it is not there anymore?
      - figgis 9y ago
        After a few moments of searching I'm going to assume that whoever specifically scraped and uploaded the page may have removed the adsense javascript. Doesn't look to be a normal archive.org thing.

magicmikexxl 9y ago

There are ways to have comments that are a lot less awful than Disqus.

spiderfarmer 9y ago
I scanned the article to see if it was interesting enough to dive into the details, but when I saw the Disqus plugin I closed the tab. What's the point of dropping Adsense if you keep the Disqus plugin?
- burntwater 9y ago
  Simple, they value the discussion capabilities provided by Disqus. Disqus was worth keeping, Adsense was not.
  - spiderfarmer 9y ago
    That’s my point exactly. Did they do the same exercise for Disqus? Or did they get all scientific on evaluating Adsense, and go with a “hunch” for Disqus? That’s like saying “Marlboro cigarettes are harmful, here’s proof, let’s grab a pack of Lucky Strikes and discuss”.
    - gwern 9y ago
      It's more like smoking two brands of cigarettes and then deciding to stop half your smoking; that is surely an improvement... This is just whataboutism. AdSense should be judged on its own merits, and it fails badly: it does nothing I want (aside from delivering quite small amounts of money) at an unacceptable cost. Maybe Disqus is also bad but I would have to ponder how much I want commenting and how to deal with the extant corpus of comments and in any case, at the moment I am gearing up for another ad test to try to figure out what about AdSense is so harmful (and help convince the doubters that there is an effect at all), so I can't test Disqus without complicating things a lot.
gwern 9y ago
Which are and which work with a static Hakyll site?

soared 9y ago

I honestly couldn't follow most of the math but found this really interesting. I used to do UX analyses at a marketing agency and would constantly link clients to [1]. Page speed is huge, especially on desktop where its easier to navigate away. OP mentions this, but I imagine the effect is massive on a site where users are actively researching a topic and have tens of tabs open and so can easily switch to a similar website. I do think this test could have been done without banner ads involved. Looking at page load speed per user and how that effects traffic would likely have the same results. From my link below: > A 1 second delay in page response can result in a 7% reduction in conversions. [1] https://blog.kissmetrics.com/loading-time/ https://blog.kissmetrics.com/loading-time/

gwern 9y ago
I'm aware of those results, but I assumed that they weren't an issue because I was using the best-practices 'async'/background loading AdSense JS snippets. The ads never block the rendering or loading of the rest of the page, they pop in a second or two later.
mattmanser 9y ago
In my experience that kissmetrics blog is a load of nonsense in practice. We massively reduced load time on the all the key pages from 4 secs to roughly <400 ms avg. No 25% bump in sales for us. At the time the client sold about 50-80 things a day on thousands of daily visits. A month later, they sold about 50-80 things a day. This was a fairly standard website similar to booking.com or any search/book site, basically location aware searches with a landing page for each venue and a "book" CTA. I did a similar thing with their mobile site a year or two later, massively reducing the page-weight and reducing loading times on the devices I tested from seconds to again sub-second load times. Again no discernible bump.
- danieltillett 9y ago
  Did you try to see what happened if you increase the load time above 4 seconds?
- icelancer 9y ago

username223 9y ago

> A decision analysis of revenue vs readers yields an acceptable damage of ~3% total traffic loss. Power analysis of historical ... traffic data demonstrates that the high autocorrelation yields very low statistical power with standard tests & regressions but acceptable power with ARIMA models. Was all that math worth your time, or would you have made more money selling tamales on the corner?

soared 9y ago

I'm curious if any ad networks or SSPs are differentiating based on load times. I wouldn't think so because big sites that drive lots of impressions tend to load slower and aren't as concerned with page speed as someone like OP. I'd also be curious to see his user to patreon donator conversion rate, and if removed ads and 'gaining' 10% of traffic would be worth it from a revenue stand point.

imh 9y ago

What do you think the mechanism is on the connection between ads and pageviews? Or the mechanism for the autocorrelation of pageviews? Have you been able connect the dots on something like sharing to sites like HN?

gwern 9y ago
> What do you think the mechanism is on the connection between ads and pageviews? The mechanism that makes the most sense to me is via influence in social media resharing. This requires affecting only a few people (power users/the '1%') but can plausibly deliver the necessary effect size. The logical next step in the analysis would be to go back into Google Analytics and pull out another set of time-series and try to split between search engine, direct, and referrals; the decrease should be concentrated as search engine < direct < referrals. I'm not looking forward to figuring out how to code that up as a time-series model, though... > Or the mechanism for the autocorrelation of pageviews? Well, some of the mechanisms there are obvious just from the referring URLs. Many people don't read HN via the front page, they use a third-party interface or a daily newsletter; so while OP is getting substantial traffic today, it'll also get traffic tomorrow from people catching up with today's 'best of HN' etc. And from HN it gets auto-posted to Twiter, and of course many people will only see the tweets later when they log into Twitter the next morning or evening. Then you have timezones - perhaps someone is asleep now even if they do read the HN frontpage directly. It might get included in blogger link roundups or sites like The Browser weeks or months from now - more autocorrelation. What about search engines? They'll notice, and will update. No one has mentioned any comparable experiments so right now my A/B test appears to be _sui generis_, so people debating the costs and benefits of advertising might link it over the next few months. And so on. Lots of possible ways.

HoyaSaxa 9y ago

As always, thanks for the very thorough write up. My immediate thoughts were highlighted in the "Discussion" section. Namely, the demographic of your website might respond more negatively to the presence of ads than the general population. This seems to be substantiated by the unusually high use of ad blockers (~60% vs ~18% [1]). I also wonder if some of the drop off could be explained by the use of Patreon and ads. For instance, I would never pay for Hulu because they still show ads. Did you receive any Patreon cancellations or negative feedback from pledged users? [1] US ad block usage as of Feb 2017 https://pagefair.com/blog/2017/adblockreport/ https://pagefair.com/blog/2017/adblockreport/

gwern 9y ago
Patreon-wise, I had ads long predating Patreon, so presumably anyone signing up was already OK with that? I don't remember anyone ever complaining about both ads & Patreon, unless there's a complaint form on Patreon.com I haven't been checking. (Looking at the signup numbers, it goes from $619 for August to $645 for September/October, which at least is in the predicted direction, but too few datapoints to tell since the usual trend is upwards.)

badrabbit 9y ago

I don't know how much research has been done on the subject,but personally I don't mind ads in general,I mind ads that target (stalk) me and intrusive ads. I can tolerate ads that target the page content or random ads just fine.

0xcde4c3db 9y ago
> I don't know how much research has been done on the subject,but personally I don't mind ads in general,I mind ads that target (stalk) me and intrusive ads. That used to be my position. I think I still don't mind honest-to-goodness advertising: this product/vendor exists and has these features. I actually do still see this kind of ad on sites connected to electrical engineering; presumably other competitive technical product areas like photography or scientific/industrial equipment have it too. But the more I've experienced the dissonance of being hit with bursts of mainstream advertising after moving to a generally lower-advertising lifestyle (ad blockers/NoScript, cord-cutting, more judicious selection of news sources, etc.), the more distinctly I've felt the reality that we're almost a century into the general trend of "advertising" being a euphemism for psychosocial manipulation. In other words, modern advertising is for-profit propaganda. In my view, it's not enough for the media of advertising to be more passive. The technology is amplifying it, but the overall strategy is what's fundamentally exploitative. Many people think that merely by being cynical about this, they're immunizing themselves. It's tempting to think that if you study the techniques and make snarky remarks like, say, "oh, look how cool this Lexus is; if I buy it that means I'm powerful and important", then you've somehow defeated it. Make no mistake: that's just another strand in the web. The idea that the advertiser wants to push is still being normalized in your mind, even through the filter of irony. I don't intend to be alarmist here. Humanity has just about survived a century of this crap, after all. Life goes on. Revolutions, both figurative and literal, haven't been rendered obsolete by the sheer marketing budgets of incumbents. But I think we have evolved a warped view of what qualifies as normal, honest communication.

gallerdude 9y ago

I used to roll my eyes every time I saw someone on HN talk about "signal to noise ratio," but I'm getting more and more swayed. Humans deal in abstractions: more clean an abstraction is, the more we can hold onto it. So when I read a ~10% difference in traffic, I'm not surprised. Art, in absolutely any form, is always about focusing on your theme. And ads, by their very nature, blur that focus.

Ruud-v-A 9y ago

I am curious, what constitutes “traffic”, and if it is page loads, why is it the right thing to measure? Users who load the page are already visiting, regardless of whether there is an ad on the page or not. (Or does it include a delay to filter visitors who immediately close the page?) And they did not know in advance whether there was going to be an ad, so their decision to visit the site is not based on the presence of ads. (Unless perhaps if they had a particularly bad experience a previous time — bad enough to remember the domain and not follow the link.) I don’t understand how turning ads on and off every two days can be used to understand long-term effects. There appears to be the assumption that there is a relation between a user seeing an ad or not, and subsequent traffic. There might be a small correlation because of visitors browsing the site, or likeliness to share, but if outside of the spikes, the majority of visitors arrive from an old external link or search engine, and they only read a single page, then how does turning ads on and off every two days measure impact on these users?

eof 9y ago
It is probably a combination of performance impact, lower page views per visit, and lower return rate.

47 9y ago

How does this experiment methodology account for cross device mixed experience? Because of the multiple devices use, There will be lot of users that have seen both variation.

gwern 9y ago
I think changes over time in device should be like any other unobserved variable like age or country or gender which might affect traffic: averaged out by randomization. The point of randomization is that on average, the two groups will be balanced. (If the intervention itself changes the mix and the changed mix then yields more traffic, then that changes what one might speculate about how it works but the simple result of ads->less-traffic is still true.)

buro9 9y ago

What I learned from running hundreds of forums is that: 1. Banners hurt unless they are relevant 2. Filtering for banners that are relevant tends to reduce revenue rather than increase as there is a smaller pool of highly relevant advertisers 3. Removing all banner adverts significantly increased engagement by every metric 4. Some revenue is possible using affiliate links within the content (re-writing existing links rather than inserting awful links on a word match basis) That last one is what I now do. Is it as profitable as running banners everywhere? No. But it is the highest possible engagement with some revenue, and if one focuses energy on reducing costs instead it can work well.

badestrand 9y ago
Thanks for sharing. I really wish there was a general web subscription with revenue-sharing between all visited websites. I hate seeing the ads on my own sites but have no viable way to replace them. And as a user I would like to support several sites, apart from disabling the adblocker.
- pbhjpbhj 9y ago
  Have you tried Flattr, I've not used it but I see it on a lot of sites and gather it fits that niche. https://thenextweb.com/tech/2017/10/24/flattr-now-lets-pay-content-creators-without-actually-anything/ https://thenextweb.com/tech/2017/10/24/flattr-now-lets-pay-c... may be of interest too.
- bertil 9y ago
  pbhjpbhj is right to mention Flattr. Patreon is also a good idea. Sponsoring is also possible if you can find something relevant and willing: a lot of the podcast and YouTube channels that I follow have audiobook platforms sponsor them.
- Mz 9y ago
  I have Patreon and a Pay Pal tip jar. I still have ads on some sites. But I have fantasies that I will someday have enough revenue from other things to justify removing ads from most sites.
- 9y ago

Mz 9y ago

I would love to remove ads from all but one of my sites. But I have not been very effective at monetizing my writing some other way. I removed them from a bunch of sites and saw a decrease in ad revenue, but did not see some noticeable difference in either traffic or tips. (I only later added Patreon.) There are myriad contributing factors that I can't isolate out. People don't like ads, but many of the people who loudly complain about ads will also very aggressively tell you that they will not tip, will not be a Patreon supporter and expect high quality content for free. They will aggressively say that your business model is your problem, not theirs, while actively shooting down any way to monetize content. These are often people who make good money as, for example, programmers. Another thing such people have told me is to get a real job, like they have, and write for free. It is a kind of classism in a sense: Writers who provide good quality web content aren't seen as valuable or serious contributors the way programmers are. Yet, at the same time, people decry both the lack of good content and the trend towards content done as content marketing. I get paid to write content done as content marketing. I rarely write anything I feel bad about. I work on projects I can believe in. But I find it frustrating because I would rather spend more of my time producing high quality, independent content for my own projects. These are projects intended to put out useful information and/or to be entertaining. I am doing some unique things that add value to the world. I run a homeless site with useful information for homeless Americans. Most homeless services have terrible websites. They are almost always donor facing, not client facing. It is very challenging for a homeless person to find useful information online. I had a college class on internet search and I had a hard time finding information I needed. The other thing is that a lot of homeless services are a very negative experience and can help keep you trapped. Services are often designed with the idea that homeless people are incompetent losers who can't be trusted to make good decisions. They often come with strings attached that can help keep a person trapped in poverty. I want to promote market based solutions that can work for homeless individuals, but can lead to a future because they continue to be viable after you get off the street. Programs to help the homeless or help poor people often actively encourage failure. I think I have worked out a couple of models that work, but I am not finding a path forward. I continually come up against the idea that I should just stop caring about marginalized people because there is no money in it. I should just go where the money is, and never mind that the world is going to hell. And I can't quite bring myself to do that because there is scary stuff happening and I feel like if I don't do something, no one will. Because rich people aren't going to solve it. The attitudes of so many of them that poor people just need a handout (UBI), not earning opportunities, is part of the problem. We have the technology to allow people with disabilities, like me and my sons, to work in a way that works for us. But I don't know how to overcome the attitudes that we shouldn't bother to try, we should just accept that we are losers unworthy of any respect. The inability to get taken seriously is a big barrier to getting engagement and traction and that is a big barrier to turning it into money. The whole thing makes me crazy. It is a case of the key is in the safe.

smsm42 9y ago

> ~60% of visitors have adblock This is much higher than I'd expect. Of course, it is a special kind of site, but still. Can we say a model is in serious trouble yet?

bertil 9y ago
I think the right metric for AdBlock (like every service) is retention. If most people who try it like it enough to remember how to install it on their following browser that model has a sigmoid-shaped problem.
Feniks 9y ago
I'm amazed they can even tell. A lot of people who use adblocking also browse as ghost traffic that can't even be analyzed.
- Freak_NL 9y ago
  Ad blocking means that your browser does not send HTTP requests for items that match the profile of ads and tracking beacons. A website can detect this by creating a local asset linked to from the HTML that is never rendered, but which looks like an ad from the ad blocker's perspective. If a browser loads your page, but subsequently won't load that dummy asset, it is probably using an ad blocker.

dfabulich 9y ago

The author's approach to estimating the value of traffic, even by subjective criteria, strikes me as puzzling, somewhat arbitrarily valuing ad-free page views at $0.02, despite the fact that they generate no direct revenue. > My website is important to me because it is what I have accomplished & is my livelihood, and if people are not reading it, that is bad. How bad? In lieu of advertising it’s hard to directly quantify the value of a page-view, so I can instead ask myself hypothetically, would I trade ~1 week of traffic for $360 (~$0.02/view, or to put it the other way, $18720/year)? Probably; that’s about the right number - with my current parlous income, I cannot casually throw away hundreds or thousands of dollars for some additional traffic and so I feel comfortable valuing page-views at ~$0.02 or a bit less. If your income is as parlous as you say, then it seems bizarre to value any amount of unmonetized traffic over cash flow. People call page views a "vanity metric" with good reason. You can't put food on your plate with page views. By valuing a single page view at $0.02, you're literally doing work for free, "for exposure." We all make fun of businesses who ask professionals to work for $0 for exposure. It's no better when you're making yourself work for unfair terms! To put this another way, I think the author can probably arrange to buy relevant traffic via Google AdWords at or less than $0.02 per page view. If traffic ("exposure") is that valuable, then, yes, stop running ads on your site, and instead buy ads of your own. But if that sounds outrageous to you, if you would never reach into your pocket to pay Google $360 for a week's worth of website traffic, then you should rethink whether unmonetized page views are really as valuable as you think they are.

gwern 9y ago
One can have enough money to value traffic but not enough to spend thousands of dollars for small amounts of additional traffic. As you have less money, the remaining money becomes much more valuable... > it seems bizarre to value any amount of ad-free traffic over cash flow. Also, while I do not know for certain how much each page-view is worth, I do know for certain that it is not exactly $0! Errors made on the assumption it is a cent or two are surely less than errors made on the assumption it is literally worthless. (As it happens, the introspection was pointless because the effect came nowhere near the threshold, and if it had, I would've had to reintrospect anyway since my Patreon has been working better and I have more money since I began the A/B test in January. There's no reason to not revisit thresholds if circumstances have changed, after all.)

dooglius 9y ago

The content is characterized as "highly technical longform static content in a minimalist layout optimized for fast loading & rendering catering to Anglophone STEM-types in the USA", which is definitely true, but I think there's an important subset not mentioned. According to http://www.gwern.net/ http://www.gwern.net/ the most popular three articles on the site are about darknet drug markets, and #4 and #5 are reviews of specific controlled substances in the US. I don't know what the ads for these pages looked like, but here's an example from a Google search for "modafinil" in incognito mode with my adblocker turned off: https://i.imgur.com/3dRssOy.png https://i.imgur.com/3dRssOy.png ["the silk road" and "lsd microdosing" didn't turn up any ads, but it's possible Google does more filtering on its own site than on others... I couldn't easily find any LSD review sites that used AdSense for a fair comparison] If I saw an ad like that in a banner at the top of the page, it's going to cause an immediate, visceral reaction: the site is sketchy, maybe trying to scam me, and any content is going to have a big conflict of interest. To be fair, since I use an ad-blocker, I'm probably more ad-sensitive than the sample set... but who hasn't gotten their fair share of similar email spam before? The CSVs don't have page information, unfortunately, but I'd expect most of the harmful effect comes from the drug-related pages.

gwern 9y ago
The modafinil page didn't have AdSense during the experiment, and if it had, I think the content would have not involved modafinil; Google was burned badly years ago by mega-fines for allowing greymarket pharmacies to advertise and I assume it's totally filtered them out, so any served ads would be targeted at other personal characteristics, presumably. (I also vaguely recollect trying to buy some adwords once while experimenting with advertising gwern.net pages, and the drug categories being totally out of bounds.) During the experiment, roughly, the most popular pages were a mix: https://www.gwern.net/About#january-2017---july-2017 https://www.gwern.net/About#january-2017---july-2017 The top 5 were modafinil (excluded from AdSense), spaced repetition (psychology), LSD microdosing, DNB FAQ (also psychology), and "Story Of Your Life" (physics/literary criticism).