9 ms·

This is a very good basic A/B test, and while it is certainly way better than nothing and something that a lot of websites should try, I think there are some im

by birken 10y ago

This is a very good basic A/B test, and while it is certainly way better than nothing and something that a lot of websites should try, I think there are some important caveats that I'm thinking of that aren't mentioned in the article: 1. How were the experiment groups chosen? You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't enough for a test like this. It is only meaningful if you split up 20,000 pages that had similar traffic profiles and were getting enough aggregate traffic to be able to notice an increase. 2. SEO tests take a long time so if you are doing it, I'd recommend more than 2 variations. You need to put the test up, wait for Google to index it, then wait a few weeks to see how the traffic changes. So since your turn-around time is at least a few weeks and maybe longer, try 4 or 8 variations if your traffic can support it. 3. I prefer my A/B tests to be a little more crazy, or at least try a crazy variation among more normal ones. In my experience the biggest gainers (and also the biggest losers) are ideas that seem crazy. Getting your feet wet with adding "(Example)" to the title is fine, but also try a crazy variation that is completely different and seeing how that does. Give yourself a chance to be surprised by your audience so you can learn more about them. And if the crazy variation loses by a lot, you have learned something important even if it isn't a winner. 4. The facts are a little dubious here. Traffic is up 50% over a timespan... there is no control. Traffic generally goes up for growing websites even if you don't do anything. You should also show your work on the 14.8% increase and give us some error bars. You say it is significant, how did you calculate this? It does seem like the test was better, but it is also important to make valid claims about it. Intellectual honesty when running A/B tests is really really important [1]. 1: http://danbirken.com/ab/testing/2014/04/08/pitfalls-of-bias-in-ab-testing.html http://danbirken.com/ab/testing/2014/04/08/pitfalls-of-bias-...

dillonforrest 10y ago
Author and RankScience cofounder here. Thanks for adding this great comment! I agree with all points here. We left out many details as we decided this amount of rigor was less exciting and valuable to the average target reader, but you are clearly cuts above! Would love to chat more about your experience with a/b testing for top of funnel and hear more about your work. Can you email me if you'd like to meet up for coffee in SF? dillon@ranksci.com
llambda 10y ago
> You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't enough for a test like this. Is this actually true? Shouldn't the fact you're choosing your pages from a very large set of pages negate this concern? Yes, you might have outliers, but your method of deriving significance should account for this. Given that we don't know the specifics of the tests (this is a case study after all and we should expect details to be elided), I'm not sure it's fair to opine as you are on their quality. I would guess a company focused on this specific problem is both aware of and actively applying all the best practices you've mentioned.
- CJefferson 10y ago
  The problem is, for example, one site on my site gets over 50% of all traffic. No random selection technique will solve that problem, unless I don't include that page.
- birken 10y ago
  I can't speak for every website, but it is my experience that traffic to diverse sets of landing pages follows much more of a power law distribution than anything close to constant. This was true at Thumbtack (where we did these types of tests regularly), and this is true of basically all of the SEO sites I run. For example, one of my sites (champsorchumps.us) has a landing page for every pro sports team. Here is a graph of the traffic to all of those landing pages in the past month I just pulled from GA: http://imgur.com/cB2igLc http://imgur.com/cB2igLc Note: The content on each landing page is basically the same (of course with different data for different sports teams), but as you can see the traffic to each page is vastly different. The top page gets nearly 25% of the traffic. Most of the pages get little or no traffic. If you consider the top page, traffic on that particular page goes up and down randomly all the time. If I have that page in a bucket with other pages for an SEO title test, and its traffic happens to randomly go up by 50%, the variance from that page alone might be equal to a naive "significant result" for the whole test. You can still run a title A/B test (something I do on basically every site I run), but you have to be thoughtful about it. You have to consider the buckets carefully and consider what gains would be significant before you run it. I'm not suggesting the authors of the post didn't think about it. Maybe they did. However I've talked to a lot of people about A/B testing and most of them don't. The problem is that all of these A/B testing posts always yada yada over the important parts of running A/B tests, proper setup and impartial analysis, and shoot straight to whatever variations they used and talk about the huge gains. So when people read them they think all they need to do is come up with some fun variations and boom, they are going to get huge gains. It just isn't true. If you aren't careful with A/B tests you can just as easily move backwards than forwards if you skip over the important parts of creating a test.
btilly 10y ago
I was not involved in this test, but here is how I would have done this. 1. I would have randomly assigned pages with more than X average conversions to the versions, and then kept a count of how many pages saw an increase or a decrease. 2. The key phrase is, "If your traffic can support it." This is a question of doing statistical calculations in advance and deciding how big a random mistake you'd be willing to accept if you were unlucky. I find that people tend to err on the side of trying to get too much information out of too little data, so I would generally push towards fewer versions, not more. 3. I strongly disagree with you on this. I would prefer to run 5 small tests over one big one because you learn more. In particular I have found many times that titles, subject lines, and the like can have a surprising impact. The area of your site with the most likely big wins from A/B tests is where you have large numbers of unengaged visitors that you are trying to get to make the snap decision to click - which is exactly what this title test measures. 4. They are trying to run a business, not conduct a research product. Once they had a 14% known win on their hands, and enough sample data to be certain of it, there is no business value in continuing to do what you know is less effective. Do as they did. Collect overall figures, say "some fraction of that is our test", and don't try to measure exactly how big a fraction it is. But to those points I would add a fifth. I have found that there is a significant "novelty" effect. For example if you're carrying on an email campaign, people will initially respond well to pretty much any subject line change the first time they see it. But not so much 3 weeks later. I doubt that was an issue here, but it is a problem to always keep in the back of your mind as a question.

by birken 10y ago

dillonforrest 10y ago

Author and RankScience cofounder here. Thanks for adding this great comment! I agree with all points here. We left out many details as we decided this amount of rigor was less exciting and valuable to the average target reader, but you are clearly cuts above! Would love to chat more about your experience with a/b testing for top of funnel and hear more about your work. Can you email me if you'd like to meet up for coffee in SF? dillon@ranksci.com

llambda 10y ago

> You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't enough for a test like this. Is this actually true? Shouldn't the fact you're choosing your pages from a very large set of pages negate this concern? Yes, you might have outliers, but your method of deriving significance should account for this. Given that we don't know the specifics of the tests (this is a case study after all and we should expect details to be elided), I'm not sure it's fair to opine as you are on their quality. I would guess a company focused on this specific problem is both aware of and actively applying all the best practices you've mentioned.

CJefferson 10y ago
The problem is, for example, one site on my site gets over 50% of all traffic. No random selection technique will solve that problem, unless I don't include that page.
birken 10y ago
I can't speak for every website, but it is my experience that traffic to diverse sets of landing pages follows much more of a power law distribution than anything close to constant. This was true at Thumbtack (where we did these types of tests regularly), and this is true of basically all of the SEO sites I run. For example, one of my sites (champsorchumps.us) has a landing page for every pro sports team. Here is a graph of the traffic to all of those landing pages in the past month I just pulled from GA: http://imgur.com/cB2igLc http://imgur.com/cB2igLc Note: The content on each landing page is basically the same (of course with different data for different sports teams), but as you can see the traffic to each page is vastly different. The top page gets nearly 25% of the traffic. Most of the pages get little or no traffic. If you consider the top page, traffic on that particular page goes up and down randomly all the time. If I have that page in a bucket with other pages for an SEO title test, and its traffic happens to randomly go up by 50%, the variance from that page alone might be equal to a naive "significant result" for the whole test. You can still run a title A/B test (something I do on basically every site I run), but you have to be thoughtful about it. You have to consider the buckets carefully and consider what gains would be significant before you run it. I'm not suggesting the authors of the post didn't think about it. Maybe they did. However I've talked to a lot of people about A/B testing and most of them don't. The problem is that all of these A/B testing posts always yada yada over the important parts of running A/B tests, proper setup and impartial analysis, and shoot straight to whatever variations they used and talk about the huge gains. So when people read them they think all they need to do is come up with some fun variations and boom, they are going to get huge gains. It just isn't true. If you aren't careful with A/B tests you can just as easily move backwards than forwards if you skip over the important parts of creating a test.

btilly 10y ago

I was not involved in this test, but here is how I would have done this. 1. I would have randomly assigned pages with more than X average conversions to the versions, and then kept a count of how many pages saw an increase or a decrease. 2. The key phrase is, "If your traffic can support it." This is a question of doing statistical calculations in advance and deciding how big a random mistake you'd be willing to accept if you were unlucky. I find that people tend to err on the side of trying to get too much information out of too little data, so I would generally push towards fewer versions, not more. 3. I strongly disagree with you on this. I would prefer to run 5 small tests over one big one because you learn more. In particular I have found many times that titles, subject lines, and the like can have a surprising impact. The area of your site with the most likely big wins from A/B tests is where you have large numbers of unengaged visitors that you are trying to get to make the snap decision to click - which is exactly what this title test measures. 4. They are trying to run a business, not conduct a research product. Once they had a 14% known win on their hands, and enough sample data to be certain of it, there is no business value in continuing to do what you know is less effective. Do as they did. Collect overall figures, say "some fraction of that is our test", and don't try to measure exactly how big a fraction it is. But to those points I would add a fifth. I have found that there is a significant "novelty" effect. For example if you're carrying on an email campaign, people will initially respond well to pretty much any subject line change the first time they see it. But not so much 3 weeks later. I doubt that was an issue here, but it is a problem to always keep in the back of your mind as a question.