4 ms·

> You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't

by llambda 10y ago

> You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't enough for a test like this. Is this actually true? Shouldn't the fact you're choosing your pages from a very large set of pages negate this concern? Yes, you might have outliers, but your method of deriving significance should account for this. Given that we don't know the specifics of the tests (this is a case study after all and we should expect details to be elided), I'm not sure it's fair to opine as you are on their quality. I would guess a company focused on this specific problem is both aware of and actively applying all the best practices you've mentioned.

CJefferson 10y ago
The problem is, for example, one site on my site gets over 50% of all traffic. No random selection technique will solve that problem, unless I don't include that page.
birken 10y ago
I can't speak for every website, but it is my experience that traffic to diverse sets of landing pages follows much more of a power law distribution than anything close to constant. This was true at Thumbtack (where we did these types of tests regularly), and this is true of basically all of the SEO sites I run. For example, one of my sites (champsorchumps.us) has a landing page for every pro sports team. Here is a graph of the traffic to all of those landing pages in the past month I just pulled from GA: http://imgur.com/cB2igLc http://imgur.com/cB2igLc Note: The content on each landing page is basically the same (of course with different data for different sports teams), but as you can see the traffic to each page is vastly different. The top page gets nearly 25% of the traffic. Most of the pages get little or no traffic. If you consider the top page, traffic on that particular page goes up and down randomly all the time. If I have that page in a bucket with other pages for an SEO title test, and its traffic happens to randomly go up by 50%, the variance from that page alone might be equal to a naive "significant result" for the whole test. You can still run a title A/B test (something I do on basically every site I run), but you have to be thoughtful about it. You have to consider the buckets carefully and consider what gains would be significant before you run it. I'm not suggesting the authors of the post didn't think about it. Maybe they did. However I've talked to a lot of people about A/B testing and most of them don't. The problem is that all of these A/B testing posts always yada yada over the important parts of running A/B tests, proper setup and impartial analysis, and shoot straight to whatever variations they used and talk about the huge gains. So when people read them they think all they need to do is come up with some fun variations and boom, they are going to get huge gains. It just isn't true. If you aren't careful with A/B tests you can just as easily move backwards than forwards if you skip over the important parts of creating a test.
- blahi 10y ago
  They use counterfactuals, so they are comparing apples to apples. There are other ways to do it. One of the easiest, yet most effective way is matching. You can match your observations with various statistical models, k-means would do a very good job here, and then run a regression. Compare parameters and you are done. It's not a rocket science.

by llambda 10y ago

CJefferson 10y ago

The problem is, for example, one site on my site gets over 50% of all traffic. No random selection technique will solve that problem, unless I don't include that page.

birken 10y ago

I can't speak for every website, but it is my experience that traffic to diverse sets of landing pages follows much more of a power law distribution than anything close to constant. This was true at Thumbtack (where we did these types of tests regularly), and this is true of basically all of the SEO sites I run. For example, one of my sites (champsorchumps.us) has a landing page for every pro sports team. Here is a graph of the traffic to all of those landing pages in the past month I just pulled from GA: http://imgur.com/cB2igLc http://imgur.com/cB2igLc Note: The content on each landing page is basically the same (of course with different data for different sports teams), but as you can see the traffic to each page is vastly different. The top page gets nearly 25% of the traffic. Most of the pages get little or no traffic. If you consider the top page, traffic on that particular page goes up and down randomly all the time. If I have that page in a bucket with other pages for an SEO title test, and its traffic happens to randomly go up by 50%, the variance from that page alone might be equal to a naive "significant result" for the whole test. You can still run a title A/B test (something I do on basically every site I run), but you have to be thoughtful about it. You have to consider the buckets carefully and consider what gains would be significant before you run it. I'm not suggesting the authors of the post didn't think about it. Maybe they did. However I've talked to a lot of people about A/B testing and most of them don't. The problem is that all of these A/B testing posts always yada yada over the important parts of running A/B tests, proper setup and impartial analysis, and shoot straight to whatever variations they used and talk about the huge gains. So when people read them they think all they need to do is come up with some fun variations and boom, they are going to get huge gains. It just isn't true. If you aren't careful with A/B tests you can just as easily move backwards than forwards if you skip over the important parts of creating a test.

blahi 10y ago
They use counterfactuals, so they are comparing apples to apples. There are other ways to do it. One of the easiest, yet most effective way is matching. You can match your observations with various statistical models, k-means would do a very good job here, and then run a regression. Compare parameters and you are done. It's not a rocket science.