5 ms·

How Coderwall grew over 50% with an A/B test

103 points by ryanb 10y ago

ssharp 10y ago
I don't really understand the methodology on this test. How are you getting a fair control/variant split in Google SERPs?
- zck 10y ago
  They took 20,000 pages, split them into test and control groups, and changed the title of questions in the test group from "$QUESTION - Coderwall" to "$QUESTION (Example) - Coderwall". They then measured how many more clicks the test group got compared to the control group.
eonw 10y ago
not a very useful or informative article. correlation is not causation and many things could make you move up the SERPS in addition to a title change.
- ryanb 10y ago
  There wasn't a sudden increase in rankings - this test showed CTR from Google increased dramatically with the change. When it was rolled out to the entire site, clicks went up, and over many weeks impressions/rankings followed.
  - eonw 10y ago
    but the algorithm also changes, so that could have also been part of it(or your competition made changes), if you dont control all the variables its flawed, IMO. Downvote all you want. I've seen 1000%+ increase without changes, so by that math, could I say that not changing anything at all can increase your CTR by that percentile?
wdewind 10y ago
> The number of users clicking on Coderwall from Google increased 14.8% (yes, this was statistically significant). Would be very curious how they calculated significance here.
- ryanb 10y ago
  Inferring causal impact using Bayesian structural time-series models: http://research.google.com/pubs/pub41854.html http://research.google.com/pubs/pub41854.html
  - wdewind 10y ago
    Thanks, but I'm not smart enough to digest this haha. If you have time could you break it down a little? My main question, I guess, is how do you calculate P-value?
- deleted 10y ago
  [deleted]
- samscully 10y ago
  You can calculate the significance using the difference in differences method popular in econometrics. I derived this result for a previous employer and it worked well. I can look up my notes and explain the method in more detail if anyone is interested.
  - asbromberg 10y ago
    I would definitely be interested in hearing about this if you have the time.
  - wojcech 10y ago
    I'd be very interested if you don't mind:)
  - wdewind 10y ago
    Yes please
JabavuAdams 10y ago
I don't understand the "delta between test and control" figure. Can anyone explain?
- jdpigeon 10y ago
  I'm confused as well. Pretty poor figure design without axis labels
birken 10y ago
This is a very good basic A/B test, and while it is certainly way better than nothing and something that a lot of websites should try, I think there are some important caveats that I'm thinking of that aren't mentioned in the article: 1. How were the experiment groups chosen? You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't enough for a test like this. It is only meaningful if you split up 20,000 pages that had similar traffic profiles and were getting enough aggregate traffic to be able to notice an increase. 2. SEO tests take a long time so if you are doing it, I'd recommend more than 2 variations. You need to put the test up, wait for Google to index it, then wait a few weeks to see how the traffic changes. So since your turn-around time is at least a few weeks and maybe longer, try 4 or 8 variations if your traffic can support it. 3. I prefer my A/B tests to be a little more crazy, or at least try a crazy variation among more normal ones. In my experience the biggest gainers (and also the biggest losers) are ideas that seem crazy. Getting your feet wet with adding "(Example)" to the title is fine, but also try a crazy variation that is completely different and seeing how that does. Give yourself a chance to be surprised by your audience so you can learn more about them. And if the crazy variation loses by a lot, you have learned something important even if it isn't a winner. 4. The facts are a little dubious here. Traffic is up 50% over a timespan... there is no control. Traffic generally goes up for growing websites even if you don't do anything. You should also show your work on the 14.8% increase and give us some error bars. You say it is significant, how did you calculate this? It does seem like the test was better, but it is also important to make valid claims about it. Intellectual honesty when running A/B tests is really really important [1]. 1: http://danbirken.com/ab/testing/2014/04/08/pitfalls-of-bias-in-ab-testing.html http://danbirken.com/ab/testing/2014/04/08/pitfalls-of-bias-...
- dillonforrest 10y ago
  Author and RankScience cofounder here. Thanks for adding this great comment! I agree with all points here. We left out many details as we decided this amount of rigor was less exciting and valuable to the average target reader, but you are clearly cuts above! Would love to chat more about your experience with a/b testing for top of funnel and hear more about your work. Can you email me if you'd like to meet up for coffee in SF? dillon@ranksci.com
samfisher83 10y ago
But why did people click more often when (Example) was put in there?
- ryanb 10y ago
  Good question -- we think it's because people searching for programming related help are lost and want clear, simple answers to their problems. Which one of these results would you click on? (query: mysql split string) http://i.imgur.com/CWxs0dK.png http://i.imgur.com/CWxs0dK.png
ourmandave 10y ago
I love a good click-bait title. What exactly grew and how much is 50%? Are you getting 15 hits per month, up from 10?
voycey 10y ago
Or they realised that coders love to copy and paste ;)

103 points by ryanb 10y ago

I don't really understand the methodology on this test. How are you getting a fair control/variant split in Google SERPs?

zck 10y ago
They took 20,000 pages, split them into test and control groups, and changed the title of questions in the test group from "$QUESTION - Coderwall" to "$QUESTION (Example) - Coderwall". They then measured how many more clicks the test group got compared to the control group.

eonw 10y ago

not a very useful or informative article. correlation is not causation and many things could make you move up the SERPS in addition to a title change.

ryanb 10y ago
There wasn't a sudden increase in rankings - this test showed CTR from Google increased dramatically with the change. When it was rolled out to the entire site, clicks went up, and over many weeks impressions/rankings followed.
- eonw 10y ago
  but the algorithm also changes, so that could have also been part of it(or your competition made changes), if you dont control all the variables its flawed, IMO. Downvote all you want. I've seen 1000%+ increase without changes, so by that math, could I say that not changing anything at all can increase your CTR by that percentile?

wdewind 10y ago

> The number of users clicking on Coderwall from Google increased 14.8% (yes, this was statistically significant). Would be very curious how they calculated significance here.

ryanb 10y ago
Inferring causal impact using Bayesian structural time-series models: http://research.google.com/pubs/pub41854.html http://research.google.com/pubs/pub41854.html
- wdewind 10y ago
  Thanks, but I'm not smart enough to digest this haha. If you have time could you break it down a little? My main question, I guess, is how do you calculate P-value?
deleted 10y ago
[deleted]
samscully 10y ago
You can calculate the significance using the difference in differences method popular in econometrics. I derived this result for a previous employer and it worked well. I can look up my notes and explain the method in more detail if anyone is interested.
- asbromberg 10y ago
  I would definitely be interested in hearing about this if you have the time.
- wojcech 10y ago
  I'd be very interested if you don't mind:)
- wdewind 10y ago
  Yes please

JabavuAdams 10y ago

I don't understand the "delta between test and control" figure. Can anyone explain?

jdpigeon 10y ago
I'm confused as well. Pretty poor figure design without axis labels

birken 10y ago

This is a very good basic A/B test, and while it is certainly way better than nothing and something that a lot of websites should try, I think there are some important caveats that I'm thinking of that aren't mentioned in the article: 1. How were the experiment groups chosen? You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't enough for a test like this. It is only meaningful if you split up 20,000 pages that had similar traffic profiles and were getting enough aggregate traffic to be able to notice an increase. 2. SEO tests take a long time so if you are doing it, I'd recommend more than 2 variations. You need to put the test up, wait for Google to index it, then wait a few weeks to see how the traffic changes. So since your turn-around time is at least a few weeks and maybe longer, try 4 or 8 variations if your traffic can support it. 3. I prefer my A/B tests to be a little more crazy, or at least try a crazy variation among more normal ones. In my experience the biggest gainers (and also the biggest losers) are ideas that seem crazy. Getting your feet wet with adding "(Example)" to the title is fine, but also try a crazy variation that is completely different and seeing how that does. Give yourself a chance to be surprised by your audience so you can learn more about them. And if the crazy variation loses by a lot, you have learned something important even if it isn't a winner. 4. The facts are a little dubious here. Traffic is up 50% over a timespan... there is no control. Traffic generally goes up for growing websites even if you don't do anything. You should also show your work on the 14.8% increase and give us some error bars. You say it is significant, how did you calculate this? It does seem like the test was better, but it is also important to make valid claims about it. Intellectual honesty when running A/B tests is really really important [1]. 1: http://danbirken.com/ab/testing/2014/04/08/pitfalls-of-bias-in-ab-testing.html http://danbirken.com/ab/testing/2014/04/08/pitfalls-of-bias-...

dillonforrest 10y ago
Author and RankScience cofounder here. Thanks for adding this great comment! I agree with all points here. We left out many details as we decided this amount of rigor was less exciting and valuable to the average target reader, but you are clearly cuts above! Would love to chat more about your experience with a/b testing for top of funnel and hear more about your work. Can you email me if you'd like to meet up for coffee in SF? dillon@ranksci.com

samfisher83 10y ago

But why did people click more often when (Example) was put in there?

ryanb 10y ago
Good question -- we think it's because people searching for programming related help are lost and want clear, simple answers to their problems. Which one of these results would you click on? (query: mysql split string) http://i.imgur.com/CWxs0dK.png http://i.imgur.com/CWxs0dK.png

ourmandave 10y ago

I love a good click-bait title. What exactly grew and how much is 50%? Are you getting 15 hits per month, up from 10?

voycey 10y ago

Or they realised that coders love to copy and paste ;)