4 ms·

Increase Signups by Adding Video to Your Page

27 points by dcancel 17y ago

yannis 17y ago
This is especially interesting for me (because normally I am put off by video), do you think is because the video highlighted the installation procedure better or do you think generally people prefer video?
dcancel 17y ago
I think it was because the video helped the user understand the product better. All about education.
josefresco 17y ago
Is the form for trials or paid customers? Seems the form is for trials which effects only the # of trials which was only marginally higher. What is the process to convert from trials to paid and is the video really the cause in the massive jump? There were also other small subtle changes like the addition of "Request Instant Price Quote" headline etc.
- dmix 17y ago
  Those small changes can be significant. A/B tests need to be as specific to one adjustment as possible. I've noticed flaws like this many times on ABtests.com. But I guess they are like any war stories, its the overall idea that counts not the details.
  - dcancel 17y ago
    Appreciate the feedback. Would love to make our site better, any suggestions would be very much appreciated. What are the primary flaws you're seeing?
    - dmix 17y ago
      I was referring to the accuracy of the test results on abtests.com in general not the site in the link. Many of the A/B examples had multiple changes to the site but only noted one as making a difference. In your case adding a headline to the form was overlooked. You were using form submissions as a metric from what I can tell. So changes specifically to the signup form itself (such as the header) is likely to have as much of an impact as the video. Did you track how many people watched the video then signed up as a metric? That may be more telling.
harrybr 17y ago
The heading is rather misleading. It actually should say "Some site somewhere increased sign-ups by adding video to their page. You may or may not get the same effect, depending on a vast myriad of factors." But I guess that's not very catchy is it.
- brown9-2 17y ago
  I seem to remember a very similar posting from abtests.com last week. I would hope that people don't expect that you can magically copy the results of someone else's A/B test to your own domain.
  - briancooley 17y ago
    I am sure they do. Ever since the "You should follow me on twitter" post (http://www.abtests.com/test/19013/other-for-dustin-curtis http://www.abtests.com/test/19013/other-for-dustin-curtis), I see that phrase everywhere. It could just be that I am noticing the phrase since I read about it, but it would be a strange coincidence to see the exact wording so often.
    - charliepark 17y ago
      I believe the guys at ABTests.com would encourage readers to consider the results on the site, to evaluate whether or not running a similar test would make sense for their own site, to then run a test, and to then post the results at ABTests.com. Everyone would benefit from the same test running on multiple sites, to see whether the results are universal or not. I'm sure they don't think that a featured test would result in identical results on a different site. But opening up data like this for peer review and replication is a good thing. Edit: Oh. dcancel said as much, just above this.
  - dcancel 17y ago
    You are absolutely right. You cannot expect to be able to replicate someone else results. Results are very dependent to your business and context. We at ABTests.com are simply trying to get more people to incorporate A/B Testing into their development process. Thanks for the feedback, we appreciate it.
aaronsw 17y ago
The Obama people, in a much, much bigger test, found that having a video on their splash page decreased signups.
- carbocation 17y ago
  Is this (esp the embedded video) the reference for your statement? http://websiteoptimizer.blogspot.com/2009/11/new-video-how-we-used-data-to-win.html http://websiteoptimizer.blogspot.com/2009/11/new-video-how-w...
wgj 17y ago
The lesson in A/B testing isn't to find magic bullets in other people's results. It's to actually do A/B testing. If you could generalize from other people's results, you could just catalog what works and be done. It doesn't work that way.
cd34 17y ago
How was the testing done? Every other pageview delivered over a 7 day period had the video? If you tested by running one for 7 days and the other for 7 days, there are numerous other possible factors. Even the comparison of Friday versus Saturday traffic could have enough variation on a small sample to skew results. Internet congestion might have slowed the site down during one test, awareness might predispose people to be more likely to convert beyond trial or even the fact that the video version renders more quickly to the surfer proving that html design and page render time may have been the deciding factor. I didn't see any mention on the site how the trial was actually conducted, but, based on the other test results, I don't see enough isolation of variables to decidedly say the video was the sole factor.

27 points by dcancel 17y ago

This is especially interesting for me (because normally I am put off by video), do you think is because the video highlighted the installation procedure better or do you think generally people prefer video?

dcancel 17y ago

I think it was because the video helped the user understand the product better. All about education.

josefresco 17y ago

Is the form for trials or paid customers? Seems the form is for trials which effects only the # of trials which was only marginally higher. What is the process to convert from trials to paid and is the video really the cause in the massive jump? There were also other small subtle changes like the addition of "Request Instant Price Quote" headline etc.

dmix 17y ago
Those small changes can be significant. A/B tests need to be as specific to one adjustment as possible. I've noticed flaws like this many times on ABtests.com. But I guess they are like any war stories, its the overall idea that counts not the details.
- dcancel 17y ago
  Appreciate the feedback. Would love to make our site better, any suggestions would be very much appreciated. What are the primary flaws you're seeing?
  - dmix 17y ago
    I was referring to the accuracy of the test results on abtests.com in general not the site in the link. Many of the A/B examples had multiple changes to the site but only noted one as making a difference. In your case adding a headline to the form was overlooked. You were using form submissions as a metric from what I can tell. So changes specifically to the signup form itself (such as the header) is likely to have as much of an impact as the video. Did you track how many people watched the video then signed up as a metric? That may be more telling.

harrybr 17y ago

The heading is rather misleading. It actually should say "Some site somewhere increased sign-ups by adding video to their page. You may or may not get the same effect, depending on a vast myriad of factors." But I guess that's not very catchy is it.

brown9-2 17y ago
I seem to remember a very similar posting from abtests.com last week. I would hope that people don't expect that you can magically copy the results of someone else's A/B test to your own domain.
- briancooley 17y ago
  I am sure they do. Ever since the "You should follow me on twitter" post (http://www.abtests.com/test/19013/other-for-dustin-curtis http://www.abtests.com/test/19013/other-for-dustin-curtis), I see that phrase everywhere. It could just be that I am noticing the phrase since I read about it, but it would be a strange coincidence to see the exact wording so often.
  - charliepark 17y ago
    I believe the guys at ABTests.com would encourage readers to consider the results on the site, to evaluate whether or not running a similar test would make sense for their own site, to then run a test, and to then post the results at ABTests.com. Everyone would benefit from the same test running on multiple sites, to see whether the results are universal or not. I'm sure they don't think that a featured test would result in identical results on a different site. But opening up data like this for peer review and replication is a good thing. Edit: Oh. dcancel said as much, just above this.
- dcancel 17y ago
  You are absolutely right. You cannot expect to be able to replicate someone else results. Results are very dependent to your business and context. We at ABTests.com are simply trying to get more people to incorporate A/B Testing into their development process. Thanks for the feedback, we appreciate it.

aaronsw 17y ago

The Obama people, in a much, much bigger test, found that having a video on their splash page decreased signups.

carbocation 17y ago
Is this (esp the embedded video) the reference for your statement? http://websiteoptimizer.blogspot.com/2009/11/new-video-how-we-used-data-to-win.html http://websiteoptimizer.blogspot.com/2009/11/new-video-how-w...

wgj 17y ago

The lesson in A/B testing isn't to find magic bullets in other people's results. It's to actually do A/B testing. If you could generalize from other people's results, you could just catalog what works and be done. It doesn't work that way.

cd34 17y ago

How was the testing done? Every other pageview delivered over a 7 day period had the video? If you tested by running one for 7 days and the other for 7 days, there are numerous other possible factors. Even the comparison of Friday versus Saturday traffic could have enough variation on a small sample to skew results. Internet congestion might have slowed the site down during one test, awareness might predispose people to be more likely to convert beyond trial or even the fact that the video version renders more quickly to the surfer proving that html design and page render time may have been the deciding factor. I didn't see any mention on the site how the trial was actually conducted, but, based on the other test results, I don't see enough isolation of variables to decidedly say the video was the sole factor.