4 ms·

It's funny, but I've noticed that all of my clients in this space are just outright opposed to a/b testing. Most don't even bother to leverage analytics at all

It's funny, but I've noticed that all of my clients in this space are just outright opposed to a/b testing. Most don't even bother to leverage analytics at all. I think more than anything they see the level of effort to ROI ratio too high. For example, a startup not testing small changes right? Well unfortunately, an entire landing page makeover, while not overly difficult can windup taking a great deal of time. The redesign alone is just such an "opinionated" process (even by people who know nothing about design...) that throwing that much investment just to test is hard to justify. So even though most startups shouldn't waste time on stuff like button colors...that's often the only thing they'd have time for. Of course I'm talking about the smaller scrappy startups. Not the ones backed by prolific amounts of capital. Don't get me wrong. This is a GREAT article, I just would like to know the best level of effort for ab testing for the smaller guys.

kylerush 12y ago
A/b testing doesn't have to mean extra work. If the small startup is going to redesign their homepage anyway (which a lot of them do), they might as well a/b test it.
- colemorrison 12y ago
  I'm not really sure what you mean by this? In my experience redesigns occur when new features are added, major updates, or changes to the team (whom chime that something should change). Yes likely everyone will redesign at some point, but to do so just for a test? Based on this article, only larger changes should be done by smaller startups. But smaller startups have less resources to throw at larger changes. I mean, what of we're not planning on redesigning our web page?
  - kylerush 12y ago
    The point of the article is to help small startups understand that a/b testing is not off the table. Of course startups will have to decide where the highest ROI is themselves. Given a point in time, the highest ROI may not be a/b testing.
  - marcosdumay 12y ago
    You don't do things for tests. You do the tests because you want to change something. That said, the kind of results startups could get by tests are usually obvious at the balance sheet (or some other big metric) anyway, without any kind of formalization. If you have one of the rare situations where it isn't, go ahead and do a formal test, but I wouldn't recommend anybody to lose some sleep over it.
jrpt 12y ago
Here's two super easy test ideas that can produce big wins. 1. Change text, specifically around your value proposition. Emphasize different things and see what happens. 2. Remove content. Make the page more focused on what's important. Here's an example that increased account creation by 43% at Optimizely: http://blog.optimizely.com/2012/10/09/optimizelys-100000th-experiment/ http://blog.optimizely.com/2012/10/09/optimizelys-100000th-e...

kylerush 12y ago

A/b testing doesn't have to mean extra work. If the small startup is going to redesign their homepage anyway (which a lot of them do), they might as well a/b test it.

colemorrison 12y ago
I'm not really sure what you mean by this? In my experience redesigns occur when new features are added, major updates, or changes to the team (whom chime that something should change). Yes likely everyone will redesign at some point, but to do so just for a test? Based on this article, only larger changes should be done by smaller startups. But smaller startups have less resources to throw at larger changes. I mean, what of we're not planning on redesigning our web page?
- kylerush 12y ago
  The point of the article is to help small startups understand that a/b testing is not off the table. Of course startups will have to decide where the highest ROI is themselves. Given a point in time, the highest ROI may not be a/b testing.
- marcosdumay 12y ago
  You don't do things for tests. You do the tests because you want to change something. That said, the kind of results startups could get by tests are usually obvious at the balance sheet (or some other big metric) anyway, without any kind of formalization. If you have one of the rare situations where it isn't, go ahead and do a formal test, but I wouldn't recommend anybody to lose some sleep over it.

jrpt 12y ago

Here's two super easy test ideas that can produce big wins. 1. Change text, specifically around your value proposition. Emphasize different things and see what happens. 2. Remove content. Make the page more focused on what's important. Here's an example that increased account creation by 43% at Optimizely: http://blog.optimizely.com/2012/10/09/optimizelys-100000th-experiment/ http://blog.optimizely.com/2012/10/09/optimizelys-100000th-e...