4 ms·
After reviewing the individual heatmaps, however, we get the impression that the site might have a bit too much going on Normal A/B split tests have a similar
by petercooper 15y ago
After reviewing the individual heatmaps, however, we get the impression that the site might have a bit too much going on
Normal A/B split tests have a similar issue. Let's say you have a sales page and you're testing colors. Red buy button, a blue button, a green button. And let's say blue wins overall with a high level of confidence.
Without drilling down (and most systems I've worked with are bad at this or can't do it at all) you can't tell if certain types of user actually preferred other scenarios. For example, visitors from east Asia might convert WAY better for the red button (red being a lucky color in China). And visitors coming from certain sites might always convert better on the green.
Problem is, your split test results show the overall results and so you optimize with the blue.. when your A/B tool could have analyzed the data and suggested segments for you. Does this tool exist (without getting users to guess segments up front)? If not, there's a ton of money being left on the table.
- mwexler 15y agoActually, this is a problem with most entry level tools. Professional level tools like Omniture Test and Target or Optimost with Autonomy Segments handle MVT testing per segment, segment defined by external variables (appended data or data from Dataxu et al), business variables (return customer, prospect, etc.), or just online variables (IP address Geo, repeat vs. new visitor, etc.) Professional level, btw, is defined as "costs a lot" and "are painful to use" and "have poor interfaces"; more recent tools give up some of the power and cost for ease of use. However, as you point out, these easier MVT tools (or Split-Test tools) focus on showing the "winner" or "winning variables", winner defined as "most impactful on the total traffic or sample of this traffic during this time". You are right that this is not the final answer... but it's a good starting place; if you solve for the majority of users, it's usually the biggest step. Then solving for unique smaller groups can often result in lower volume but higher value conversions... if you have time. That being said, including segment variables in the analysis is a wonderful thing to do, and I encourage it. I look forward to when Visual Website Optimizer and other "entry level" tools include these in their analyses by default... and we can look to the end of Optimost, Omniture TnT, and the other "enterprise" tools.
- DanielRibeiro 15y agoSeems to me like you are talking about Cohort Analysis: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html But you are right, a tool for this would be nice.