5 ms·

A/B testing is high on hype and promise but low on actual results if you follow it though to actual metrics. I've done various forms of A/B throughout most of m

• Aleman360 11y ago
  > A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form. But but but telemetry is evil!
  • ross-life 11y ago
    Telemetry in a specific product? Fine. Telemetry in my OS* that has access to everything? No. * That I cannot turn off.
    • quanticle 11y ago
      What if the OS is the product?
      • ross-life 11y ago
        An OS is a product of course, but to me it's on a completely different level to "just" an application product. It has access to everything, I want to be able to trust it and know exactly what it's doing. Telemetry I can't turn off ruins that. Having it is fine, just let me turn it off... I like Windows and really like VS, I was often literally the only (non-VM) Windows user in a sea of OS X at several offices but Windows 10, the flip-flopping UI, the ads, and the telemetry pushed me to Linux and building my own desktops.
    • Aleman360 11y ago
      A modern OS is nothing but a bunch of apps and the kernel. Most of the apps run in sandboxes.
    • frik 11y ago
      Exactly. This is why Win10 is so bad. You simply cannot deactivate the telemetry at all (as an end user)!
  • walterbell 11y ago
    Instrumentation can be activated with customer consent, if support is requested by the customer.
• cle 11y ago
  One underused but valuable use of A/B testing is to safely roll out new features--even if you aren't really interested in optimization. I've seen many cases where A/B testing has caught bugs that would have cost the company a significant amount of money and customer trust. The overall utility of A/B tests is a complex topic and largely depends on the skill of the experimenter. Well-designed experiments are extraordinarily useful. A/B tests are one tool for product development, but they can't replace it.
  • jacalata 11y ago
    I wouldn't call that A/B testing. It looks pretty similar under the hood but you'd normally call that a phased or gradual rollout and have different goals and metrics for it.
    • ampersandy 11y ago
      As you said, what cle described is a gradual rollout. You can do both at the same time though -- simply start your test on X% of the population and then roll it out to more people as you run your test. This is also a great way to run long-term holdouts.
• michaelcampbell 11y ago
  Perhaps an A/B test on A/B testing vs. not is in order.
• kevan 11y ago
  At small scales it's probably better to spend your time working on the core product than to make small tweaks, but once you're operating at a large scale a 0.1% conversion bump from a small change (like making the cents display in the price smaller) can mean millions of dollars in increased conversion.
• enraged_camel 11y ago
  >>A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form. In my experience this only works for obvious, glaring flaws, such as "I hate that the phone number field enforces a specific format" or "Why the hell are your password requirements so stringent?" But a lot of UX is about what happens on the unconscious level. A user will never tell you, "Your bright red background feels too aggressive/hostile" because they simply don't recognize it at the conscious level to be able to verbalize it. The signup or checkout form might make them feel uneasy, but they won't know why. I find that the best way to optimize websites is to do A/B testing first, and then talk to your users about the subset of the results (which might be the majority) that you find confusing or surprising. Basically, never rely on just one method, because then you will be hamstrung by the shortcomings of that method.
• sweezyjeezy 11y ago
  Disclaimer : I work for an A/B testing platform. There is nothing inherently wrong with A/B testing, and it's basically the only way to verify that anything you do has a positive impact. What's created a hype bubble is articles and platforms saying that minor UI changes, colour of buttons etc. make a real difference to conversion rates. Good tests based on well thought out hypotheses relevant to your actual customers, can and sometimes do help websites find measurable gains. It is not a silver bullet, and no matter how careful you are there is always statistical uncertainty, it won't always help.
• frozenport 11y ago
  You described something with small benefits but limited downside, especially if your company has a team of a dozen or so people.
• matwood 11y ago
  A/B (or champion/challenger as we called it in the 90s) testing is fine. The problem with these tools is that they do not alleviate the need to understand statistics. A common place where people get stuck when A/B testing is in a local minimum or maximum. For example, instead of tweaking button colors someone should instead redesign the whole form. Talking to users can help with this, but users are notoriously bad at not knowing what they really want.
  • TheLogothete 11y ago
    The most common place people get stuck in a/b testing is proper design of experiments, i.e. absolutely no methodology (i.e. proper blocking) at all. It's ridiculous how a/b testing is perpetuated to be of paramount importance. A/B testing would be one of the last things I would do, if I worked at a small ecommerce company, for example. What lead to the current state is venture-backed companies and "growth hackers" and "conversion optimization experts" selling false promises to people who haven't read a marketing textbook in their life.
• ryporter 11y ago
  A/B Testing is a way to conduct an experiment. Instrumentation and talking to users is another good way to gain insights, but it not an experiment. They are two different (and often complementary) activities. Many, many people have successfully used A/B Testing. I've personally used it to great effect several times. I certainly don't make decisions purely based on the statistical results, but I find it to be an extremely useful input to the decision making process. All models are flawed; some are useful.
• arihant 11y ago
  There is a big difference between conducting an experiment and understanding the results. Even talking to users would not help much if one continues to build a faster horse.
• jrpt 11y ago
  How is this the top comment? It's recommending changing your website without knowing what the effect is, which is terrible advice. I've personally run tests and seen tests achieve statistically significant (and very large) results that weren't merely fixing bugs or moving buttons within reach of a user. It could be things like adding a modal, changing copy (yes, literally just adding or changing words on the page), removing content, changing an image, or reordering funnel steps. Think about that: by merely changing some sentences, your website could be 10% more successful next month. But you won't know whether it's working without a/b testing. You don't just create tests randomly, but based off of an understanding of the product, goals, signup funnel, etc. which may involve talking to users, but ultimately verifying the change is for the better through a/b testing. I would agree that there are a number of people running a/b tests that don't understand the math or know what they are doing, but that isn't an indictment of a/b testing.
  • bildung 11y ago
    > I've personally run tests and seen tests achieve statistically significant (and very large) results that weren't merely fixing bugs or moving buttons within reach of a user. It could be things like adding a modal, changing copy (yes, literally just adding or changing words on the page), removing content, changing an image, or reordering funnel steps. You should read the article. It explained very well why a) statistical significance is meaningless when split testing websites (you are guaranteed to archive significance if you run the test long enough; the math behind significance testing doesn't work like assumed by Optimizely et.al.) and b) most tools on the market are inherently unable to prove effect size by the way tests are designed. Just do this short experiment yourself: Do an a/b test with identical versions of you site. Spoiler: With 100% certainty, you'll see both significant results and a sizable effect size difference.
    • cbr 11y ago
      statistical significance is meaningless when split testing websites (you are guaranteed to archive significance if you run the test long enough Wait, what? Not if you use the right test to evaluate your change. This article isn't showing that A/B testing is bunk, it's showing that Optimizely is using an inappropriate test. Do an a/b test with identical versions of you site. Spoiler: With 100% certainty, you'll see both significant results and a sizable effect size difference. The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical. Running your A vs A experiment and evaluating it correctly you'll find results significant at the p < N rate N% of the time. And the longer you run it the lower your estimate of the effect size will be.
• rubberstamp 11y ago
  Each page could contain a small feedback area in which the actual product user can specify what feature he/she would like to see improved. Changing color of text is not much improvement, unless the text was unreadable or same color as background. No amount of data driven can replace what the actual product user says.
  • madeofpalk 11y ago
    The problem is that actual product users would probably say they want faster horses.
• simonswords82 11y ago
  I'm very surprised this is the top comment too, where's patio11 and his ridiculously large amount of positive and useful split testing examples when you need him? Anyway, on to your comment: > A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form. I don't know what this means. Can you elaborate? What exactly is "significant instrumentation"? > That, or actually build a product that users want instead of chasing after pointless metrics. Pointless metrics? I'm not sure what metrics you measure but visitor engagement and visitor to trial conversion rates are the lifeblood of a software sales and marketing funnel. When somebody in marketing considers A/B testing as a means to increasing trial and paying customer conversions, they're operating on the basis that their product already provides value. The A/B tests are a means of establishing the best method for communicating that value. Better communication = more trials. More trials = a larger pool of potential customers to convert to paying customers. It's pretty basic. > Most of the A/B website optimization industry is an elaborate scam What?? Had you said a number of people in marketing could be in danger of blindly relying on their tools (e.g. Optimizely) and not applying common sense or checking the maths I would have wholeheartedly agreed. In my personal experience with the various websites I market, and given the hundreds of split testing examples I've read from people I do not feel are in on the "scam", I've got to strongly disagree with this point.