4 ms·

Reading the article, and realizing that these are probably the best arguments against A/B tests that designers can offer, has actually made me change my opinion

Reading the article, and realizing that these are probably the best arguments against A/B tests that designers can offer, has actually made me change my opinion in favor of A/B testing. I'll try to go for a point-by-point reply: > A/B testing may only be as effective as the designs being tested True, but taken on face value, these words mean that A/B testing becomes more valuable when you have great designs made by great designers. Presumably not what the author wanted to say. > As is true with any usability test, you gotta question the motives behind the participants’ answers/reactions. True, but how does the designer know in advance that "questioning the motives" and "applying healthy skepticism" will yield conclusions favorable to the designer? > A/B testing is typically used for tightly focused comparisons of granular elements of an interface, resulting in poor pastiches with results drawn from different tests. This point is correct, as far as I can see. It's actually the strongest argument against A/B tests that I know. > Would you A/B test the Wii or Dyson or Prius or iPhone? Against what? This point doesn't have any bearing on whether you should A/B test websites. > A/B testing locks you into just two comparative options, an exclusively binary (and thus limited) way of thinking. This point is wrong. A couple months ago we ran an A/B test with four alternatives. > A designer (and team) should have an opinion at the table and be willing to defend it, not simply cave into a simplistic math test for interfaces. This point only asserts what the author is trying to prove.

jcc80 15y ago
Yes, this post makes a lot of assumptions. I don't understand why I would question the motives of people who just want to use my site/product to complete tasks as simply as possible. And binary thinking? Just because I do one A/B test doesn't mean the matter is settled. The "winner" of the first test can be tested against other options, whether it's right away or 6 months down the road. It goes on and on & hopefully we get closer and closer to "perfect".
ahoyhere 15y ago
"Reading the article, and realizing that these are probably the best arguments against A/B tests that designers can offer..." What makes you say that? Did you split-test them? (Serious question.)
- cousin_it 15y ago
  Serious answer: because I haven't seen any designer offer better arguments against A/B testing, though they often write about disliking it. If someone holds an opinion but doesn't articulate the reasons for holding it, I'm willing to give them the benefit of the doubt. But if they try to articulate the reasons and the result is emotional and full of fallacies, I start to think that it's just rationalization and the actual reason is something else, perhaps some opaque emotional reaction that they're trying to justify instead of examining it.
  - ahoyhere 15y ago
    If you read a lot, and you probably do, you know that the vast majority of people can't express their way out of a paper box. (Especially if you read stuff written by "amateurs," such as almost all of the internet.) That doesn't mean that what they are saying is incorrect, just that they are very bad at explaining it. Ex: Most people "believe" in seasons, and evolution. But you ever try to get them to explain what makes those things WORK?

jcc80 15y ago

Yes, this post makes a lot of assumptions. I don't understand why I would question the motives of people who just want to use my site/product to complete tasks as simply as possible. And binary thinking? Just because I do one A/B test doesn't mean the matter is settled. The "winner" of the first test can be tested against other options, whether it's right away or 6 months down the road. It goes on and on & hopefully we get closer and closer to "perfect".

ahoyhere 15y ago

"Reading the article, and realizing that these are probably the best arguments against A/B tests that designers can offer..." What makes you say that? Did you split-test them? (Serious question.)

cousin_it 15y ago
Serious answer: because I haven't seen any designer offer better arguments against A/B testing, though they often write about disliking it. If someone holds an opinion but doesn't articulate the reasons for holding it, I'm willing to give them the benefit of the doubt. But if they try to articulate the reasons and the result is emotional and full of fallacies, I start to think that it's just rationalization and the actual reason is something else, perhaps some opaque emotional reaction that they're trying to justify instead of examining it.
- ahoyhere 15y ago
  If you read a lot, and you probably do, you know that the vast majority of people can't express their way out of a paper box. (Especially if you read stuff written by "amateurs," such as almost all of the internet.) That doesn't mean that what they are saying is incorrect, just that they are very bad at explaining it. Ex: Most people "believe" in seasons, and evolution. But you ever try to get them to explain what makes those things WORK?