3 ms·
This seems to be a perennial (50-year-long) debate in design / HCI / user-interface circles. The hardcore extremes are a branch that comes out of traditional de
by _delirium 16y ago
This seems to be a perennial (50-year-long) debate in design / HCI / user-interface circles. The hardcore extremes are a branch that comes out of traditional design and architecture, which is radically against anything that reeks of "focus groups"; and the branch that instead comes out of industrial psychology, which won't do anything unless it has a p-value. In practice most designers actually building things seem to do something in between. You can't make every design decision using an A/B test unless you're prepared to run hundreds of thousands of them, because designing something non-trivial (say, a new IDE) requires quite a lot of design decisions. But you can use A/B testing (or quantitative user studies more generally) to get data on specific groups of alternatives, or to hone in on a particular subset of the design space. A lot of the argument is over the relevant proportions, and which other methods to use to supplement quantitative A/B-type testing.
One school, for example, is that you should generally aim to understand your design and audience, via things like ethnographic interviews, examination of existing designs, qualitative user studies, thought about why stuff worked/didn't, theories about user interaction and design in certain areas, etc., and then use quantitative tests as more of a sanity-checking tool (are users reacting in line with how we were expecting them to?). That's somewhat more in line with a traditional scientific view of theory formation followed by testing the predictions that theory makes. Other approaches have the quantitative tests taking a much more active, central role in the design process, more akin to the machine-learning view of induction from data, where you're agnostic about "why" something works, and just follow where the data takes you. I think that latter gets towards the approach some designers find stifling to creativity, particularly in how it's applied in e.g. metrics-based game design.
- paraschopra 16y agoAll great points. My only argument in the article was that it is just a tool which should be used if you specific case for using it. It isn't inherently bad, you just need to see where it is a good fit. Like you said, design can be more than sum of its parts so doing thousands of tests for designing a new IDE (for instance) is impossible. Though if you have a specific question to answer: should I have toolbar instead of menu bar, then of course you can do A/B testing.