I agree with lukestevens; there's a ton of value in A/B testing. A willingness to destroy your attractive but unusable mock is what separates the graphic design

I agree with lukestevens; there's a ton of value in A/B testing. A willingness to destroy your attractive but unusable mock is what separates the graphic designers from the interface designers. That said.... A/B testing can be extremely dangerous: - It's VERY EASY to design your test incorrectly. The most common result is that your test really ends up measuring some other factor of the design, or is chock-full of confounding factors. Any conclusions you draw from it will almost certainly be incorrect. - It's EVEN EASIER to over-generalize from your data. People clicked on the green button more than the red button...people must like green buttons! Quick, redo the design with a green palette! These conclusions are always terribly tempting to make, since it feels like you've uncovered some deep truth about your users. You probably haven't, and it probably can't be generalized beyond the exact thing that you tested. - It's easy to substitute A/B testing for critical thinking and self-editing. Two options? Don't worry about deciding! A/B test them! Another decision needs to be made? We'll let the users decide! The problem here is twofold: a). When designing something interactive, there are simply too many decisions to be made. You better get comfortable with making them, because there isn't enough time in the world to A/B test them all. b). (related to below) Your product will end up looking like it was designed-by-committee. Which it was. - Related to the previous one, you can get trapped into doing "design by A/B testing". It's a simple algorithm. Get something that looks okay, then do an A/B test on some small changes. Pick the best one. Repeat. Unfortunately, the nature of A/B testing is that it is inherently isolated. If you're testing correctly, you've eliminated all your confounding factors. You're just looking at one aspect of your system. The problem is that interfaces are inherently tightly coupled. Everything tends to depend on everything else, and then you start throwing in abstract concepts like "feel" and "friction" and general "look" of the site (how well does it fit together? does it seem to have personality? Does it respect the metaphors it creates? Is it consistent?). A/B testing can't measure that kind of stuff. You also get into problems of local maxima. This is essentially a hill-climbing algorithm. Unless you're willing to occasionally start descending a hill, you're going to get stuck in an okay-but-could-be-better hump forever. In short: A/B testing is powerful. Don't abuse it. And for godsakes, design your damn tests correctly.