A/B testing is just one component of a good product development cycle. The easiest pitfall, though, is that the data only tells you what -- not why. This can le

A/B testing is just one component of a good product development cycle. The easiest pitfall, though, is that the data only tells you what -- not why. This can lead you to make data-based design decisions blindly. I think the ideal design and development cycle incorporates A/B testing at the end to spot any outliers: You should already have done in-person usability and user testing. It's extremely cheap and, in some cases, almost magical. Seeing a real person use your product will guarantee you plenty of "a-ha!" moments. After testing with real users, I think it's appropriate to test with statistical users: You've already tackled the most glaring issues with what you're building, and the knowledge of how real people use your website can often help show you the "why" of the data, rather than just "what". Steve Krug's "Don't Make Me Think" has a chapter near the end on user testing on a shoestring budget. I'd highly recommend it.