4 ms·

Or c) that the "better" design is in fact provably worse for users and is worth negative money, which seems to be the main gripe in the post. I actually think

by Cushman 15y ago

Or c) that the "better" design is in fact provably worse for users and is worth negative money, which seems to be the main gripe in the post. I actually think it goes deeper than just narrative; there's a difference of ontology. To a designer, a "better" design means a more aesthetic one. To a businessman, it means a more profitable one. They're not completely contradictory, but they don't always match up either— so it shouldn't be surprising that there's some cognitive dissonance going on when the A/B test says the worse design is actually better.

_delirium 15y ago
There's a little bit of a timescale issue, also; are you building something for a specific, short-term purpose, or is does it impact a larger strategy? A/B testing is at its best where the measurable quantity is short-term and largely independent of anything else, like conversion on a landing page. Things get more murky when you want longer-term things, like building a coherent/consistent brand, where every product/image/campaign you do doesn't wildly jump around--- imagine an Apple where every product was ruthlessly optimized in isolation, with no attention to whether it fit into Apple's larger design sense. A designer might not have the right answers there either, but the right thing to A/B test is a bit trickier, because you aren't testing decisions that are independent and easy to isolate. What you want is the overall set of decisions that maximizes 5-year profits, which might not be simply the sum of decisions that maximizes profit in every individual case. One subset of solutions is to have someone responsible for vetoing elements of individual campaigns that might cause collateral damage bigger than the campaign's value. For example, a Disney landing page that wildly outconverts other possibilities might still get vetoed if it does something with the Disney characters that isn't in line with the image they want for them. Another is to have broader design parameters imposed ahead of time, with A/B testing restricted to a smaller set of design variables permitted to vary, like Apple does.

by Cushman 15y ago

_delirium 15y ago

There's a little bit of a timescale issue, also; are you building something for a specific, short-term purpose, or is does it impact a larger strategy? A/B testing is at its best where the measurable quantity is short-term and largely independent of anything else, like conversion on a landing page. Things get more murky when you want longer-term things, like building a coherent/consistent brand, where every product/image/campaign you do doesn't wildly jump around--- imagine an Apple where every product was ruthlessly optimized in isolation, with no attention to whether it fit into Apple's larger design sense. A designer might not have the right answers there either, but the right thing to A/B test is a bit trickier, because you aren't testing decisions that are independent and easy to isolate. What you want is the overall set of decisions that maximizes 5-year profits, which might not be simply the sum of decisions that maximizes profit in every individual case. One subset of solutions is to have someone responsible for vetoing elements of individual campaigns that might cause collateral damage bigger than the campaign's value. For example, a Disney landing page that wildly outconverts other possibilities might still get vetoed if it does something with the Disney characters that isn't in line with the image they want for them. Another is to have broader design parameters imposed ahead of time, with A/B testing restricted to a smaller set of design variables permitted to vary, like Apple does.