3 ms·
Why don't (some) developers like, e.g., the Lean Startup? Because you can execute your job perfectly, and drive the business right off an effing cliff. It str
by patio11 15y ago
Why don't (some) developers like, e.g., the Lean Startup? Because you can execute your job perfectly, and drive the business right off an effing cliff. It strikes at some of our core narratives, like "successful startups are made on the backs of technical competence, largely by upstart engineers who are a) ridiculously competent and b) scorned by people in positions of power."
You can't tell someone who really feels that narrative that, by the way, technical competence is not sufficient for success. (To say nothing of saying it isn't necessary.)
So why don't designers like A/B testing? Well, what's their narrative? User-loving highly-skilled artist scorned by people in positions of power. (Wow, we have a trend here.) They don't want to be told that a) users provably don't care that this design looks better or is more intuitive or b) that beautiful/responsive/intuitive/etc design is not always worth actual money.
- Cushman 15y agoOr c) that the "better" design is in fact provably worse for users and is worth negative money, which seems to be the main gripe in the post. I actually think it goes deeper than just narrative; there's a difference of ontology. To a designer, a "better" design means a more aesthetic one. To a businessman, it means a more profitable one. They're not completely contradictory, but they don't always match up either— so it shouldn't be surprising that there's some cognitive dissonance going on when the A/B test says the worse design is actually better.
- _delirium 15y agoThere's a little bit of a timescale issue, also; are you building something for a specific, short-term purpose, or is does it impact a larger strategy? A/B testing is at its best where the measurable quantity is short-term and largely independent of anything else, like conversion on a landing page. Things get more murky when you want longer-term things, like building a coherent/consistent brand, where every product/image/campaign you do doesn't wildly jump around--- imagine an Apple where every product was ruthlessly optimized in isolation, with no attention to whether it fit into Apple's larger design sense. A designer might not have the right answers there either, but the right thing to A/B test is a bit trickier, because you aren't testing decisions that are independent and easy to isolate. What you want is the overall set of decisions that maximizes 5-year profits, which might not be simply the sum of decisions that maximizes profit in every individual case. One subset of solutions is to have someone responsible for vetoing elements of individual campaigns that might cause collateral damage bigger than the campaign's value. For example, a Disney landing page that wildly outconverts other possibilities might still get vetoed if it does something with the Disney characters that isn't in line with the image they want for them. Another is to have broader design parameters imposed ahead of time, with A/B testing restricted to a smaller set of design variables permitted to vary, like Apple does.
- deleted 15y ago[deleted]
- deleted 15y ago[deleted]
- wpietri 15y agoThe Lean Startup approach strongly encourages following your instincts and taking risks. What it discourages is taking risks that are larger than necessary out of arrogance or false pride. A good example is WebVan. They blew a billion dollars trying to do internet-based grocery shopping at scale. They learned the world wasn't ready. They could have learned that for maybe 1% of that cost. And then had the other 99% of the money to find something that actually works. I think my own startup is another good example. We started out with an idea that we thought was great. After funding, we discovered two funded competitors. We did a bunch of small-scale tests of our product, discovered it wouldn't scale, and killed it after 7 weeks. The competitors spent 9 months to a year and millions of dollars. You can see my co-founder describe our experience in this 5-minute Ignite talk: http://vimeo.com/24749599 http://vimeo.com/24749599 Instincts are great. They're also frequently wrong. That's why we test our ideas vigorously before we bet big on them.
- ahoyhere 15y agoYou're right. Nobody wants to be told that they're wrong about their most preciously held beliefs about the value of their skills. But for everyone who's championing split-testing, they should remember a little bit about where it first became popular: Direct Mail. Direct Mail copywriters/designers were the first people to truly use split-testing as a major part of their professional arsenal. Split-testing gave us long-form sales letters with typewriter fonts, blue backgrounds, yellow highlighter effects, and pictures of Ferraris and McMansions and screenshots of commission checks. Why? Because they all work better.
- kolektiv 15y agoThat's interesting though, because you seem to be implying (forgive me if I'm wrong) that we shouldn't have typewriter fonts, yellow highlights, etc. Or at least that it somehow isn't worth the trade off. But as you say, they do work. Don't get me wrong, I find those design elements you're talking about to be far from my tastes, but provably I'm not "normal" within the markets under test. At what point what you say to a business "you might make more money like this, but don't do it anyway because it's awful"? Apologies if I'm reading more in to your statement than was intended. I've seen enough of your work to know that you get both design and business hence I think that statement interesting!
- ahoyhere 15y agoI'm not saying the yellow-highlighter crowd is wrong, not at all. But would YOU buy from them? Your question exposes a really important fact about split-testing: People are complex. Different stuff works for different people. The yellow highlighter thing tends to make people like us wanna hurl… it triggers immediate distrust. Meanwhile, information marketers still find that it works. You can't know who a person is when you split-test against them. More importantly, you need a LOT of traffic, and even minor temporary changes in traffic composition can affect the quality of your data. You will see that traffic from HN responds to & converts very differently than traffic from Lifehacker, and again different from Small Biz Trends. Initially you may find certain changes increase your conversion rates… and two months later you find you have lost almost all those customers, or that a disproportionate amount of them have asked for refunds, or been shitty to you in the support channels. Net result: a loss, which you only found out about months after you were so confident that your testing showed a gain. Would the other half of the split test do better? You can never know, because you cannot replicate the exact scenario ever again. Traffic is not created equal; traffic is made up of different individuals. You can only ever get a feeling of what works for some of them some of the time, depending on several factors. Take the yellow-highlighter example again: Why does it work for infomarketers when it makes people like us flee in the other direction? They sell to different people. Why would the Hacker News crowd be worse customers than the Lifehacker crowd? Because they are different types of people. Unless you run a months-long test covering all parts of your business (refunds, avg turnover rate, length of customer stay, # of support inquiries/nastiness in support inquiries, resource usage, server bills, etc), tho, split-testing can't tell you which types of people are better for you. And if you DO do that, when exactly will you have time left to expand and market your product? I'm not saying split-testing is wrong or useless, but people use it to try to create scientific certainty when it's absolutely NOT possible to do so.
- duopixel 15y agoAs a designer I only speak for myself because I know my view is quite different from other designers. Imagine this scenario: you bring four aliens to a table, you give them a deck of cards, and you explain the mechanics of poker (but not the actual rules). You get them to play, remain silent throughout the game, and at the end you announce the winner. On the first game, an alien wins with a pair of sevens. So the aliens think "hey, it's the sevens that have value!", and pretty soon the aliens are fighting over the sevens, and they call it the game of sevens. A|B testing gives you a tiny glimpse of a wicked system that happens to work under certain circumstances. I love A|B testing, but I know you can't make all encompassing announcements that state that buttons should always be beveled (as Jakob Nielsen says) because it depends on the specific circumstances of the design. Just look at gmail's new design, I'm sure it's pretty well tested and there's no trace of beveled buttons. I remember an interesting piece you wrote on changing the logo for Bingo Card Creator, and you announced that logos are worth nothing. But then comes GAP with their horrendous logo and the internet goes batshit over it. I think we are in the alchemy ages of design, and there is a lot of resistance in adopting scientific methods for validating our work. But we must understand that every test is a tiny glimpse of the overall rules of the game. Every little piece of evidence is precious in understanding the whole system.
- dasil003 15y agoI totally agree with your conclusion, so don't take this as an overall disagreement, but the flaw in the analogy is that the rules of poker are simple, and the complexity is emergent from that point. The rules of cognition in regards to visual and interactive input may well be simple at the very low levels (neurons let's say), but even if we could isolate those rules the ability to cognitively extrapolate principles of design would be beyond the capability of the human brain.
- statictype 15y agoSo what's the narrative for consultants who preach the value of A/B testing everything? :)