5 ms·

You're right. Nobody wants to be told that they're wrong about their most preciously held beliefs about the value of their skills. But for everyone who's champ

You're right. Nobody wants to be told that they're wrong about their most preciously held beliefs about the value of their skills. But for everyone who's championing split-testing, they should remember a little bit about where it first became popular: Direct Mail. Direct Mail copywriters/designers were the first people to truly use split-testing as a major part of their professional arsenal. Split-testing gave us long-form sales letters with typewriter fonts, blue backgrounds, yellow highlighter effects, and pictures of Ferraris and McMansions and screenshots of commission checks. Why? Because they all work better.

kolektiv 15y ago
That's interesting though, because you seem to be implying (forgive me if I'm wrong) that we shouldn't have typewriter fonts, yellow highlights, etc. Or at least that it somehow isn't worth the trade off. But as you say, they do work. Don't get me wrong, I find those design elements you're talking about to be far from my tastes, but provably I'm not "normal" within the markets under test. At what point what you say to a business "you might make more money like this, but don't do it anyway because it's awful"? Apologies if I'm reading more in to your statement than was intended. I've seen enough of your work to know that you get both design and business hence I think that statement interesting!
- ahoyhere 15y ago
  I'm not saying the yellow-highlighter crowd is wrong, not at all. But would YOU buy from them? Your question exposes a really important fact about split-testing: People are complex. Different stuff works for different people. The yellow highlighter thing tends to make people like us wanna hurl… it triggers immediate distrust. Meanwhile, information marketers still find that it works. You can't know who a person is when you split-test against them. More importantly, you need a LOT of traffic, and even minor temporary changes in traffic composition can affect the quality of your data. You will see that traffic from HN responds to & converts very differently than traffic from Lifehacker, and again different from Small Biz Trends. Initially you may find certain changes increase your conversion rates… and two months later you find you have lost almost all those customers, or that a disproportionate amount of them have asked for refunds, or been shitty to you in the support channels. Net result: a loss, which you only found out about months after you were so confident that your testing showed a gain. Would the other half of the split test do better? You can never know, because you cannot replicate the exact scenario ever again. Traffic is not created equal; traffic is made up of different individuals. You can only ever get a feeling of what works for some of them some of the time, depending on several factors. Take the yellow-highlighter example again: Why does it work for infomarketers when it makes people like us flee in the other direction? They sell to different people. Why would the Hacker News crowd be worse customers than the Lifehacker crowd? Because they are different types of people. Unless you run a months-long test covering all parts of your business (refunds, avg turnover rate, length of customer stay, # of support inquiries/nastiness in support inquiries, resource usage, server bills, etc), tho, split-testing can't tell you which types of people are better for you. And if you DO do that, when exactly will you have time left to expand and market your product? I'm not saying split-testing is wrong or useless, but people use it to try to create scientific certainty when it's absolutely NOT possible to do so.
  - kolektiv 15y ago
    Thanks for the clarification - I think to some extent we're arguing on the same side. Points about traffic I would certainly agree with, and to some extent it's as much about attraction as about increasing conversion metrics once obtained. I'm not sure I'd agree with the last point about time - I think that balancing business generation, expansion and refinement is possible - but perhaps not so much amongst startup size businesses, which is more HN territory. In that case it should certainly be looked at as "is it better to try and get 0.1% more people to click buy or 20% more people to arrive" - or whatever you think your likely ratios are, hence cost/benefit call.
CoffeeDregs 15y ago
Sing it, sister! >Nobody wants to be told that they're wrong about their >most preciously held beliefs about the value of their skills. That's one of the reasons I suggest testing everyone's design/interaction suggestions. Then I publish in internal emails the results of tests. It does wonders for reducing "preciously held beliefs". The number of bullshit, I-wanna-say-something-smart-too suggestions plummets and we're left with meaningful, committed suggestions. >But for everyone who's championing split-testing, they >should remember a little bit about where it first became popular: Direct Mail. Definitely. We studied this a bit in B-School and it was fascinating to realize that all that crappy, bizarre mail I get is often very carefully crafted. Catalog companies were doing split-testing on prices before vacuum tubes were invented. I hadn't considered this, but I should also be looking for demographic targeting cues in my mail just as I do on websites.

kolektiv 15y ago

That's interesting though, because you seem to be implying (forgive me if I'm wrong) that we shouldn't have typewriter fonts, yellow highlights, etc. Or at least that it somehow isn't worth the trade off. But as you say, they do work. Don't get me wrong, I find those design elements you're talking about to be far from my tastes, but provably I'm not "normal" within the markets under test. At what point what you say to a business "you might make more money like this, but don't do it anyway because it's awful"? Apologies if I'm reading more in to your statement than was intended. I've seen enough of your work to know that you get both design and business hence I think that statement interesting!

ahoyhere 15y ago
I'm not saying the yellow-highlighter crowd is wrong, not at all. But would YOU buy from them? Your question exposes a really important fact about split-testing: People are complex. Different stuff works for different people. The yellow highlighter thing tends to make people like us wanna hurl… it triggers immediate distrust. Meanwhile, information marketers still find that it works. You can't know who a person is when you split-test against them. More importantly, you need a LOT of traffic, and even minor temporary changes in traffic composition can affect the quality of your data. You will see that traffic from HN responds to & converts very differently than traffic from Lifehacker, and again different from Small Biz Trends. Initially you may find certain changes increase your conversion rates… and two months later you find you have lost almost all those customers, or that a disproportionate amount of them have asked for refunds, or been shitty to you in the support channels. Net result: a loss, which you only found out about months after you were so confident that your testing showed a gain. Would the other half of the split test do better? You can never know, because you cannot replicate the exact scenario ever again. Traffic is not created equal; traffic is made up of different individuals. You can only ever get a feeling of what works for some of them some of the time, depending on several factors. Take the yellow-highlighter example again: Why does it work for infomarketers when it makes people like us flee in the other direction? They sell to different people. Why would the Hacker News crowd be worse customers than the Lifehacker crowd? Because they are different types of people. Unless you run a months-long test covering all parts of your business (refunds, avg turnover rate, length of customer stay, # of support inquiries/nastiness in support inquiries, resource usage, server bills, etc), tho, split-testing can't tell you which types of people are better for you. And if you DO do that, when exactly will you have time left to expand and market your product? I'm not saying split-testing is wrong or useless, but people use it to try to create scientific certainty when it's absolutely NOT possible to do so.
- kolektiv 15y ago
  Thanks for the clarification - I think to some extent we're arguing on the same side. Points about traffic I would certainly agree with, and to some extent it's as much about attraction as about increasing conversion metrics once obtained. I'm not sure I'd agree with the last point about time - I think that balancing business generation, expansion and refinement is possible - but perhaps not so much amongst startup size businesses, which is more HN territory. In that case it should certainly be looked at as "is it better to try and get 0.1% more people to click buy or 20% more people to arrive" - or whatever you think your likely ratios are, hence cost/benefit call.

CoffeeDregs 15y ago

Sing it, sister! >Nobody wants to be told that they're wrong about their >most preciously held beliefs about the value of their skills. That's one of the reasons I suggest testing everyone's design/interaction suggestions. Then I publish in internal emails the results of tests. It does wonders for reducing "preciously held beliefs". The number of bullshit, I-wanna-say-something-smart-too suggestions plummets and we're left with meaningful, committed suggestions. >But for everyone who's championing split-testing, they >should remember a little bit about where it first became popular: Direct Mail. Definitely. We studied this a bit in B-School and it was fascinating to realize that all that crappy, bizarre mail I get is often very carefully crafted. Catalog companies were doing split-testing on prices before vacuum tubes were invented. I hadn't considered this, but I should also be looking for demographic targeting cues in my mail just as I do on websites.