3 ms·

That's interesting though, because you seem to be implying (forgive me if I'm wrong) that we shouldn't have typewriter fonts, yellow highlights, etc. Or at leas

That's interesting though, because you seem to be implying (forgive me if I'm wrong) that we shouldn't have typewriter fonts, yellow highlights, etc. Or at least that it somehow isn't worth the trade off. But as you say, they do work. Don't get me wrong, I find those design elements you're talking about to be far from my tastes, but provably I'm not "normal" within the markets under test. At what point what you say to a business "you might make more money like this, but don't do it anyway because it's awful"? Apologies if I'm reading more in to your statement than was intended. I've seen enough of your work to know that you get both design and business hence I think that statement interesting!

ahoyhere 15y ago
I'm not saying the yellow-highlighter crowd is wrong, not at all. But would YOU buy from them? Your question exposes a really important fact about split-testing: People are complex. Different stuff works for different people. The yellow highlighter thing tends to make people like us wanna hurl… it triggers immediate distrust. Meanwhile, information marketers still find that it works. You can't know who a person is when you split-test against them. More importantly, you need a LOT of traffic, and even minor temporary changes in traffic composition can affect the quality of your data. You will see that traffic from HN responds to & converts very differently than traffic from Lifehacker, and again different from Small Biz Trends. Initially you may find certain changes increase your conversion rates… and two months later you find you have lost almost all those customers, or that a disproportionate amount of them have asked for refunds, or been shitty to you in the support channels. Net result: a loss, which you only found out about months after you were so confident that your testing showed a gain. Would the other half of the split test do better? You can never know, because you cannot replicate the exact scenario ever again. Traffic is not created equal; traffic is made up of different individuals. You can only ever get a feeling of what works for some of them some of the time, depending on several factors. Take the yellow-highlighter example again: Why does it work for infomarketers when it makes people like us flee in the other direction? They sell to different people. Why would the Hacker News crowd be worse customers than the Lifehacker crowd? Because they are different types of people. Unless you run a months-long test covering all parts of your business (refunds, avg turnover rate, length of customer stay, # of support inquiries/nastiness in support inquiries, resource usage, server bills, etc), tho, split-testing can't tell you which types of people are better for you. And if you DO do that, when exactly will you have time left to expand and market your product? I'm not saying split-testing is wrong or useless, but people use it to try to create scientific certainty when it's absolutely NOT possible to do so.
- kolektiv 15y ago
  Thanks for the clarification - I think to some extent we're arguing on the same side. Points about traffic I would certainly agree with, and to some extent it's as much about attraction as about increasing conversion metrics once obtained. I'm not sure I'd agree with the last point about time - I think that balancing business generation, expansion and refinement is possible - but perhaps not so much amongst startup size businesses, which is more HN territory. In that case it should certainly be looked at as "is it better to try and get 0.1% more people to click buy or 20% more people to arrive" - or whatever you think your likely ratios are, hence cost/benefit call.

ahoyhere 15y ago

I'm not saying the yellow-highlighter crowd is wrong, not at all. But would YOU buy from them? Your question exposes a really important fact about split-testing: People are complex. Different stuff works for different people. The yellow highlighter thing tends to make people like us wanna hurl… it triggers immediate distrust. Meanwhile, information marketers still find that it works. You can't know who a person is when you split-test against them. More importantly, you need a LOT of traffic, and even minor temporary changes in traffic composition can affect the quality of your data. You will see that traffic from HN responds to & converts very differently than traffic from Lifehacker, and again different from Small Biz Trends. Initially you may find certain changes increase your conversion rates… and two months later you find you have lost almost all those customers, or that a disproportionate amount of them have asked for refunds, or been shitty to you in the support channels. Net result: a loss, which you only found out about months after you were so confident that your testing showed a gain. Would the other half of the split test do better? You can never know, because you cannot replicate the exact scenario ever again. Traffic is not created equal; traffic is made up of different individuals. You can only ever get a feeling of what works for some of them some of the time, depending on several factors. Take the yellow-highlighter example again: Why does it work for infomarketers when it makes people like us flee in the other direction? They sell to different people. Why would the Hacker News crowd be worse customers than the Lifehacker crowd? Because they are different types of people. Unless you run a months-long test covering all parts of your business (refunds, avg turnover rate, length of customer stay, # of support inquiries/nastiness in support inquiries, resource usage, server bills, etc), tho, split-testing can't tell you which types of people are better for you. And if you DO do that, when exactly will you have time left to expand and market your product? I'm not saying split-testing is wrong or useless, but people use it to try to create scientific certainty when it's absolutely NOT possible to do so.

kolektiv 15y ago
Thanks for the clarification - I think to some extent we're arguing on the same side. Points about traffic I would certainly agree with, and to some extent it's as much about attraction as about increasing conversion metrics once obtained. I'm not sure I'd agree with the last point about time - I think that balancing business generation, expansion and refinement is possible - but perhaps not so much amongst startup size businesses, which is more HN territory. In that case it should certainly be looked at as "is it better to try and get 0.1% more people to click buy or 20% more people to arrive" - or whatever you think your likely ratios are, hence cost/benefit call.