5 ms·

I don't buy this article at all. I am no longer with the company, but around 10 years ago, I turned a failing consultancy into an SEO SaaS company. Turning rev

I don't buy this article at all. I am no longer with the company, but around 10 years ago, I turned a failing consultancy into an SEO SaaS company. Turning revenues from -40k/month to 90+k/month after laying off more than half the staff when we lost our last client and deciding to take our destiny into our own hands building a suite of SEO products. We ran split tests on almost every design change. They weren't always tiny, some were entire page layouts. But even on things like changing the header text saw measurable differences. I built the ability to split test into every facet of our framework. We had the ability to split test pages, transactional emails, marketing emails, and we would tweak everything all the time. If we noticed group a wasn't performing as well as b, we would drop a. There were some tiny things that made major differences: - Email "From" being a person's name and not just the service bumpped onboarding but nearly 20%. - The onboarding subject "Almost Done..." performed about 10% better than "Activate your account" - Having the video auto play converted visitors almost 50% more often than them having to click the play button (it may be evil, but it makes money). - Having the video be an animation vs just a spokesperson talking into a webcam converted 5% better with the same script (+2% better with just reading the text from the page). We had a large audience after we launched our first product free of charge for a limited time (linklicious.co -- formerly linklicious.me -- formerly lts.me), then used that momentum to built out a new product every month or so for the next couple of years. It was a great job in a terrible industry. 80% of my job was inventing new ideas.

tobr 8y ago
How many impressions were you running if you were able to measure a 2% difference?
- chudi 8y ago
  If you want to be completely sure, a really small amount, lets say your baseline is 20% to detect a 2% difference you just need 2 x 6,347 views for each test Check this tools[0] and this article [1] [0] https://www.evanmiller.org/ab-testing/sample-size.html#!20;80;5;2;0 https://www.evanmiller.org/ab-testing/sample-size.html#!20;8... [1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html
- jermaustin1 8y ago
  When we implemented split-testing-by-default baked into the software itself, we saw maybe 1000 visitors a week from various sources. Our free conversion rate was at about 3% and once we were "fully optimized" we were closer to 10-15%, and our paid conversions were about 10-30% of the free users depending on the month (December and January always sucked). Most of these numbers were measured after at least a week of leaving the experiments running. We never tested more than experiment at a time per visitor, but our framework had the ability to run multiple tests simultaneously, while cookie-ing a visitor to a single test.
shanghaiaway 8y ago
-Take your email list -Split it in two equal halfs -Send each half the same email -Open and click rates will differ between the two lists
- teisman 8y ago
  But if you apply statistics, you will see that the open rates only differ significantly 5% of the time, if you use a 95% confidence interval.
  - jrochkind1 8y ago
    a) 5% is still one in 20. Are you doing 20 trials or more a month? (Of course, it won't be _exactly_ 5% of the time... there's some other statistics we could calculate to say how likely it is to differ by more than some specified delta to 5% depending on how many trials you run... oh my) b) That's assuming you are using statistics right. It's quite hard to use statistics right.
    - kingo55 8y ago
      That's way better odds than most people's gut feel. Everyone thinks they don't need to run experiments because THEY already know what works. Did you know, in experimentation programs run by Microsoft, Google and Amazon, roughly two thirds of their ideas have no impact or hurt the metrics they were designed to improve? And yet rookie web Devs or marketing assistants "know" better. Source: https://www.google.com.au/url?sa=t&source=web&rct=j&url=https://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf&ved=2ahUKEwjL6MaIt_reAhWE2qQKHbKvA4kQFjACegQICRAB&usg=AOvVaw0OX6ICAkRXx0ot0nggDv-L https://www.google.com.au/url?sa=t&source=web&rct=j&url=http... (Read section three of this paper by a Microsoft distinguished engineer)
      - nostrademons 8y ago
        It's actually more like 80% for Google. I spent a lot of time running various experiments on Search. I'll point out a major difference, though: Microsoft, Google, and Amazon are already highly optimized. They've had millions of man-hours put into optimizing everything from product design to UI to UI element positioning to wording to colors to fonts. When you get a new, beneficial result in a change to Google, it's usually because the environment has changed while you weren't looking, and the user is different. That doesn't apply to a new business that's just learning how to sell their product. In a new business, by definition, you've spent zero time micro-optimizing your message & product. You can get really big wins almost by accident, if you happen to have stumbled into a product category where users actually want what you're building.
sixo 8y ago
> Having the video auto play converted visitors almost 50% more often than them having to click the play button (it may be evil, but it makes money). Because the ones who weren't going to convert never came back
- stevenwliao 8y ago
  It may or may not be true that they never came back. A good split test tool will tell you that as well.
  - TeMPOraL 8y ago
    I'm increasingly of the opinion that split-testing tools are the hidden way Satan influences the world. If your A/B tests keep telling you that you should make the product more user-hostile and less useful, that you should manipulate the users and abuse the commons for short-term gains, something is very wrong - with the tools, the tests, or the world.
    - stevenwliao 8y ago
      Split testing can be used by companies who care about their users and companies who don't. It just tells you how your users react. It's not a substitute for user empathy or regulations.

tobr 8y ago

How many impressions were you running if you were able to measure a 2% difference?

chudi 8y ago
If you want to be completely sure, a really small amount, lets say your baseline is 20% to detect a 2% difference you just need 2 x 6,347 views for each test Check this tools[0] and this article [1] [0] https://www.evanmiller.org/ab-testing/sample-size.html#!20;80;5;2;0 https://www.evanmiller.org/ab-testing/sample-size.html#!20;8... [1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html
jermaustin1 8y ago
When we implemented split-testing-by-default baked into the software itself, we saw maybe 1000 visitors a week from various sources. Our free conversion rate was at about 3% and once we were "fully optimized" we were closer to 10-15%, and our paid conversions were about 10-30% of the free users depending on the month (December and January always sucked). Most of these numbers were measured after at least a week of leaving the experiments running. We never tested more than experiment at a time per visitor, but our framework had the ability to run multiple tests simultaneously, while cookie-ing a visitor to a single test.

shanghaiaway 8y ago

-Take your email list -Split it in two equal halfs -Send each half the same email -Open and click rates will differ between the two lists

teisman 8y ago
But if you apply statistics, you will see that the open rates only differ significantly 5% of the time, if you use a 95% confidence interval.
- jrochkind1 8y ago
  a) 5% is still one in 20. Are you doing 20 trials or more a month? (Of course, it won't be _exactly_ 5% of the time... there's some other statistics we could calculate to say how likely it is to differ by more than some specified delta to 5% depending on how many trials you run... oh my) b) That's assuming you are using statistics right. It's quite hard to use statistics right.
  - kingo55 8y ago
    That's way better odds than most people's gut feel. Everyone thinks they don't need to run experiments because THEY already know what works. Did you know, in experimentation programs run by Microsoft, Google and Amazon, roughly two thirds of their ideas have no impact or hurt the metrics they were designed to improve? And yet rookie web Devs or marketing assistants "know" better. Source: https://www.google.com.au/url?sa=t&source=web&rct=j&url=https://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf&ved=2ahUKEwjL6MaIt_reAhWE2qQKHbKvA4kQFjACegQICRAB&usg=AOvVaw0OX6ICAkRXx0ot0nggDv-L https://www.google.com.au/url?sa=t&source=web&rct=j&url=http... (Read section three of this paper by a Microsoft distinguished engineer)
    - nostrademons 8y ago
      It's actually more like 80% for Google. I spent a lot of time running various experiments on Search. I'll point out a major difference, though: Microsoft, Google, and Amazon are already highly optimized. They've had millions of man-hours put into optimizing everything from product design to UI to UI element positioning to wording to colors to fonts. When you get a new, beneficial result in a change to Google, it's usually because the environment has changed while you weren't looking, and the user is different. That doesn't apply to a new business that's just learning how to sell their product. In a new business, by definition, you've spent zero time micro-optimizing your message & product. You can get really big wins almost by accident, if you happen to have stumbled into a product category where users actually want what you're building.

sixo 8y ago

> Having the video auto play converted visitors almost 50% more often than them having to click the play button (it may be evil, but it makes money). Because the ones who weren't going to convert never came back

stevenwliao 8y ago
It may or may not be true that they never came back. A good split test tool will tell you that as well.
- TeMPOraL 8y ago
  I'm increasingly of the opinion that split-testing tools are the hidden way Satan influences the world. If your A/B tests keep telling you that you should make the product more user-hostile and less useful, that you should manipulate the users and abuse the commons for short-term gains, something is very wrong - with the tools, the tests, or the world.
  - stevenwliao 8y ago
    Split testing can be used by companies who care about their users and companies who don't. It just tells you how your users react. It's not a substitute for user empathy or regulations.