9 ms·

I could not disagree with this more. I remember vividly having this "low-intent" vs "high-intent" debate at Thumbtack, when we rolled out changes that A/B test

by birken 8y ago

I could not disagree with this more. I remember vividly having this "low-intent" vs "high-intent" debate at Thumbtack, when we rolled out changes that A/B tests showed increased conversion (by a lot), but some people in the company thought the changes were ugly and "off-brand" and argued they brought in the wrong type of customers. So we ran the test again that we knew raised conversion by a lot, and then followed the 2 cohorts of customers and watched their behavior. The control group vs the 10% more from whatever the test was that increased conversion. They behaved exactly the same. They came back again at the same rates. They made the same amount of profit (per customer). Their response rates to emails were the same. They closed jobs at the same rates. As far as we could tell they were identical. I have to admit I was a little surprised too, but for our business it didn't seem this "high-intent" vs "low-intent" distinction existed. And with that out of the way we continued to optimize conversion rates, and our revenue continued to go up. Every company is different so I don't want to generalize too much, but if somebody tells me they ran an A/B test that said some key flow went up 10%, but then afterwards the traffic/revenue/whatever didn't go up 10%, I think the most likely candidate is bad test design. Humans are really good at rigging A/B tests to produce wrong results in their favor. I guarantee every single company who isn't maniacal about A/B testing does at least one of the following: - Uses a tool to grade A/B tests that isn't statistically sound - Let's people check tests too often and allows them to stop the test when it hits a good result - Running a test with a lot of similar variations and cherry picking the best one - Doesn't plan for enough traffic to detect the percentage of change their test is likely to produce All of these create the potential for the perceived gains of the A/B test not matching up with real world result. I'm not saying the distinction between "low-intent" and "high-intent" customers doesn't exist, but it is fairly easy to test for. Do that test for your business and see if that distinction exists. But don't use it as some magical explanation for why your A/B tests aren't producing the results you want as this article suggests.

sokoloff 8y ago
> Let's people check test too often and allows them to stop the test when it hits a good result I admit to attempting to be guilty of this in the past and being stopped by our analytics team (in the sense that they took the time to patiently explain to me why what I was doing was statistically unsound). It's not obvious, IMO.
- alexbeloi 8y ago
  That is actually one of the biggest contributing factors to the replication crisis in science, lots of scientists have been making this error for decades. Very not obvious.
- birken 8y ago
  I completely agree. We went through all of this stuff as our A/B testing was evolving at Thumbtack. They seem so simple, but the deeper you get you realize they are only simple if you don't "cheat", which everybody does. And you'll only realize you are cheating in the first place if you have somebody who is maniacal about testing. If you create a culture where positive A/B tests are lauded (which is good!), then you create a lot of people who want A/B tests to finish in positive ways. For those people, it doesn't really matter if their A/B test actually improves things, only if it looks like it does. This isn't nefarious, this is just human nature, but it creates a lot of creativity and energy at finding ways to making winning A/B tests. We'd have people run a test, where their new variation got off to a bad start, then say "oh, it was a bug", then they'd fix some irrelevant thing and start over just to reset the counters. I was guilty of it sometimes. You get excited about your tests and want them to win. That is why it is critical either to have gatekeepers like an analytics team to keep you honest or have a really specific protocol on how your company runs tests and only consider results of tests that followed the protocol.
- marcosdumay 8y ago
  Well, you can arrange the test so checking often is not a problem. It may even be the optimal way. But yes, statistics is not obvious at all.
- btilly 8y ago
  Meh. There is statistics for the purpose of uncovering Truth, and statistics for the purpose of making a business decision. The difference is that when we talk about Truth, a small error is still an error. When we make business decisions, it is fine to make a decision that is probably right, and we know isn't far wrong. Here is a perfectly valid test procedure that illustrates the difference. Decide the most time you would be willing to spend to get a test result. Multiply that by current conversion rates to get N, the number of conversions that you expect to see by the end of the test. Start running the test with two variations. Stop at any point if one variation is at least sqrt(N) conversions ahead of the other. Stop at N if there is no clear winner and go with whoever is ahead, even by a hair. Here are features of this test procedure. o You always make a decision. o Running a test has a known fixed cost. You know how long it takes. And a bad idea will cost you no more than sqrt(N) conversions to test. o The results are very simple and easy to understand. o Your answers are usually right. o Your bad decisions are not very bad. If the true conversion rate for one version is better by 1/sqrt(N), you've got a 95% chance of making the right choice. You will probably never make a mistake as big as 2/sqrt(N). The result is a test procedure that is a horrible approach for doing science, but an excellent tool for improving a business. You'll never find it in a statistics class. And I'm sure it would horrify your analytics team.
Karrot_Kream 8y ago
Well said. The problem is, most companies don't really build up a model for conversions, ignore sample sizes, and do not fit their A/B test results along the model, they just blindly look for the higher percentage. I guess this is the problem if you don't have any statistics-aware engineers.
ssharp 8y ago
I think it's always good to attach A/B tests and other web analytics to individuals so you can go back and do this type of segmentation and analysis. A/B test software like VWO is great, but when it comes to analytics and measurement, looking at real people (assuming you have an identifier for them) is much better than looking at pixel'd representations of a person. Attaching A/B test variations, sourcing data (UTMs), and other data to actual customer/user records in your database is a great start to understanding your results better. That allows you to do things like suss out "intent" (essentially conversion rate), "quality" (essentially LTV), etc. down the road and really understand the value of your tests and acquisition channels. I'm always surprised when companies with more than enough resources to do this don't do this.
gwbas1c 8y ago
Did you read the article? The real point has less to do with intent that you realize. Most of the discussion has to do with having healthy skepticism for a vendor making exaggerated claims; and ensuring that you retain customers.
samd 8y ago
So damn true, it's so difficult to get statistically significant A/B test results, and the popular tools actively lead you astray. It's rare enough for a startup to have the traffic to meaningfully get results on their homepage before the heat death of the universe, let alone random landing or in-app pages. I'd recommend anyone reading this who does or wants to do A/B tests read: https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html It was one of the trickiest lessons I had to learn. Once you realize you need to set your sample size in advance, you actually have to do the math to figure out the traffic you'll need. That's when reality hits you.
- birken 8y ago
  Yup. The best thing for running an A/B test you can do, by far, is setting up the rules in advance. Our protocol was something like this: This test is going to have 2 variations, looking for a 5% increase of the conversion rate that is currently X%, and to do so the test will run for Y iterations (based on the company standards for significance and power). If the test shows a >= 5% increase, it wins and we use it. Yay! If not, we assume it is no different than the baseline, record the results and discard it. You are welcome to peek at the results all you want, but no tests are stopped early and no decisions are made until it reaches the set amount of iterations. This isn't the only good or valid A/B testing protocol, but it does force people to consider the costs of their tests in advance (in terms of time and iterations required), which I think had a positive effect on the type of tests people ran. Just having the calculator discourages people from running tests with tons of variations looking for tiny increases (like the famous try different colors of your submit button), because just looking at the iterations required by the calculator it becomes obvious to everybody that those types of tests just cannot show any meaningful results for your average website.
draw_down 8y ago
Hmm, sounds like A/B tests are really good, assuming you run them in a way that almost nobody runs them in practice.

by birken 8y ago

sokoloff 8y ago

> Let's people check test too often and allows them to stop the test when it hits a good result I admit to attempting to be guilty of this in the past and being stopped by our analytics team (in the sense that they took the time to patiently explain to me why what I was doing was statistically unsound). It's not obvious, IMO.

alexbeloi 8y ago
That is actually one of the biggest contributing factors to the replication crisis in science, lots of scientists have been making this error for decades. Very not obvious.
birken 8y ago
I completely agree. We went through all of this stuff as our A/B testing was evolving at Thumbtack. They seem so simple, but the deeper you get you realize they are only simple if you don't "cheat", which everybody does. And you'll only realize you are cheating in the first place if you have somebody who is maniacal about testing. If you create a culture where positive A/B tests are lauded (which is good!), then you create a lot of people who want A/B tests to finish in positive ways. For those people, it doesn't really matter if their A/B test actually improves things, only if it looks like it does. This isn't nefarious, this is just human nature, but it creates a lot of creativity and energy at finding ways to making winning A/B tests. We'd have people run a test, where their new variation got off to a bad start, then say "oh, it was a bug", then they'd fix some irrelevant thing and start over just to reset the counters. I was guilty of it sometimes. You get excited about your tests and want them to win. That is why it is critical either to have gatekeepers like an analytics team to keep you honest or have a really specific protocol on how your company runs tests and only consider results of tests that followed the protocol.
marcosdumay 8y ago
Well, you can arrange the test so checking often is not a problem. It may even be the optimal way. But yes, statistics is not obvious at all.
btilly 8y ago
Meh. There is statistics for the purpose of uncovering Truth, and statistics for the purpose of making a business decision. The difference is that when we talk about Truth, a small error is still an error. When we make business decisions, it is fine to make a decision that is probably right, and we know isn't far wrong. Here is a perfectly valid test procedure that illustrates the difference. Decide the most time you would be willing to spend to get a test result. Multiply that by current conversion rates to get N, the number of conversions that you expect to see by the end of the test. Start running the test with two variations. Stop at any point if one variation is at least sqrt(N) conversions ahead of the other. Stop at N if there is no clear winner and go with whoever is ahead, even by a hair. Here are features of this test procedure. o You always make a decision. o Running a test has a known fixed cost. You know how long it takes. And a bad idea will cost you no more than sqrt(N) conversions to test. o The results are very simple and easy to understand. o Your answers are usually right. o Your bad decisions are not very bad. If the true conversion rate for one version is better by 1/sqrt(N), you've got a 95% chance of making the right choice. You will probably never make a mistake as big as 2/sqrt(N). The result is a test procedure that is a horrible approach for doing science, but an excellent tool for improving a business. You'll never find it in a statistics class. And I'm sure it would horrify your analytics team.

Karrot_Kream 8y ago

Well said. The problem is, most companies don't really build up a model for conversions, ignore sample sizes, and do not fit their A/B test results along the model, they just blindly look for the higher percentage. I guess this is the problem if you don't have any statistics-aware engineers.

ssharp 8y ago

I think it's always good to attach A/B tests and other web analytics to individuals so you can go back and do this type of segmentation and analysis. A/B test software like VWO is great, but when it comes to analytics and measurement, looking at real people (assuming you have an identifier for them) is much better than looking at pixel'd representations of a person. Attaching A/B test variations, sourcing data (UTMs), and other data to actual customer/user records in your database is a great start to understanding your results better. That allows you to do things like suss out "intent" (essentially conversion rate), "quality" (essentially LTV), etc. down the road and really understand the value of your tests and acquisition channels. I'm always surprised when companies with more than enough resources to do this don't do this.

gwbas1c 8y ago

Did you read the article? The real point has less to do with intent that you realize. Most of the discussion has to do with having healthy skepticism for a vendor making exaggerated claims; and ensuring that you retain customers.

samd 8y ago

So damn true, it's so difficult to get statistically significant A/B test results, and the popular tools actively lead you astray. It's rare enough for a startup to have the traffic to meaningfully get results on their homepage before the heat death of the universe, let alone random landing or in-app pages. I'd recommend anyone reading this who does or wants to do A/B tests read: https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html It was one of the trickiest lessons I had to learn. Once you realize you need to set your sample size in advance, you actually have to do the math to figure out the traffic you'll need. That's when reality hits you.

birken 8y ago
Yup. The best thing for running an A/B test you can do, by far, is setting up the rules in advance. Our protocol was something like this: This test is going to have 2 variations, looking for a 5% increase of the conversion rate that is currently X%, and to do so the test will run for Y iterations (based on the company standards for significance and power). If the test shows a >= 5% increase, it wins and we use it. Yay! If not, we assume it is no different than the baseline, record the results and discard it. You are welcome to peek at the results all you want, but no tests are stopped early and no decisions are made until it reaches the set amount of iterations. This isn't the only good or valid A/B testing protocol, but it does force people to consider the costs of their tests in advance (in terms of time and iterations required), which I think had a positive effect on the type of tests people ran. Just having the calculator discourages people from running tests with tons of variations looking for tiny increases (like the famous try different colors of your submit button), because just looking at the iterations required by the calculator it becomes obvious to everybody that those types of tests just cannot show any meaningful results for your average website.

draw_down 8y ago

Hmm, sounds like A/B tests are really good, assuming you run them in a way that almost nobody runs them in practice.