3 ms·

This article has a valid message that basically boils down to "correlation does not imply causation." While the discussion centers around the problems with maki

by Wilduck 15y ago

This article has a valid message that basically boils down to "correlation does not imply causation." While the discussion centers around the problems with making decisions based off of aggregate statistics, the underlying theme that having a piece of data doesn't mean you know what caused it, boils down to the tried and true correlation/causation problem. Unfortunately, I don't think the author teased these notions apart very well. He does suggest one solution to this problem, doing A/B tests. That's a fine solution where it works. However, as he mentioned it is hard to do outside of limited circumstances. At this point, more sophisticated tools are necessary. Coming from a background in economics/econometrics, I'm acutely aware of the statistical difficulties that arise when you cannot perform controlled experiments. Here, the root of the issue seems to be endogeniety [1]. For example, in his discussion the 'Stickiness' of facebook he claims: "Stickiness" is an aggregate number, it represents the result of the quality of the app. It's a result. It's not a cause. I think this only gets us half way to the problem. Yes, to some extent it measures the quality of the app, but there is also a feedback loop, where stickiness increases the quality of the app. So, if you're trying to measure the effect of quality on stickiness, or vice versa, you need some way of disentangling this feedback loop. These tools exist. If you're interested in estimating both effects jointly, you can use Two Staged Least Squares [2]. If you're just looking to estimate one side, you can use an instrumental variable approach [3]. These approaches are used all the time in economics, as the data involved is almost always messy. I'm curious why I hear so much about A/B testing, but rarely hear about other statistical methods for assessing changes in usage patterns. Are they not being used? Am I simply looking in the wrong places? Or is A/B testing really sufficient in most cases? [1] http://en.wikipedia.org/wiki/Endogeneity_(economics) http://en.wikipedia.org/wiki/Endogeneity_(economics) [2] http://en.wikipedia.org/wiki/Simultaneous_equations_model#Estimation http://en.wikipedia.org/wiki/Simultaneous_equations_model#Es... [3] http://en.wikipedia.org/wiki/Instrumental_variable http://en.wikipedia.org/wiki/Instrumental_variable

DanielBMarkham 15y ago
Author here. Thanks for the comment. My intent wasn't really to tease apart much or even to shed any light on statistics, but just to persuade the reader that this is both a critical area for startups and an area where our intuition is for shit. I'd like to hear more about this. Anybody else with applied knowledge out there? Seems to me there are two problems here. The first is that we live in a world with lots of data yet little information. Understanding the limits of what we can know is important. (And I hope it didn't all come down to "correlation does not equal causation" for you. The Monty Hall problem is definitely not an example of that) The second thing is taking the pre-existing math and applying it in a useful manner. The second problem is probably much more interesting to this crowd, but I was concerned that most of us (myself included) don't recognize the first problem to the degree we should. The essay actually began as a rant against trusting too much in aggregate numbers, but at the end I was left with having to make a recommendation, and the best I could come up with -- the most practical solution that is easily explainable and implementable -- is A/B testing. Are there other options that people are using?
- jcr 15y ago
  > Out of all the startup skills I've studied, this one -- effectively inferring intent from reams of numbers -- is probably the most difficult. "There are only two types of people in this world; those who can extrapolate from incomplete information." It's a humorous way to get the same point across. It's particularly fun to say in person during a discussion, then just stop speaking and wait for the point to hit home. The more interesting question is why human beings get a warm, fuzzy feeling when they extrapolate from incomplete information (or extrapolate from incorrect information, or extrapolate incorrectly from correct information)? I haven't bothered to count them all, but there are plenty of myths about some deity dragging something bright or burning across the sky to explain the rising and setting of the sun.
noelwelsh 15y ago
Regarding your last point, we're trying to change things! As a economist you might be familiar with the bandit problem. We're bringing bandit algorithms to the web at http://www.mynaewb.com/ http://www.mynaewb.com/ Our results show a BIG improvement over A/B (see the blog).
jcr 15y ago
> I'm curious why I hear so much about A/B testing, but rarely hear about other statistical methods for assessing changes in usage patterns. Are they not being used? Am I simply looking in the wrong places? Or is A/B testing really sufficient in most cases? I believe part of the problem is you actually know the definition of "A/B Testing" and the real definition (testing a single, specific change) is mostly lost on the majority of those using the phrase. In other words, "A/B Testing" in common use has essentially become slang for "Multi-Variate Testing" and other forms of testing. For example, if you only change the color of a button, you're (formally) doing "A/B Testing" but if you change both the color and location of a button, you're (formally) doing "Multi-Variate Testing." None the less, you'll often see the later called the former in common usage.

by Wilduck 15y ago

DanielBMarkham 15y ago

Author here. Thanks for the comment. My intent wasn't really to tease apart much or even to shed any light on statistics, but just to persuade the reader that this is both a critical area for startups and an area where our intuition is for shit. I'd like to hear more about this. Anybody else with applied knowledge out there? Seems to me there are two problems here. The first is that we live in a world with lots of data yet little information. Understanding the limits of what we can know is important. (And I hope it didn't all come down to "correlation does not equal causation" for you. The Monty Hall problem is definitely not an example of that) The second thing is taking the pre-existing math and applying it in a useful manner. The second problem is probably much more interesting to this crowd, but I was concerned that most of us (myself included) don't recognize the first problem to the degree we should. The essay actually began as a rant against trusting too much in aggregate numbers, but at the end I was left with having to make a recommendation, and the best I could come up with -- the most practical solution that is easily explainable and implementable -- is A/B testing. Are there other options that people are using?

jcr 15y ago
> Out of all the startup skills I've studied, this one -- effectively inferring intent from reams of numbers -- is probably the most difficult. "There are only two types of people in this world; those who can extrapolate from incomplete information." It's a humorous way to get the same point across. It's particularly fun to say in person during a discussion, then just stop speaking and wait for the point to hit home. The more interesting question is why human beings get a warm, fuzzy feeling when they extrapolate from incomplete information (or extrapolate from incorrect information, or extrapolate incorrectly from correct information)? I haven't bothered to count them all, but there are plenty of myths about some deity dragging something bright or burning across the sky to explain the rising and setting of the sun.

noelwelsh 15y ago

Regarding your last point, we're trying to change things! As a economist you might be familiar with the bandit problem. We're bringing bandit algorithms to the web at http://www.mynaewb.com/ http://www.mynaewb.com/ Our results show a BIG improvement over A/B (see the blog).

jcr 15y ago

> I'm curious why I hear so much about A/B testing, but rarely hear about other statistical methods for assessing changes in usage patterns. Are they not being used? Am I simply looking in the wrong places? Or is A/B testing really sufficient in most cases? I believe part of the problem is you actually know the definition of "A/B Testing" and the real definition (testing a single, specific change) is mostly lost on the majority of those using the phrase. In other words, "A/B Testing" in common use has essentially become slang for "Multi-Variate Testing" and other forms of testing. For example, if you only change the color of a button, you're (formally) doing "A/B Testing" but if you change both the color and location of a button, you're (formally) doing "Multi-Variate Testing." None the less, you'll often see the later called the former in common usage.