4 ms·

I hypothesize that many of these statistical tests have led to worse outcomes. Not in the commonly espoused bad for society externality view, but as a straight

I hypothesize that many of these statistical tests have led to worse outcomes. Not in the commonly espoused bad for society externality view, but as a straight up bad for business view. There are three problems I see with A/B tests on a vast scale for small tests. 1. If the effect size is incredibly small, which most minor UI changes will be, finding statistical tests to prove them is really difficult. If you’re looking for an incredibly small positive effect, even with hundreds of thousands of sample size, the probability of rejecting the null hypothesis while the effect is actually a negative influence is surprisingly high! Very easy to make mistakes. 2) short term gains on engagement may lead to long term disengagement. 3) business incentives for management are easily misaligned. I would imagine a dominant negative influence is managers exaggerating the statistical influence found in a test because that means they get to lead the change, on an otherwise vast tech ecosystem the performance of which probably won’t change all that much. Attribution is also hard (how sure are they on how much to attribute here?) so credit is difficult to allocate beyond initial value sizing.

CrazyCatDog 8y ago
I literally did a research project for a firm recently where we lowered wages. Workers are super monitored, I have data on every minute of their day. Post wage cut, workers worked just as hard. Awesome! 5 months later, the best people are (significantly) gone. 7 months later, and it is demonstrably Obvious that this was a value destroying move--average fixed effects look horrible now, yet this is only clear to those of us watching from the outside. Internally, the difference is completely overlooked. I needed to find cites, and ironically I found the Parent article this morning. This one is way better about the lies we tell ourselves: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791
- b_tterc_p 8y ago
  How do you measure people like that?
jlinowski 8y ago
1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes have tiny effects, and therefore not detectable and a waste of time, is a flawed assumption. Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect). More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation. 2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.

CrazyCatDog 8y ago

I literally did a research project for a firm recently where we lowered wages. Workers are super monitored, I have data on every minute of their day. Post wage cut, workers worked just as hard. Awesome! 5 months later, the best people are (significantly) gone. 7 months later, and it is demonstrably Obvious that this was a value destroying move--average fixed effects look horrible now, yet this is only clear to those of us watching from the outside. Internally, the difference is completely overlooked. I needed to find cites, and ironically I found the Parent article this morning. This one is way better about the lies we tell ourselves: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791

b_tterc_p 8y ago
How do you measure people like that?

jlinowski 8y ago

1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes have tiny effects, and therefore not detectable and a waste of time, is a flawed assumption. Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect). More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation. 2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.