3 ms·
Agreed it's super hard. I think that's why you need to be careful about looking at data until you've got your hypothesis. Too much data and you'll overfit or in
by camwest 3y ago
Agreed it's super hard. I think that's why you need to be careful about looking at data until you've got your hypothesis. Too much data and you'll overfit or introduce bias. I think of it more as a way to update your gut with an updated map of the state.
- bcrosby95 3y agoI haven't done this in a while, but somewhere in my notes I have a formula to determine sample size based upon power and the effect we want to detect. After we hit that size we would analyze the results. If they were ambiguous we just picked the one we subjectively felt was the better choice.
- criddell 3y agoI'm guessing you would have have an idea of what the right answer was going to be before you ran the analysis. How often was your guess wrong? How often did the data tell you to do something that you were uncomfortable with? Did you ever use your taste or judgement to go against what the algorithm said you should do?
- bcrosby95 3y agoWe had guesses, but nothing particularly informed. My recollection is that we were right more often than not in our guesses, but it's been a while. We only ran experiments on changes we were willing to make. We never went against conclusive results due to judgement. We also used data a bit less formally. For example, we had something similar to Google analytics' "funnel" feature of how people arrived at some of our pages, and we noticed a usage pattern that indicated we were missing a feature. When we added the feature directly it was one of our most used features.