4 ms·
Such a plot really needs error bars. It's possible that the difference is not statistically significant.
by michaelhoffman 15y ago
Such a plot really needs error bars. It's possible that the difference is not statistically significant.
- Jabbles 15y agoError bars on what? The ratings themselves can be measured with perfect accuracy, and the results were averaged over 5000 merchants, so any error in the mean would be too small to see.
- orijing 15y agoIt's true that the average clearly moved, but the author was trying to make a generalized inference from the average, that there was a structural break in the "true average" insofar as the measured average is just a sample/noisy measurement of the "true average". Therefore, error bars/confidence bars would indicate how likely the structural break in measured outcomes is to reflect a structural break in the underlying "true average." The volatility/variance comes from the fact that 1. not everyone rates their experience, 2. there's variability in the experience, 3. different people have different expectations or likelihood to rate high/low, etc. I hope that clarified what I believe was the parent's meaning.
- MBCook 15y agoThat break at day 0 was because they reset the moving average calculation on that date.
- eggspurt 15y agoWith error bars/lines on that plot, this discussion wouldn't have been needed. There are nice ways of doing error contours - showing the 1% quantile lines on top and on the bottom.
- gzervas 15y agoYou're right, statistical significance is a valid concern but error bars is not the way to go. Check the paper for a rigorous analysis. We are not drawing our conclusions from a single plot.
- michaelhoffman 15y agoI didn't think that you were (and having glanced at the paper now, there seems to be a lot more meat to it), but I think others will see only the plot. In my opinion, such a plot is misleading given the large variance I would expect in the rating.