3 ms·

I don't think this is statistically significant.

• steve-benjamins 6y ago
  Hi, I'm the author— this is 98% confidence.
  • pottertheotter 6y ago
    I think people are probably more concerned that it doesn't have economic significance. It might have statistical significance, but the effect is still very small, so does it really matter? That's a common trap people forget to consider. You see this a lot in finance research where some variable is statistically significant in a model, but the difference in the economic outcome is so small that it doesn't matter. In this case, these are indeed tiny percentages. But you're going from something that a lot of people dislike and that is more complicated (from a technical standpoint), so we can simplify things and user interaction with live chat is not impacted.
    • zamalek 6y ago
      You'd hope that the percentages here would be small: people need to be facing a problem with your product before they are part of the sample population. There is a lot of potential economic impact here; during a major problem (which WILL happen) the number of people looking for support would increase sharply. If people are blind to your support system, you could be saying goodbye to a large amount of customers. Ignoring stuff like this is a far more common trap. We spend a lot of money on firefighters, even though we hope they never have any fires to fight.
• solumos 6y ago
  I disagree, I think it's statistically significant. I ran a quick z-test: p0 = 34 p1 = 45 n0 = 8004 n1 = 6622 z = -2.0924 p = 0.03662 So, the improvement is statistically significant at 95% confidence (p < 0.05) As far as practical significance, that's debatable . . .
  • qyph 6y ago
    A z test relies on the normal approximation, no? I don't think that is appropriate with proportions so close to 0.