5 ms·

Maybe people like fresh things? Here's an A/B test I recently ran on some notification messages. There were two messages, only difference is the word "new" in

by bemmu 10y ago

Maybe people like fresh things? Here's an A/B test I recently ran on some notification messages. There were two messages, only difference is the word "new" in there. A: "You received a message from the admins about your Facebook page." B: "You received a new message from the admins about your Facebook page." A clickthrough 15.64%, B clickthrough 15.95%. About 1000 samples per group. Not sure if noise.

nothrabannosir 10y ago
.3% on 1k? That's three people.. Yes, noise.
- nkrisc 10y ago
  That's a great way to get a gut check on the validity of the result. Put it in real terms.
- msellout 10y ago
  I'm often annoyed when reading reports that use percentages instead of the actual ratios. For example, "2 out of 3" is much better than "66.7%".
pjc50 10y ago
https://www.mccallum-layton.co.uk/tools/statistic-calculators/confidence-interval-for-proportions-calculator/ https://www.mccallum-layton.co.uk/tools/statistic-calculator... Sample Size: 1000 Observed Proportion: 15.64% Confidence Level: 95% Confidence Interval: ±2.25 Range for the true population proportion: 13.39% to 17.89% It's noise.
- bemmu 10y ago
  Hmm, I tried playing around with this tool. When I enter 250000 in sample size I get ranges that don't overlap. Does that mean I would need that many samples to answer this?
  - cscheid 10y ago
    Yes. The bigger your sample size, the bigger your ability to see small effects. A test with 1000 elements can only tell apart big effects; a test with 250000 can tell apart much smaller ones. This is the idea of https://en.wikipedia.org/wiki/Statistical_power https://en.wikipedia.org/wiki/Statistical_power
  - deleted 10y ago
    [deleted]