5 ms·

> I’m very, very open to the notion that there exist circumstances under which this test would have come down the other way. There almost certainly are circums

> I’m very, very open to the notion that there exist circumstances under which this test would have come down the other way. There almost certainly are circumstances in which result would have been reversed. Testing the hypothesis that logo A is better than logo B, he did get a significant result (n = 17,000, p < 0.01). But if he was attempting to test the hypothesis that logos from population A (custom-designed) are better than logos from population B (off-the-rack), his results are completely insignificant, since n in that case was 2. You'd have to test many more logos to reach that sort of conclusion.

patio11 16y ago
OK, quick stats 101: The hypothesis isn't that A > B, it is that A = B (the null hypothesis). 99% certainty is to reject the null hypothesis, leading to the conclusion that A is better. The second experiment you've outlined is gibberish, which involves a series of words spelled similar to ones you can find in a stats book, connected in a manner which leads to nowhere.
- nollidge 16y ago
  What's gibberish about it? I'm saying if you want to know if off-the-rack logos are, on average, better than custom-designed ones, you have to compare lots of them, not just one example of each. What's the problem with that?
  - patio11 16y ago
    Let's not talk A/B tests and talk medicine for a moment, shall we? Here we have aspirin, here we have a placebo. There isn't a population of aspirin pills and a population of placebo pills. There aren't many different populations of placebo pills. The populations are of people. We're trying to make inferences about the population of people exposed to aspirin versus the population of people exposed to a sugar pill, but since it is not possible to bring all of them in, we make those inferences based on random samples of those two populations. Now here is where I start getting murky: results for the particular things we're interested in are binary: the aspirin worked or it didn't. You converted or you didn't. The Central Limit Theorem says that, if you sample n individual unconnected random binary events from a population, and you keep repeating that sample, the distribution of your measured percentage of success will approximate the normal distribution, centered around (with the mean on) the true population statistic. That is important for a couple of reasons, but the most immediate one is that it gives us a statistical figleaf to hide under for using two samples to make inferences about whether the population mean for the first distribution is highly likely to not be the same as the population mean for the second distribution. i.e. whether the A and the B are likely to be materially different. So let's walk this back to where we were talking about potentially testing multiple different types of sugar pill. Do you get why that is sort of incoherent in this framework? There are key assumptions like "independence" that two different types sugar pills surely do not share, but I don't know if you can even get to that objection before you deal with the notion that there is no coherently cognizable population made out of sugar pills. It is 4 AM in the morning. I am not sure that is stats class talking and not sleep deprivation. This is probably covered in a good Stats 101 book -- if you have further questions, they're likely more coherent than I am. (Or ask btilly -- he has a gift for explaining this.)
    - nollidge 16y ago
      I've got Stats 101 knowledge, biostats in fact, but perhaps I've misapplied it here. I'm at work now, so I'll maybe revisit my thinking later.

patio11 16y ago

OK, quick stats 101: The hypothesis isn't that A > B, it is that A = B (the null hypothesis). 99% certainty is to reject the null hypothesis, leading to the conclusion that A is better. The second experiment you've outlined is gibberish, which involves a series of words spelled similar to ones you can find in a stats book, connected in a manner which leads to nowhere.

nollidge 16y ago
What's gibberish about it? I'm saying if you want to know if off-the-rack logos are, on average, better than custom-designed ones, you have to compare lots of them, not just one example of each. What's the problem with that?
- patio11 16y ago
  Let's not talk A/B tests and talk medicine for a moment, shall we? Here we have aspirin, here we have a placebo. There isn't a population of aspirin pills and a population of placebo pills. There aren't many different populations of placebo pills. The populations are of people. We're trying to make inferences about the population of people exposed to aspirin versus the population of people exposed to a sugar pill, but since it is not possible to bring all of them in, we make those inferences based on random samples of those two populations. Now here is where I start getting murky: results for the particular things we're interested in are binary: the aspirin worked or it didn't. You converted or you didn't. The Central Limit Theorem says that, if you sample n individual unconnected random binary events from a population, and you keep repeating that sample, the distribution of your measured percentage of success will approximate the normal distribution, centered around (with the mean on) the true population statistic. That is important for a couple of reasons, but the most immediate one is that it gives us a statistical figleaf to hide under for using two samples to make inferences about whether the population mean for the first distribution is highly likely to not be the same as the population mean for the second distribution. i.e. whether the A and the B are likely to be materially different. So let's walk this back to where we were talking about potentially testing multiple different types of sugar pill. Do you get why that is sort of incoherent in this framework? There are key assumptions like "independence" that two different types sugar pills surely do not share, but I don't know if you can even get to that objection before you deal with the notion that there is no coherently cognizable population made out of sugar pills. It is 4 AM in the morning. I am not sure that is stats class talking and not sleep deprivation. This is probably covered in a good Stats 101 book -- if you have further questions, they're likely more coherent than I am. (Or ask btilly -- he has a gift for explaining this.)
  - nollidge 16y ago
    I've got Stats 101 knowledge, biostats in fact, but perhaps I've misapplied it here. I'm at work now, so I'll maybe revisit my thinking later.