4 ms·

Almost. The difference is time, so the populations involved are 'the people that used the site last month' and 'the people that used the site last week'. Usua

Almost. The difference is time, so the populations involved are 'the people that used the site last month' and 'the people that used the site last week'. Usually your assumption is that these are comparable, but that's not necessarily true. Furthermore, the effect is only meaningful if that assumption is true, for most business cases (i.e. you want this effect to hold in the future). In practice, a lot of effects do disappear when you repeat a test because there was some unaccounted for factor that varied between them. It's a good sanity check. But you're right, much of the purpose is to discover mean regression. This is something that happens more often than you'd expect because you tend to be focusing large effects, many of which will simply be due to chance.

Vinnl 10y ago
So basically, it means waiting a while before performing the test again? Since "larger sample size" mostly just means "keep the test running for longer", so the difference is that in that case, the "second" test is very close in time to the "first" test?
- EdHominem 10y ago
  Not entirely. Even if run right after one another, running two tests buckets the results into early/late, which simply using a larger sample set does not. Assume your first samples show the effect and the later ones don't. The first test will show a strong correlation and the second not. The larger test will simply show a weaker correlation.
  - Vinnl 10y ago
    So what would the conclusions be in the two-test and large-test case? My assumption would be that if one of the two tests would fail, then the correlation in a large test would also have been so low as to mean no effect?
    - EdHominem 10y ago
      The issue is that you probably wouldn't do two tests if you'd planned a short one. So you'd randomly get an overly strong, or overly weak, result. The actual effect depends on your data and how noisy it is, and if that correlates to external factors, etc. You need to know the characteristics of your data then you can decide on your desired confidence and only then pick a test designed to give you that based on your data.
- tedsanders 10y ago
  Not asking me, but yes. I think the underlying issue is whether you assume the process that generated the data yesterday is similar enough to the process that generated data today. In truth, all data points are unique and all data is high dimensional data. It only becomes workable in low dimensions once we make assumptions like "the process that produced this point yesterday is the same as the process that produced this second days point today." Separating the tests makes it slightly less difficult to test for violations of these assumptions (e.g., model drift). But otherwise it seems silly and arbitrary to commit yourself you two medium tests instead of one big test. At least two medium tests let you filter out likely failures if you allow yourself to quit midway. (This whole business of tests and halting is silly anyhow.)
  - Vinnl 10y ago
    Right, thanks. And am I correct in thinking that your final sentence before the parentheses means that two medium tests are worse, because you don't want to filter out likely failures?

Vinnl 10y ago

So basically, it means waiting a while before performing the test again? Since "larger sample size" mostly just means "keep the test running for longer", so the difference is that in that case, the "second" test is very close in time to the "first" test?

EdHominem 10y ago
Not entirely. Even if run right after one another, running two tests buckets the results into early/late, which simply using a larger sample set does not. Assume your first samples show the effect and the later ones don't. The first test will show a strong correlation and the second not. The larger test will simply show a weaker correlation.
- Vinnl 10y ago
  So what would the conclusions be in the two-test and large-test case? My assumption would be that if one of the two tests would fail, then the correlation in a large test would also have been so low as to mean no effect?
  - EdHominem 10y ago
    The issue is that you probably wouldn't do two tests if you'd planned a short one. So you'd randomly get an overly strong, or overly weak, result. The actual effect depends on your data and how noisy it is, and if that correlates to external factors, etc. You need to know the characteristics of your data then you can decide on your desired confidence and only then pick a test designed to give you that based on your data.
tedsanders 10y ago
Not asking me, but yes. I think the underlying issue is whether you assume the process that generated the data yesterday is similar enough to the process that generated data today. In truth, all data points are unique and all data is high dimensional data. It only becomes workable in low dimensions once we make assumptions like "the process that produced this point yesterday is the same as the process that produced this second days point today." Separating the tests makes it slightly less difficult to test for violations of these assumptions (e.g., model drift). But otherwise it seems silly and arbitrary to commit yourself you two medium tests instead of one big test. At least two medium tests let you filter out likely failures if you allow yourself to quit midway. (This whole business of tests and halting is silly anyhow.)
- Vinnl 10y ago
  Right, thanks. And am I correct in thinking that your final sentence before the parentheses means that two medium tests are worse, because you don't want to filter out likely failures?