3 ms·

You can calculate the significance using the difference in differences method popular in econometrics. I derived this result for a previous employer and it work

You can calculate the significance using the difference in differences method popular in econometrics. I derived this result for a previous employer and it worked well. I can look up my notes and explain the method in more detail if anyone is interested.

asbromberg 10y ago
I would definitely be interested in hearing about this if you have the time.
wojcech 10y ago
I'd be very interested if you don't mind:)
wdewind 10y ago
Yes please
- samscully 10y ago
  Found my notes on difference in differences method. First model the sessions originating on page [i] at time [t] as: S[it] = b0 + b1 * X[it] + b2 * A[i] + b3 * D[it] + e Where: S[it]: number of sessions originating on page [i] at time [t] b0: base traffic for all pages A[i]: base traffic for page [i] X[it]: seasonal traffic for page [i] at time [t] D[it]: dummy variable, 1 for after treatment 0 for control and before treatment b3: effect size of treatment e: noise/error term Taking the average over time: E[S[i]] = b0 + b1 * E[X[i]] + b2 * A[i] + b3 * E[D[i]] + e Take the average from S[it] (the first "differences"): S[it] - E[S[i]] = (b0 - b0) + b1(X[it] - E[X[i]]) + b2(A[i] - A[i]) + b3(D[it] - E[D[i]]) + e which simplifies to: S[it] - E[S[i]] = b1(X[it] - E[X[i]]) + b3(D[it] - E[D[i]]) + e Split all pages [i] in to control and treatment groups 0 and 1. Take the "difference in differences" of the two groups: (S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b1(X[1t] - E[X[1]] - X[0t] + E[X[0]]) + b3(D[1t] - E[D[1]] - D[0t] + E[D[0]]) + e Given D[0t] = E[D[0]] = 0, and for a large number of pages X[it] - E[X[i]] = X[jt] - E[X[j]]: (S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b3(D[1t] - E[D[1]]) + e Which gives a formula in the form of y = mx from which we can calculate the effect size b3 and the p value using linear regression. The key assumption is that X[it] - E[X[i]] = X[jt] - E[X[j]] for large numbers of pages. I found this assumption does hold in practice. I'm not a statistician so forgive me if I've made any statistical errors but I think this analysis is correct. I might write up a blog post at some point to explain it in more detail.
  - wdewind 10y ago
    Thank you!

asbromberg 10y ago

I would definitely be interested in hearing about this if you have the time.

wojcech 10y ago

I'd be very interested if you don't mind:)

wdewind 10y ago

Yes please

samscully 10y ago
Found my notes on difference in differences method. First model the sessions originating on page [i] at time [t] as: S[it] = b0 + b1 * X[it] + b2 * A[i] + b3 * D[it] + e Where: S[it]: number of sessions originating on page [i] at time [t] b0: base traffic for all pages A[i]: base traffic for page [i] X[it]: seasonal traffic for page [i] at time [t] D[it]: dummy variable, 1 for after treatment 0 for control and before treatment b3: effect size of treatment e: noise/error term Taking the average over time: E[S[i]] = b0 + b1 * E[X[i]] + b2 * A[i] + b3 * E[D[i]] + e Take the average from S[it] (the first "differences"): S[it] - E[S[i]] = (b0 - b0) + b1(X[it] - E[X[i]]) + b2(A[i] - A[i]) + b3(D[it] - E[D[i]]) + e which simplifies to: S[it] - E[S[i]] = b1(X[it] - E[X[i]]) + b3(D[it] - E[D[i]]) + e Split all pages [i] in to control and treatment groups 0 and 1. Take the "difference in differences" of the two groups: (S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b1(X[1t] - E[X[1]] - X[0t] + E[X[0]]) + b3(D[1t] - E[D[1]] - D[0t] + E[D[0]]) + e Given D[0t] = E[D[0]] = 0, and for a large number of pages X[it] - E[X[i]] = X[jt] - E[X[j]]: (S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b3(D[1t] - E[D[1]]) + e Which gives a formula in the form of y = mx from which we can calculate the effect size b3 and the p value using linear regression. The key assumption is that X[it] - E[X[i]] = X[jt] - E[X[j]] for large numbers of pages. I found this assumption does hold in practice. I'm not a statistician so forgive me if I've made any statistical errors but I think this analysis is correct. I might write up a blog post at some point to explain it in more detail.
- wdewind 10y ago
  Thank you!