3 ms·

If the average time to cancel is 61 days, and they waited only 2 months since the changes to calculate the new cancellation rates, it stands to reason that the

by ehsanu1 14y ago

If the average time to cancel is 61 days, and they waited only 2 months since the changes to calculate the new cancellation rates, it stands to reason that the cancellation rate will rise over the next few months, right? Assuming a normal distribution (probably not that accurate, but it's just a guess), the final cancellation rate would be about double that measured until now. Am I missing something here?

jamiequint 14y ago
The key here is that it is a cohort analysis. So while your analysis is correct and the cancellation rate will likely rise over time, the key thing to check when comparing a pre-change cohort to a post-change cohort to test for improvement is that the cancellation rates at two weeks post-signup are significantly different. Obviously its the most ideal if you can run the tests in parallel to reduce the risk of selection bias. However, sometimes that is not always feasible, and with a result like 85%+ improvement is not really necessary to do so in order to assume that the control was beaten.
- jmilloy 14y ago
  I don't really understand. Isn't the "cohort analysis" part of it how the data about the gyms that cancelled was extracted? That shouldn't affect how you compare the results once you've made changes. I'm also not sure I understand what you mean by "comparing a pre-change cohort to a post-change cohort" because we're not comparing the cohorts, we're comparing the cancellation rate among the whole list of customers.
  - jamiequint 14y ago
    Ah, the article is actually extremely unclear. What I took his conclusion to mean was that some cohort of gyms that had signed up post-change had lower cancellation rates than pre-change customers. If the analysis just took into account the overall rate across all customers is prone to all sorts of errors (including the one you mention).
    - ehsanu1 14y ago
      My impression was that the pre-change cancellation rate was based on all customers over the lifetime of the product (they did calculate the 61 day average from this), whereas the post-change was only for the new customers in the last two months (the max it could be, and the only thing it makes sense to measure).
losvedir 14y ago
I think I see what you're saying (given an average cancellation time, only half the customers will have cancelled by that time), but I think it depends on how they calculated it. For instance, they could know the average number of cancellations in a week, and see how that number has dropped. I think this would work out, provided it doesn't matter too much whether the two actions they've taken (the two emails) occur right when the gym signs up or later on. You're right that an average time to cancel of 61 days implies some may cancel at, say, 80 days. But some gym signed up 20 days before this cohort analysis began, and could have been the one to cancel had it not been for these changes.

by ehsanu1 14y ago

jamiequint 14y ago

The key here is that it is a cohort analysis. So while your analysis is correct and the cancellation rate will likely rise over time, the key thing to check when comparing a pre-change cohort to a post-change cohort to test for improvement is that the cancellation rates at two weeks post-signup are significantly different. Obviously its the most ideal if you can run the tests in parallel to reduce the risk of selection bias. However, sometimes that is not always feasible, and with a result like 85%+ improvement is not really necessary to do so in order to assume that the control was beaten.

jmilloy 14y ago
I don't really understand. Isn't the "cohort analysis" part of it how the data about the gyms that cancelled was extracted? That shouldn't affect how you compare the results once you've made changes. I'm also not sure I understand what you mean by "comparing a pre-change cohort to a post-change cohort" because we're not comparing the cohorts, we're comparing the cancellation rate among the whole list of customers.
- jamiequint 14y ago
  Ah, the article is actually extremely unclear. What I took his conclusion to mean was that some cohort of gyms that had signed up post-change had lower cancellation rates than pre-change customers. If the analysis just took into account the overall rate across all customers is prone to all sorts of errors (including the one you mention).
  - ehsanu1 14y ago
    My impression was that the pre-change cancellation rate was based on all customers over the lifetime of the product (they did calculate the 61 day average from this), whereas the post-change was only for the new customers in the last two months (the max it could be, and the only thing it makes sense to measure).

losvedir 14y ago

I think I see what you're saying (given an average cancellation time, only half the customers will have cancelled by that time), but I think it depends on how they calculated it. For instance, they could know the average number of cancellations in a week, and see how that number has dropped. I think this would work out, provided it doesn't matter too much whether the two actions they've taken (the two emails) occur right when the gym signs up or later on. You're right that an average time to cancel of 61 days implies some may cancel at, say, 80 days. But some gym signed up 20 days before this cohort analysis began, and could have been the one to cancel had it not been for these changes.