6 ms·

How to better calculate churn rates

166 points by cmogni1 6y ago

encoderer 6y ago
It stands out that it's using customer count and not revenue churn. In a well running business you might see shrinking customer count but net-negative churn because you're growing revenue on the accounts that stay with you. This is really very common in any SaaS that gets big. Calculating the customer churn is valuable for LTV computation but it doesn't seem as useful as a KPI for a growing saas business as net revenue churn rate.
- davidivadavid 6y ago
  Exactly. It's kind of well-known at this point that negative net dollar churn is what people should aim for. The minutiae of risk modelling for something as inherently volatile seem beside the point. Maybe if you're a stock analyst?
  - mathattack 6y ago
    For small sample sizes (enterprise customers) it’s tricky to get too crazy on stats. Simple models are better.
    - jrott 6y ago
      practically if you are dealing with large enterprise customers it's better to use qualitative data from your account managers to figure out how much risk there is with each customer.
      - mathattack 6y ago
        My experience is “they love us” isn’t as strong a retention predictor as cold hard usage metrics.
        jrott 6y ago
        They love us seems to never be that useful. I actually meant more in terms of there worried about foo or finding out about the dumb workaround there using. That stuff seems to be an early warning sign that something isn’t going well.
baron_harkonnen 6y ago
It honestly delights me to see someone correctly point out that churn is a classic example of survival analysis problem. It's shocking how many data scientist I've known how no idea how to correctly model churn, typically trying to build some predictive model, which is just a more sophisticated extension of first formula in the article. It's insane how many business problems are really survival analysis problems where you have some hazard function and censored data. In my experience having data scientists that can correctly identify this is very rare (and a great sign of how broken that entire field is right now).
- qsort 6y ago
  Not going to argue with you that the field isn't broken -- it very clearly is -- but the problem the article explains is a rather simple one. Your setting could be non-contractual. You might not be able to have reliable data to frame the problem as survival analysis with censored data. You might have different groups of customers with different behaviors. Assuming a constant failure rate is the same as assuming the underlying process is Poisson, which is not an entirely stupid working hypothesis in many cases. The moral of the story is "by the love of god look at your data before making dumb models".
  - sebastianmarkow 6y ago
    Different population behaviours could be modeled via multi-level models.
    - qsort 6y ago
      They can, but you start running into problems with the mathematical formulation of your model. For example, Weibull's CF doesn't have a closed form, which means it's hard to find convolutions even with itself, so that finding e.g. the sum of Weibull distributed r.v. is hard, even if they are iid. Again, modeling done right is super hard.
- sebastianmarkow 6y ago
  This is the comment I was hoping to find here. I can only recommend reading McElreath’s Statistical Rethinking Book/Course in that context. https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/
mathattack 6y ago
The key here is cohort analysis. If you are on 1 year contracts, then the relevant denominator is revenue 12 months ago, and the numerator is revenue from those customers. Just like NPS there can be a lot of intellectual dishonesty.
dkulchenko 6y ago
I recently modeled churn for my subscription box calmbox (https://thecalmbox.com https://thecalmbox.com) by using an actuarial table (http://www.lifeexpectancy.org/lifetable.shtml http://www.lifeexpectancy.org/lifetable.shtml, formulas at the bottom) which I found to be a lot more informative than the standard churn formula. Subscriptions still active at month x are represented as `l(x)` and subscriptions that "die" (cancel/expire) in a given month are represented as `d(x)`. This gives you a "life expectancy" and a "mortality rate" (so, churn) for any given number of months that a customer has been subscribed. So I can project how long someone will stay subscribed when they're brand new (at month 0) and how long they likely still have when they're at month 8 (longer than at month 0, funnily enough). With those subscriptions where the month-specific churn will largely decrease the longer someone's subscribed (after passing the initial high-churn first months), this allows measuring/projecting churn on a much more granular level.
- bmcahren 6y ago
  Segmenting this data by signup date is useful as well. If I'm not mistaken, you're aligning all time periods to the same imaginary start date. This gives you the grand perspective but ignores very real changes in your application, team, marketplace, advertising, and product-fit over time.
  - xyzzy_plugh 6y ago
    Yes, you should absolutely segment into cohorts. Ideally you can tag users such that you can slice them not just by time, but what funnel they used or what marketing they've been exposed to (emails, etc.) Just measuring the grand perspective, as you say, can be a very poor indicator for what is working well, and what isn't.
rubyn00bie 6y ago
I quite liked the more technical parts of the article and practical modelling tips for customer churn rates. But... I would more interested in a focused, more terse, article just examining how the modelling techniques you use can impact your understanding of a given data set. The bullshitting for the premise just sort of kept me distracted... Unless a16z updated their page recently enough to be different, they very plainly say there are multiple types of churn, then provide two clear examples, here's the first sentence about churn: > There’s all kinds of churn — dollar churn, customer churn, net dollar churn — and there are varying definitions for how churn is measured. I also find it a bit of a disservice they don't discuss other churn types while dismissing the notion completely in the introduction. Statistics is a really "it depends" kind of subject and if you don't put forth good effort to explain the assumptions it can be really hard to follow and even harder to correctly apply. Next, I find it a bit off-putting they mostly modeled their way out of the problem instead of addressing the bigger meta question "how do I find out if customer churn is a meaningful metric for MY business?" This is a big assumption made by the article (customer churn modeled correctly is useful) but not supported by it. Finally, I find this in the conclusion to be quite simply HI-LARIOUS for how brazen it is; while, IMHO, kind of missing their own point: > The numbers we've just computed are perhaps the most central of all for a subscription business, yet it seems like most people have never been thought how to properly compute them.
cpard 6y ago
Calculating churn is one of these things that there's a complete dichotomy on the web when it comes to the available information. You usually find very simplistic approaches to a quite complex problem and there's amazing research that is not easily accessible. The article is going towards the right direction but as other commenters mentioned, churn is an example of a survival analysis problem. A typical model used for modeling churn is pareto/nbd. Wikipedia is a great starting point https://en.wikipedia.org/wiki/Buy_Till_you_Die https://en.wikipedia.org/wiki/Buy_Till_you_Die
deleted 6y ago
[deleted]
andrenotgiant 6y ago
Something I've never gotten a good answer to: Say I have two groups of 1000 customers (e.g. US customers vs UK customers) and I want to compare the churn rates between them. BUT these are users of varying ages, they didn't all sign up at the same month. It could be that one group is slightly younger or older than the other. How do you compare churn rate of groups that aren't cohorted by age?
- dcl 6y ago
  You can try the proportional hazard model https://en.wikipedia.org/wiki/Proportional_hazards_model https://en.wikipedia.org/wiki/Proportional_hazards_model In this setting, things like age, gender, or other features of interest are the covariates/predictors. You can then also use indicator features like UK or US for geography. If the geography based features are significant, then the difference in churn between the regions is perhaps not simply due to age etc.
Jorge1o1 6y ago
Great article but even this article contains some minor nitpicky inaccuracies in modeling churn. The Lomax distribution (sometimes described as the gamma-exponential) would only be appropriate if we're modeling a continuous time process. For a contractual product with discrete time intervals (aka monthly contract), it would be more accurate to use a geometric rather than exponential model such as the Beta-Geometric.
- numlocked 6y ago
  Bingo! The classic work on this is from Fader and Hardie: https://faculty.wharton.upenn.edu/wp-content/uploads/2012/04/Fader_hardie_jim_07.pdf https://faculty.wharton.upenn.edu/wp-content/uploads/2012/04... A friend and I made an implementation in Python a while back: https://github.com/chrisclark/retentionizer https://github.com/chrisclark/retentionizer
  - jwithington 6y ago
    Super glad to find this! Have you also seen the lifetimes library? https://github.com/CamDavidsonPilon/lifetimes https://github.com/CamDavidsonPilon/lifetimes
frankcort 6y ago
Seems like something Profitwell or Chartmogul should add to their offerings. We rely heavily on Chartmogul for our weekly stats/scorecard and it looks like we've been wrongly celebrating our "super low churn rate" for a while now!
pauldickwin 6y ago
When you let marketing people do the math, they come up with the "constant" churn equation.
- malshe 6y ago
  Both Pete Fader and Bruce Hardy are "marketing people": https://marketing.wharton.upenn.edu/profile/faderp/ https://marketing.wharton.upenn.edu/profile/faderp/ http://brucehardie.com http://brucehardie.com
paulstovell 6y ago
Absolutely not an expert here, but I do obsess about our churn numbers. It seems this post is trying to make a model to predict churn or guess what future churn might be. But the main case for understanding churn to me is historical. Last month we had 10 customers paying us $10 each. Two of them cancelled. I have gross churn of $20. That's not a model, just a historical fact. What's "wrong" with that calculation that these models solve? I will definitely get different churn numbers if I look at groups of customers in cohorts based on when they first signed up, or enterprise-vs-SMB, or whatever. So I have reports to create all of those cohorts and show me the churn for each. I look at that data - with all the historical context - and make a decision about what to do next in the business. "Selling to SMB's was a good idea, but wow they churn a lot more. Let's focus our marketing on enterprises this month." To me the churn calculation at the top of the article is plenty useful. I'm not sure what value I'd get from these advanced models. If they just exist to predict future churn or LTV, that doesn't seem particularly useful.
- paulstovell 6y ago
  Also if the problem is "churn as a single number" is wrong because it misleads people (the example being churn is up because the company is growing) surely the answer is to show a cohort analysis or historical data - not a forward looking or "best fit" model?
- onlyrealcuzzo 6y ago
  Predicting LTV is quite useful for marketing and advertising.

166 points by cmogni1 6y ago

It stands out that it's using customer count and not revenue churn. In a well running business you might see shrinking customer count but net-negative churn because you're growing revenue on the accounts that stay with you. This is really very common in any SaaS that gets big. Calculating the customer churn is valuable for LTV computation but it doesn't seem as useful as a KPI for a growing saas business as net revenue churn rate.

davidivadavid 6y ago
Exactly. It's kind of well-known at this point that negative net dollar churn is what people should aim for. The minutiae of risk modelling for something as inherently volatile seem beside the point. Maybe if you're a stock analyst?
- mathattack 6y ago
  For small sample sizes (enterprise customers) it’s tricky to get too crazy on stats. Simple models are better.
  - jrott 6y ago
    practically if you are dealing with large enterprise customers it's better to use qualitative data from your account managers to figure out how much risk there is with each customer.
    - mathattack 6y ago
      My experience is “they love us” isn’t as strong a retention predictor as cold hard usage metrics.
      - jrott 6y ago
        They love us seems to never be that useful. I actually meant more in terms of there worried about foo or finding out about the dumb workaround there using. That stuff seems to be an early warning sign that something isn’t going well.

baron_harkonnen 6y ago

It honestly delights me to see someone correctly point out that churn is a classic example of survival analysis problem. It's shocking how many data scientist I've known how no idea how to correctly model churn, typically trying to build some predictive model, which is just a more sophisticated extension of first formula in the article. It's insane how many business problems are really survival analysis problems where you have some hazard function and censored data. In my experience having data scientists that can correctly identify this is very rare (and a great sign of how broken that entire field is right now).

qsort 6y ago
Not going to argue with you that the field isn't broken -- it very clearly is -- but the problem the article explains is a rather simple one. Your setting could be non-contractual. You might not be able to have reliable data to frame the problem as survival analysis with censored data. You might have different groups of customers with different behaviors. Assuming a constant failure rate is the same as assuming the underlying process is Poisson, which is not an entirely stupid working hypothesis in many cases. The moral of the story is "by the love of god look at your data before making dumb models".
- sebastianmarkow 6y ago
  Different population behaviours could be modeled via multi-level models.
  - qsort 6y ago
    They can, but you start running into problems with the mathematical formulation of your model. For example, Weibull's CF doesn't have a closed form, which means it's hard to find convolutions even with itself, so that finding e.g. the sum of Weibull distributed r.v. is hard, even if they are iid. Again, modeling done right is super hard.
sebastianmarkow 6y ago
This is the comment I was hoping to find here. I can only recommend reading McElreath’s Statistical Rethinking Book/Course in that context. https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/

mathattack 6y ago

The key here is cohort analysis. If you are on 1 year contracts, then the relevant denominator is revenue 12 months ago, and the numerator is revenue from those customers. Just like NPS there can be a lot of intellectual dishonesty.

dkulchenko 6y ago

I recently modeled churn for my subscription box calmbox (https://thecalmbox.com https://thecalmbox.com) by using an actuarial table (http://www.lifeexpectancy.org/lifetable.shtml http://www.lifeexpectancy.org/lifetable.shtml, formulas at the bottom) which I found to be a lot more informative than the standard churn formula. Subscriptions still active at month x are represented as `l(x)` and subscriptions that "die" (cancel/expire) in a given month are represented as `d(x)`. This gives you a "life expectancy" and a "mortality rate" (so, churn) for any given number of months that a customer has been subscribed. So I can project how long someone will stay subscribed when they're brand new (at month 0) and how long they likely still have when they're at month 8 (longer than at month 0, funnily enough). With those subscriptions where the month-specific churn will largely decrease the longer someone's subscribed (after passing the initial high-churn first months), this allows measuring/projecting churn on a much more granular level.

bmcahren 6y ago
Segmenting this data by signup date is useful as well. If I'm not mistaken, you're aligning all time periods to the same imaginary start date. This gives you the grand perspective but ignores very real changes in your application, team, marketplace, advertising, and product-fit over time.
- xyzzy_plugh 6y ago
  Yes, you should absolutely segment into cohorts. Ideally you can tag users such that you can slice them not just by time, but what funnel they used or what marketing they've been exposed to (emails, etc.) Just measuring the grand perspective, as you say, can be a very poor indicator for what is working well, and what isn't.

rubyn00bie 6y ago

I quite liked the more technical parts of the article and practical modelling tips for customer churn rates. But... I would more interested in a focused, more terse, article just examining how the modelling techniques you use can impact your understanding of a given data set. The bullshitting for the premise just sort of kept me distracted... Unless a16z updated their page recently enough to be different, they very plainly say there are multiple types of churn, then provide two clear examples, here's the first sentence about churn: > There’s all kinds of churn — dollar churn, customer churn, net dollar churn — and there are varying definitions for how churn is measured. I also find it a bit of a disservice they don't discuss other churn types while dismissing the notion completely in the introduction. Statistics is a really "it depends" kind of subject and if you don't put forth good effort to explain the assumptions it can be really hard to follow and even harder to correctly apply. Next, I find it a bit off-putting they mostly modeled their way out of the problem instead of addressing the bigger meta question "how do I find out if customer churn is a meaningful metric for MY business?" This is a big assumption made by the article (customer churn modeled correctly is useful) but not supported by it. Finally, I find this in the conclusion to be quite simply HI-LARIOUS for how brazen it is; while, IMHO, kind of missing their own point: > The numbers we've just computed are perhaps the most central of all for a subscription business, yet it seems like most people have never been thought how to properly compute them.

cpard 6y ago

Calculating churn is one of these things that there's a complete dichotomy on the web when it comes to the available information. You usually find very simplistic approaches to a quite complex problem and there's amazing research that is not easily accessible. The article is going towards the right direction but as other commenters mentioned, churn is an example of a survival analysis problem. A typical model used for modeling churn is pareto/nbd. Wikipedia is a great starting point https://en.wikipedia.org/wiki/Buy_Till_you_Die https://en.wikipedia.org/wiki/Buy_Till_you_Die

deleted 6y ago

[deleted]

andrenotgiant 6y ago

Something I've never gotten a good answer to: Say I have two groups of 1000 customers (e.g. US customers vs UK customers) and I want to compare the churn rates between them. BUT these are users of varying ages, they didn't all sign up at the same month. It could be that one group is slightly younger or older than the other. How do you compare churn rate of groups that aren't cohorted by age?

dcl 6y ago
You can try the proportional hazard model https://en.wikipedia.org/wiki/Proportional_hazards_model https://en.wikipedia.org/wiki/Proportional_hazards_model In this setting, things like age, gender, or other features of interest are the covariates/predictors. You can then also use indicator features like UK or US for geography. If the geography based features are significant, then the difference in churn between the regions is perhaps not simply due to age etc.

Jorge1o1 6y ago

Great article but even this article contains some minor nitpicky inaccuracies in modeling churn. The Lomax distribution (sometimes described as the gamma-exponential) would only be appropriate if we're modeling a continuous time process. For a contractual product with discrete time intervals (aka monthly contract), it would be more accurate to use a geometric rather than exponential model such as the Beta-Geometric.

numlocked 6y ago
Bingo! The classic work on this is from Fader and Hardie: https://faculty.wharton.upenn.edu/wp-content/uploads/2012/04/Fader_hardie_jim_07.pdf https://faculty.wharton.upenn.edu/wp-content/uploads/2012/04... A friend and I made an implementation in Python a while back: https://github.com/chrisclark/retentionizer https://github.com/chrisclark/retentionizer
- jwithington 6y ago
  Super glad to find this! Have you also seen the lifetimes library? https://github.com/CamDavidsonPilon/lifetimes https://github.com/CamDavidsonPilon/lifetimes

frankcort 6y ago

Seems like something Profitwell or Chartmogul should add to their offerings. We rely heavily on Chartmogul for our weekly stats/scorecard and it looks like we've been wrongly celebrating our "super low churn rate" for a while now!

pauldickwin 6y ago

When you let marketing people do the math, they come up with the "constant" churn equation.

malshe 6y ago
Both Pete Fader and Bruce Hardy are "marketing people": https://marketing.wharton.upenn.edu/profile/faderp/ https://marketing.wharton.upenn.edu/profile/faderp/ http://brucehardie.com http://brucehardie.com

paulstovell 6y ago

Absolutely not an expert here, but I do obsess about our churn numbers. It seems this post is trying to make a model to predict churn or guess what future churn might be. But the main case for understanding churn to me is historical. Last month we had 10 customers paying us $10 each. Two of them cancelled. I have gross churn of $20. That's not a model, just a historical fact. What's "wrong" with that calculation that these models solve? I will definitely get different churn numbers if I look at groups of customers in cohorts based on when they first signed up, or enterprise-vs-SMB, or whatever. So I have reports to create all of those cohorts and show me the churn for each. I look at that data - with all the historical context - and make a decision about what to do next in the business. "Selling to SMB's was a good idea, but wow they churn a lot more. Let's focus our marketing on enterprises this month." To me the churn calculation at the top of the article is plenty useful. I'm not sure what value I'd get from these advanced models. If they just exist to predict future churn or LTV, that doesn't seem particularly useful.

paulstovell 6y ago
Also if the problem is "churn as a single number" is wrong because it misleads people (the example being churn is up because the company is growing) surely the answer is to show a cohort analysis or historical data - not a forward looking or "best fit" model?
onlyrealcuzzo 6y ago
Predicting LTV is quite useful for marketing and advertising.