5 ms·

It honestly delights me to see someone correctly point out that churn is a classic example of survival analysis problem. It's shocking how many data scientist

It honestly delights me to see someone correctly point out that churn is a classic example of survival analysis problem. It's shocking how many data scientist I've known how no idea how to correctly model churn, typically trying to build some predictive model, which is just a more sophisticated extension of first formula in the article. It's insane how many business problems are really survival analysis problems where you have some hazard function and censored data. In my experience having data scientists that can correctly identify this is very rare (and a great sign of how broken that entire field is right now).

qsort 6y ago
Not going to argue with you that the field isn't broken -- it very clearly is -- but the problem the article explains is a rather simple one. Your setting could be non-contractual. You might not be able to have reliable data to frame the problem as survival analysis with censored data. You might have different groups of customers with different behaviors. Assuming a constant failure rate is the same as assuming the underlying process is Poisson, which is not an entirely stupid working hypothesis in many cases. The moral of the story is "by the love of god look at your data before making dumb models".
- sebastianmarkow 6y ago
  Different population behaviours could be modeled via multi-level models.
  - qsort 6y ago
    They can, but you start running into problems with the mathematical formulation of your model. For example, Weibull's CF doesn't have a closed form, which means it's hard to find convolutions even with itself, so that finding e.g. the sum of Weibull distributed r.v. is hard, even if they are iid. Again, modeling done right is super hard.
sebastianmarkow 6y ago
This is the comment I was hoping to find here. I can only recommend reading McElreath’s Statistical Rethinking Book/Course in that context. https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/
epistasis 6y ago
I think about this everytime Backblaze publishes their new stats. It's a great resource, but some standard survival analysis would go a looooong way, and would let them incorporate usage stats amount other things. I even emailed once to offer to help, but they were (very understandably) not interested.
dcl 6y ago
> It's shocking how many data scientist I've known how no idea how to correctly model churn, This is because most 'data scientists' have never actually studied much statistics. And for the rest, it's easy to forgot all the cool tools we learnt about at university when everyone in the corporate world just says 'give me the average revenue per customer', 'give me the average tenure' etc, etc.
- equality_1138 6y ago
  what have data scientists generally studied if not for statistics?
  - dcl 6y ago
    In my experience, 'data scientist' has become a very loose term, especially in the not terribly sophisticated corporate world. Typically, 'data science' isn't even very scientific, but a useful set of heuristics to get good model outcomes on data. Rarely is actual 'statistics' being done to get scientifically rigours results. People employed as 'data scientists' often come from comp sci/physics/eng/econ etc. They might be great at fitting models that also have great out of sample prediction performance, but they may have never heard of maximum likelihood, or know when to use a Wilcoxon rank sum test.
    - mcrad 6y ago
      Not very sophisticated corporate world? I'd argue it's very sophisticated to be loose with the meaning of scientist. We have a global debate happening about pseudo science due to some widespread acceptance to call anyone with a degree and some quantitative info a scientist.
  - TrackerFF 6y ago
    A lot, and I mean a lot of Data Scientists are just what we called Business Analysts, Analysts, etc. There's no cohesive group of Data Scientists, and they come from all walks of life. Economics, Business Administration, Life Sciences, Computer Science, Engineering, etc. There absolutely are professional Data Scientists out there, that haven't touched more than Stats and Probability 101. (But FWIW, a lot of DS - even though they may lack the formal education, pick up these things underway)
esyir 6y ago
Oof, this is me right now. What makes it worse is that I've also done survival analysis, but never quite made the jump that churn is just a survival problem, though it's exceedingly obvious once you see it.
stevofolife 6y ago
Agree. Interestingly, are there any other examples where a problem can be solved with Y instead of X?
- disgruntledphd2 6y ago
  Life time value is a time series problem mostly brutalised into a regression format.
equality_1138 6y ago
broken what field? for how long?
disgruntledphd2 6y ago
Yeah, agreed. Spark even has functions for survival analysis (Kaplan-Meier I think), which means that one can actually run this kind of analysis in a consumer-tech space. I remember spending weeks trying to get the right sampling/compute so that I could actually use survival analysis, so I would argue that it's not that people weren't aware of this, rather that (some) didn't have tools to allow them to use the methods effectively.

qsort 6y ago

Not going to argue with you that the field isn't broken -- it very clearly is -- but the problem the article explains is a rather simple one. Your setting could be non-contractual. You might not be able to have reliable data to frame the problem as survival analysis with censored data. You might have different groups of customers with different behaviors. Assuming a constant failure rate is the same as assuming the underlying process is Poisson, which is not an entirely stupid working hypothesis in many cases. The moral of the story is "by the love of god look at your data before making dumb models".

sebastianmarkow 6y ago
Different population behaviours could be modeled via multi-level models.
- qsort 6y ago
  They can, but you start running into problems with the mathematical formulation of your model. For example, Weibull's CF doesn't have a closed form, which means it's hard to find convolutions even with itself, so that finding e.g. the sum of Weibull distributed r.v. is hard, even if they are iid. Again, modeling done right is super hard.

sebastianmarkow 6y ago

This is the comment I was hoping to find here. I can only recommend reading McElreath’s Statistical Rethinking Book/Course in that context. https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/

epistasis 6y ago

I think about this everytime Backblaze publishes their new stats. It's a great resource, but some standard survival analysis would go a looooong way, and would let them incorporate usage stats amount other things. I even emailed once to offer to help, but they were (very understandably) not interested.

dcl 6y ago

> It's shocking how many data scientist I've known how no idea how to correctly model churn, This is because most 'data scientists' have never actually studied much statistics. And for the rest, it's easy to forgot all the cool tools we learnt about at university when everyone in the corporate world just says 'give me the average revenue per customer', 'give me the average tenure' etc, etc.

equality_1138 6y ago
what have data scientists generally studied if not for statistics?
- dcl 6y ago
  In my experience, 'data scientist' has become a very loose term, especially in the not terribly sophisticated corporate world. Typically, 'data science' isn't even very scientific, but a useful set of heuristics to get good model outcomes on data. Rarely is actual 'statistics' being done to get scientifically rigours results. People employed as 'data scientists' often come from comp sci/physics/eng/econ etc. They might be great at fitting models that also have great out of sample prediction performance, but they may have never heard of maximum likelihood, or know when to use a Wilcoxon rank sum test.
  - mcrad 6y ago
    Not very sophisticated corporate world? I'd argue it's very sophisticated to be loose with the meaning of scientist. We have a global debate happening about pseudo science due to some widespread acceptance to call anyone with a degree and some quantitative info a scientist.
- TrackerFF 6y ago
  A lot, and I mean a lot of Data Scientists are just what we called Business Analysts, Analysts, etc. There's no cohesive group of Data Scientists, and they come from all walks of life. Economics, Business Administration, Life Sciences, Computer Science, Engineering, etc. There absolutely are professional Data Scientists out there, that haven't touched more than Stats and Probability 101. (But FWIW, a lot of DS - even though they may lack the formal education, pick up these things underway)

esyir 6y ago

Oof, this is me right now. What makes it worse is that I've also done survival analysis, but never quite made the jump that churn is just a survival problem, though it's exceedingly obvious once you see it.

stevofolife 6y ago

Agree. Interestingly, are there any other examples where a problem can be solved with Y instead of X?

disgruntledphd2 6y ago
Life time value is a time series problem mostly brutalised into a regression format.

equality_1138 6y ago

broken what field? for how long?

disgruntledphd2 6y ago

Yeah, agreed. Spark even has functions for survival analysis (Kaplan-Meier I think), which means that one can actually run this kind of analysis in a consumer-tech space. I remember spending weeks trying to get the right sampling/compute so that I could actually use survival analysis, so I would argue that it's not that people weren't aware of this, rather that (some) didn't have tools to allow them to use the methods effectively.