5 ms·

Predicting Churn: When Do Veterans Quit?

67 points by ukdm 14y ago

praptak 14y ago
This is a bit scary. Imagine your employer using machine learning on your git commit patterns (or swearwords per comment) to check when you might quit.
- yummyfajitas 14y ago
  I can imagine it. Why is it scary? Would it also be scary if your manager observed your attitude/emotions through personal interaction and deduced you might want to quit?
  - dagw 14y ago
    I'd be all for a tool like that, assuming it actually worked and it was used in a productive way. Imagine if you're getting tired of writing RoR code all day and pondering quitting and all of a sudden your boss comes up to you and says "hey how would you like stop coding Rails for a while and join a team that's about to evaluate Clojure for an upcoming project". That's just the sort of thing that could make everybody happy.
    - xradionut 14y ago
      You're assuming that managers may actual care about the intellectual and professional needs of the staff instead of fulfilling their bosses goals. (Or maybe I'm just jaded and need to find some greener pastures... :) )
    - sillysaurus 14y ago
      Most bosses in the US would use these metrics to more quickly replace those who were about to quit.
      - Evbn 14y ago
        Only if the boss has replacements lined up, which is rare in professional jobs.
        GFischer 14y ago
        When I took a HR course (for MBAs), I learned that Human Resources is supposed to have a replacement roster (it's part of their job), I guess that's not common in practice, but they should.
    - GFischer 14y ago
      The problem is, at least in my experience, the ratio of people who would use it for good vs those who would misuse it would be very bad. Of course, it's also the employee's fault for getting into such an environment in the first place, but sometimes you just need the job. I use ManicTime Tracker, if my boss got hold of all the data contained in there, he would probably consider firing me - I average more than 2 solid hours of surfing the web on any given workday... wait, I have the exact data :) , that's the point! I was at my work PC for 1059 hours so far this year. Of those, I spent 325 hours on Firefox and 95 on Chrome, that averages about 13 hours a week of web surfing (probably 10 hours procrastinating or reading and 3 actually researching problems). I also spend a shocking 8 hours a week reading and replying to mail (to be fair, we don't have any bug tracking or project management, so mail becomes both), and 5 hours a week on SQL (I do a lot of querying and reporting). I'm less than 4 hours a week actually on a development environment, split between VB6, .NET and Forte4GL (our ugly legacy system). And I was hired to be a "systems analyst"... And that's only time spent at the PC, it doesn't count time wasted on meetings and stuff. This is a real-life example of why they say that on a large corporation, you only do actual work 1 or 2 hours a day (as opposed to a startup where you might do code or programming-related stuff 6 or 7 hours, I hope :) ). It's really depressing to put it in numbers. Fortunately, I'm going to quit next year and dedicate full-time to my startup (which is just getting started right now :) ).
      - JoeAltmaier 14y ago
        It's called Short Timer's Syndrome. When you know you're leaving, your dedication and work ethic plummet. You don't need a lot of statistics to notice it; you just have to be looking. Many managers are too busy looking up the management tree, sucking up to superiors, working their own career to actually manage their subordinates. If you're expecting to leave then you're in luck, they won't notice. But that's probably the reason you're leaving too!
        GFischer 14y ago
        I didn't know it had an official name :) . There are papers and everything (example: http://www.opus12.org/uploads/O12-SCI-V02-N01-P30.pdf http://www.opus12.org/uploads/O12-SCI-V02-N01-P30.pdf ) I guess surfing the web is a form of short-timing - I can't actually leave early or come in late, since I'm heavily penalized for those, like any self-respecting bureaucracy, my company equals time at the desk with productivity, and heavily penalizes lateness.
  - yarrel 14y ago
    A manager who has time to log all my actions throughout the day then study them to try and guess whether he's doing such a bad job that I'm thinking of quitting has probably answered the question before they even start.
  - praptak 14y ago
    For me the difference is between expected and unexpected. I expect bosses to observe my attitude/emotions through personal interaction and even review my commits by hand (it's even mandatory where I work.) I do not expect them to compute extensive statistics on what I do in the repository and draw deep conclusions from that, it's somehow too close to what I consider private. It's obviously subjective.
    - Evbn 14y ago
      Well, now it is expected. Welcome to the post-privacy era.
- benkant 14y ago
  Google does something like this, apparently: http://online.wsj.com/article/SB124269038041932531.html#mod=testMod http://online.wsj.com/article/SB124269038041932531.html#mod=...
- cadr 14y ago
  If your boss needs machine learning to see that you aren't happy, there are much bigger issues.
Morg 14y ago
This is indeed scary. Engineered addiction. Soon to be, automated management of customer addiction. A few more years and some people will be unpluggable - Imagine Diablo3:Inferno, except it raises the bar if it sees you're a good player, lowers it if you're getting a bit too hammered, increases the magic find rate when you look a bit distracted (mouse click and key hits density can give you that information, if you exclude chat commands) or keeps the good drops in stock to release them just before you stop playing, so you go on for another hour to check that stuff out. Sure thing is, the next generation will have even more addiction to fight.
westi 14y ago
This is really interesting. I can think of great possibilities for using these methods to analyse the behaviour of users in any number of online services to identify the ones that might need some form of out reach to help them out and keep them as users rather than loosing them due to the problems they are having.
patio11 14y ago
You can do a very, very similar thing for non-game SaaS services, too. (Hubspot calls theirs CHI -- Customer Happiness Index. It attempts to predict who is going to quit based on backtested statistical models. In practice, you can get almost sickeningly good results with heuristics that take ~3 lines to program. You can also then save those accounts by something as simple as "Have the sales team email them and talk" or, in my recent experience, "Send them an automated email reminding them that the software exists.")
- archon 14y ago
  As a user, those emails have sometimes had a negative effect for the company emailing me. Something they send me reminds me that the program exists, that I'm still paying a monthly fee for it, and that I really don't use it very much at all. But in all fairness, I doubt anything they could have said or offered would have changed my mind at that point. It was just a matter of when I cancelled, not if.
  - karterk 14y ago
    The reason for emailing you could also be to find out why you aren't using the app. Even if one ends up closing the account as a result, the information gained is valuable for improving the product.
  - scott_w 14y ago
    I think your last sentence is important - you were already gone, it was just a matter of time. If the numbers work such that losing 1% of customers means keeping 3% i.e. the email prompts them to leave/stay respectively, then it's worth sending. Getting rid of low-value users - in this case those not playing - can have long term benefits of focusing the business on getting more custom instead of getting comfortable on revenue that may disappear at any point.
  - rfergie 14y ago
    Stochastic Solutions have several papers about this problem: http://www.stochasticsolutions.com/papers.html http://www.stochasticsolutions.com/papers.html In a subscription business who can be saved, who will leave regardless and then the tricky group of people who will leave if they get a sales call?
aaronjg 14y ago
That's a really interesting approach. Basically mining a couple of key features from the data set, and plugging it into a supervised learning algorithm. The really cool thing is the simplicity of the features used, just daily activity and daily playtime. The write up of the methodology is very clear, but I'd love to see some more description of the results. Ninety-five percent accuracy is a pretty bold claim, and I'd love to see some ROC curves to back it up!
- mturmon 14y ago
  The high accuracy surprised me too. To compute their accuracy, their methodology seems to require determining whether someone is a veteran user, and having a clear quit time for them (otherwise the user can't be used for training or testing). Maybe after you make these determinations, the resulting population is easier to deal with.
wccrawford 14y ago
My previous employer knew I was going to quit long before I told them. But then, the fact that I was sending out resumes was probably a huge hint. I started sending them when I was a little disgruntled. Eventually, they heaped crap on me until I got serious about it and found a new job. Not once did they attempt to do anything to stop me, and asking them to pay me what I'm worth was met with, "Can you wait a year?" This, despite the fact that every review I ever got was great. Not just good, great. The only complaints I got was that I was too quick to answer questions literally. If someone asked if something was possible, I told them. They wanted me to read their minds and ask the question they should have asked, instead of the question they did. I eventually learned to do that, even, though. I have to admit it made things smoother. But jeez was it a pain.
zach 14y ago
The best response would be to offer a reason for people to come back after they leave for a week or so. The worst response would to offer a reason for people to leave for a week or so if they weren't going to already. If the company basically offers an incentive for not logging in, word will get around very fast. Then players have an excuse for taking a break, that they are gaming the system to earn an incentive. Penalizing players is ineffective too because you want to welcome back your wayward customers instead of starting to burn bridges ("I'm going to throw out your stuff! Okay, I'm putting it in the trash right now!") So unfortunately, I think all a company can do is have emails and community managers get in touch. Of course, that encourages players to stop playing for a week whenever they want to escalate a customer service issue. It's like you can't win. I discovered at Blizzard that guild features were enormously strategic. When players get into an active guild, especially with people they already know, it becomes hard to leave. The glue that keeps players put is social expectations.
ShabbyDoo 14y ago
The article cites customer service interactions as the obvious (and obviously effective) remediation step once a player is identified as likely to quit in a few weeks. I'd like to know if this is really true. What if the game's fundamental problem is that players become bored over time? What can a CSR do to fix this? Probably not much. The analysis shows that changes in gameplay frequency are far from a random walk and do indeed predict future declines, but it doesn't offer much in the way of clues for how to keep customers engaged. What if it could be shown that a breakdown of a player's in-game social structure (I'm not a gamer, so I'm being vague) also predicts churn? Perhaps helping the player to find some new relationships would be effective. Is game play repetitive? Maybe decreasing frequency of novel experiences predicts churn? This would be good feedback for game designers. Predicting the future isn't of much valuable if it isn't actionable.
bloaf 14y ago
I bet you could apply this method to other participation-based business. One that springs immediately to mind is gym membership.
pydave 14y ago
There was an article with a similar goal previously posted to Gamasutra from altdevblogaday, but it's got data and software for you to play along: http://www.altdevblogaday.com/2012/08/10/business-analytics-with-regression/ http://www.altdevblogaday.com/2012/08/10/business-analytics-...

67 points by ukdm 14y ago

This is a bit scary. Imagine your employer using machine learning on your git commit patterns (or swearwords per comment) to check when you might quit.

yummyfajitas 14y ago
I can imagine it. Why is it scary? Would it also be scary if your manager observed your attitude/emotions through personal interaction and deduced you might want to quit?
- dagw 14y ago
  I'd be all for a tool like that, assuming it actually worked and it was used in a productive way. Imagine if you're getting tired of writing RoR code all day and pondering quitting and all of a sudden your boss comes up to you and says "hey how would you like stop coding Rails for a while and join a team that's about to evaluate Clojure for an upcoming project". That's just the sort of thing that could make everybody happy.
  - xradionut 14y ago
    You're assuming that managers may actual care about the intellectual and professional needs of the staff instead of fulfilling their bosses goals. (Or maybe I'm just jaded and need to find some greener pastures... :) )
  - sillysaurus 14y ago
    Most bosses in the US would use these metrics to more quickly replace those who were about to quit.
    - Evbn 14y ago
      Only if the boss has replacements lined up, which is rare in professional jobs.
      - GFischer 14y ago
        When I took a HR course (for MBAs), I learned that Human Resources is supposed to have a replacement roster (it's part of their job), I guess that's not common in practice, but they should.
  - GFischer 14y ago
    The problem is, at least in my experience, the ratio of people who would use it for good vs those who would misuse it would be very bad. Of course, it's also the employee's fault for getting into such an environment in the first place, but sometimes you just need the job. I use ManicTime Tracker, if my boss got hold of all the data contained in there, he would probably consider firing me - I average more than 2 solid hours of surfing the web on any given workday... wait, I have the exact data :) , that's the point! I was at my work PC for 1059 hours so far this year. Of those, I spent 325 hours on Firefox and 95 on Chrome, that averages about 13 hours a week of web surfing (probably 10 hours procrastinating or reading and 3 actually researching problems). I also spend a shocking 8 hours a week reading and replying to mail (to be fair, we don't have any bug tracking or project management, so mail becomes both), and 5 hours a week on SQL (I do a lot of querying and reporting). I'm less than 4 hours a week actually on a development environment, split between VB6, .NET and Forte4GL (our ugly legacy system). And I was hired to be a "systems analyst"... And that's only time spent at the PC, it doesn't count time wasted on meetings and stuff. This is a real-life example of why they say that on a large corporation, you only do actual work 1 or 2 hours a day (as opposed to a startup where you might do code or programming-related stuff 6 or 7 hours, I hope :) ). It's really depressing to put it in numbers. Fortunately, I'm going to quit next year and dedicate full-time to my startup (which is just getting started right now :) ).
    - JoeAltmaier 14y ago
      It's called Short Timer's Syndrome. When you know you're leaving, your dedication and work ethic plummet. You don't need a lot of statistics to notice it; you just have to be looking. Many managers are too busy looking up the management tree, sucking up to superiors, working their own career to actually manage their subordinates. If you're expecting to leave then you're in luck, they won't notice. But that's probably the reason you're leaving too!
      - GFischer 14y ago
        I didn't know it had an official name :) . There are papers and everything (example: http://www.opus12.org/uploads/O12-SCI-V02-N01-P30.pdf http://www.opus12.org/uploads/O12-SCI-V02-N01-P30.pdf ) I guess surfing the web is a form of short-timing - I can't actually leave early or come in late, since I'm heavily penalized for those, like any self-respecting bureaucracy, my company equals time at the desk with productivity, and heavily penalizes lateness.
- yarrel 14y ago
  A manager who has time to log all my actions throughout the day then study them to try and guess whether he's doing such a bad job that I'm thinking of quitting has probably answered the question before they even start.
- praptak 14y ago
  For me the difference is between expected and unexpected. I expect bosses to observe my attitude/emotions through personal interaction and even review my commits by hand (it's even mandatory where I work.) I do not expect them to compute extensive statistics on what I do in the repository and draw deep conclusions from that, it's somehow too close to what I consider private. It's obviously subjective.
  - Evbn 14y ago
    Well, now it is expected. Welcome to the post-privacy era.
benkant 14y ago
Google does something like this, apparently: http://online.wsj.com/article/SB124269038041932531.html#mod=testMod http://online.wsj.com/article/SB124269038041932531.html#mod=...
cadr 14y ago
If your boss needs machine learning to see that you aren't happy, there are much bigger issues.

Morg 14y ago

This is indeed scary. Engineered addiction. Soon to be, automated management of customer addiction. A few more years and some people will be unpluggable - Imagine Diablo3:Inferno, except it raises the bar if it sees you're a good player, lowers it if you're getting a bit too hammered, increases the magic find rate when you look a bit distracted (mouse click and key hits density can give you that information, if you exclude chat commands) or keeps the good drops in stock to release them just before you stop playing, so you go on for another hour to check that stuff out. Sure thing is, the next generation will have even more addiction to fight.

westi 14y ago

This is really interesting. I can think of great possibilities for using these methods to analyse the behaviour of users in any number of online services to identify the ones that might need some form of out reach to help them out and keep them as users rather than loosing them due to the problems they are having.

patio11 14y ago

You can do a very, very similar thing for non-game SaaS services, too. (Hubspot calls theirs CHI -- Customer Happiness Index. It attempts to predict who is going to quit based on backtested statistical models. In practice, you can get almost sickeningly good results with heuristics that take ~3 lines to program. You can also then save those accounts by something as simple as "Have the sales team email them and talk" or, in my recent experience, "Send them an automated email reminding them that the software exists.")

archon 14y ago
As a user, those emails have sometimes had a negative effect for the company emailing me. Something they send me reminds me that the program exists, that I'm still paying a monthly fee for it, and that I really don't use it very much at all. But in all fairness, I doubt anything they could have said or offered would have changed my mind at that point. It was just a matter of when I cancelled, not if.
- karterk 14y ago
  The reason for emailing you could also be to find out why you aren't using the app. Even if one ends up closing the account as a result, the information gained is valuable for improving the product.
- scott_w 14y ago
  I think your last sentence is important - you were already gone, it was just a matter of time. If the numbers work such that losing 1% of customers means keeping 3% i.e. the email prompts them to leave/stay respectively, then it's worth sending. Getting rid of low-value users - in this case those not playing - can have long term benefits of focusing the business on getting more custom instead of getting comfortable on revenue that may disappear at any point.
- rfergie 14y ago
  Stochastic Solutions have several papers about this problem: http://www.stochasticsolutions.com/papers.html http://www.stochasticsolutions.com/papers.html In a subscription business who can be saved, who will leave regardless and then the tricky group of people who will leave if they get a sales call?

aaronjg 14y ago

That's a really interesting approach. Basically mining a couple of key features from the data set, and plugging it into a supervised learning algorithm. The really cool thing is the simplicity of the features used, just daily activity and daily playtime. The write up of the methodology is very clear, but I'd love to see some more description of the results. Ninety-five percent accuracy is a pretty bold claim, and I'd love to see some ROC curves to back it up!

mturmon 14y ago
The high accuracy surprised me too. To compute their accuracy, their methodology seems to require determining whether someone is a veteran user, and having a clear quit time for them (otherwise the user can't be used for training or testing). Maybe after you make these determinations, the resulting population is easier to deal with.

wccrawford 14y ago

My previous employer knew I was going to quit long before I told them. But then, the fact that I was sending out resumes was probably a huge hint. I started sending them when I was a little disgruntled. Eventually, they heaped crap on me until I got serious about it and found a new job. Not once did they attempt to do anything to stop me, and asking them to pay me what I'm worth was met with, "Can you wait a year?" This, despite the fact that every review I ever got was great. Not just good, great. The only complaints I got was that I was too quick to answer questions literally. If someone asked if something was possible, I told them. They wanted me to read their minds and ask the question they should have asked, instead of the question they did. I eventually learned to do that, even, though. I have to admit it made things smoother. But jeez was it a pain.

zach 14y ago

The best response would be to offer a reason for people to come back after they leave for a week or so. The worst response would to offer a reason for people to leave for a week or so if they weren't going to already. If the company basically offers an incentive for not logging in, word will get around very fast. Then players have an excuse for taking a break, that they are gaming the system to earn an incentive. Penalizing players is ineffective too because you want to welcome back your wayward customers instead of starting to burn bridges ("I'm going to throw out your stuff! Okay, I'm putting it in the trash right now!") So unfortunately, I think all a company can do is have emails and community managers get in touch. Of course, that encourages players to stop playing for a week whenever they want to escalate a customer service issue. It's like you can't win. I discovered at Blizzard that guild features were enormously strategic. When players get into an active guild, especially with people they already know, it becomes hard to leave. The glue that keeps players put is social expectations.

ShabbyDoo 14y ago

The article cites customer service interactions as the obvious (and obviously effective) remediation step once a player is identified as likely to quit in a few weeks. I'd like to know if this is really true. What if the game's fundamental problem is that players become bored over time? What can a CSR do to fix this? Probably not much. The analysis shows that changes in gameplay frequency are far from a random walk and do indeed predict future declines, but it doesn't offer much in the way of clues for how to keep customers engaged. What if it could be shown that a breakdown of a player's in-game social structure (I'm not a gamer, so I'm being vague) also predicts churn? Perhaps helping the player to find some new relationships would be effective. Is game play repetitive? Maybe decreasing frequency of novel experiences predicts churn? This would be good feedback for game designers. Predicting the future isn't of much valuable if it isn't actionable.

bloaf 14y ago

I bet you could apply this method to other participation-based business. One that springs immediately to mind is gym membership.

pydave 14y ago

There was an article with a similar goal previously posted to Gamasutra from altdevblogaday, but it's got data and software for you to play along: http://www.altdevblogaday.com/2012/08/10/business-analytics-with-regression/ http://www.altdevblogaday.com/2012/08/10/business-analytics-...