9 ms·

NPS doesn’t say anything particularly useful

109 points by boraginaceae 5y ago

yuliyp 5y ago
The point of NPS vs free cash flow or operational metrics is that it is in theory a leading indicator instead of a trailing indicator. It's nice (but a bit too late) to know that your business is screwed because your customers left, it's another to know that your business is about to be screwed because your customers are on the verge of leaving if an opportunity appears. The criticisms of NPS pointed out (that it's a measure of a high-variance metric) are fair, but the conclusion is not.
- keiran_cull 5y ago
  What exactly would you suggest it's a leading indicator of? Churn? Ultimately if it matters, it should have to be an input to the cash flow time series -- mediated by churn -- no? (Article means to frame free cash flow as a series rather than a point-in-time value). If you're interested in forecasting, as a company, you should know the things that mean your customer is not doing well within the specific context of your specific business. E.g. in software, you should have a customer health score that's built up from product data. Or you could ask simple questions that are easily interpreted, e.g. asking "how would you rate X's value for money," or "how satisfied are you with X," or even "have you recommended X in the last Y months." These things have a cleaner relationship to future business metrics and a tidier interpretation.
  - andrewingram 5y ago
    My own understanding is that it’s supposed to be a prediction input organic growth and nothing more. All it’s really telling you is whether a customer is likely to provide you with another customer, which is why only really high values answers matter, because “hell yeahs” are all that really matter. I’m skeptical as to its usefulness even for this, but most companies I’ve worked for use it as a general KPI which is even more aggravating.
    - marcosdumay 5y ago
      If you go into the trouble of collecting it, it's reasonable to use as a KPI, as it carries useful information. I imagine you are annoyed that they optimize for it. And yes, it's not reasonable to optimize for it. Why do people optimize every KPI?
      - Talanes 5y ago
        >Why do people optimize every KPI? Because if it's not worth optimizing around, how key can it be?
        marcosdumay 5y ago
        You can do a lot of things with a number that are not optimizing. For example, you can satisfy it, you can alert on behavior, or you can use it as a control (but ok, maybe this one makes a two number KPI instead of two KPIs).
  - lmeyerov 5y ago
    Hopefully, it should be a leading indicator for negative churn -- per-account and new account growth / virality -- and for more complicated products, a way to slice that across different personas and features. ('bakers at high-end restaurants have high NPS for the new precision thermometer tracker while those at regular shops ignore them and have low NPS.') Esp in early days of B2B products, it's hard to get that because you don't have the volumes and velocity of b2c nor a good way to detect and attribute viral activation. If you are in a startup and not riding the channel of some megacorp, even more so. Alternatives like signups or other activation checkpoints, or say qualitative interviews, are also interesting, but even more spotty. (Ex: startups raising based on GitHub stars.) We don't do NPS as we have our plate full with known funnel holes through less annoying data collection methods, but as soon as we are happy with the baseline funnel, that's the simple next step.
  - yuliyp 5y ago
    I'd expect it to be a leading indicator of user growth / churn, yes.
- hakfoo 5y ago
  One of the most common contexts for NPS scoring is giving them to direct retail customers as a phone/email followup survey. This is the most dangerous context for it, because a customer who isn't familiar with the system, is sort of primed to have the wrong expectations. They're thinking in terms of "how satisfied was I", which is a different scale entirely with "would you recommend the product". But the NPS question, even when explicitly explained, reads close enough that they'll answer it with the "how satisfied am I" answer, NPS might have worked in a context of a professionally proctored focus group where everyone understood the question and discussed it, but I'm worried you're losing a lot of information when you turn it to one digit on a keypad.
mdorazio 5y ago
I didn't see discussed here the main drivers you might want NPS vs. outcome- (and company) specific metrics: 1) it's harder to game and/or bullshit. This is a massive issue with surveying customers and then using their responses to feed back into internal performance goals & rewards. 2) because it has grown to be such a common customer experience metric, it's possible to benchmark against others, which isn't really possible with question types that are very specific to your business I'll agree that NPS is overrated, mostly because the Net Promoter organization and survey companies have spent large amounts of time and money pushing it as The One True Metric for decades, but it still has its merits and its place.
- riversflow 5y ago
  > it's harder to game and/or bullshit. . . it's possible to benchmark against others, I don't have access to A/B testing of a large business to understand how things affect responses. But I certainly intuit that there is a significance in when/how you ask someone for feedback. e.g. If you ask your question directly after the checkout screen you'll get a significantly different response than if you survey people randomly who visit your website, or email customers a set period after making a purchase/ upon transaction completion. I've seen all these, which brings me to the second part, how do you benchmark against a competitors data, and why would you trust a competitors data?
  - mdorazio 5y ago
    Because if you're a big company you don't run your own surveys - you pay one of the big CX companies like Maritz, Medallia, InMoment, etc. to run them for you or for smaller feedback not requiring a full management system you hire one of the 100+ research firms to do it for you. Because these firms run surveys for tons of clients, they generally have standard practices for how & when feedback is collected and are willing to share industry-standard benchmarks for things like NPS. I worked with two of the names above on a 2-year stint with a major auto manufacturer that sent over a million surveys a year and NPS comparisons were one of many tools used to get meaning out of responses.
madeofpalk 5y ago
My "problem" with NPS is when essential services, which I have next to no choice over using, ask me if i would recommend them to others. No, IRS, I would never recommend your service to friends or family. That's not really indicative of anything.
- yjftsjthsd-h 5y ago
  Ah, like the good old > How likely are you to recommend Windows 10 to a friend or colleague? Please explain why you gave this score. > I need you to understand that people don't have conversations where they randomly recommend operating systems to one another
  - chrisseaton 5y ago
    But they do - people ask each other if it's worth updating macOS for example - I've literally been asked if I'd recommend updating many times - I've asked people if they'd recommend a Linux distribution - etc.
  - muzani 5y ago
    I think people recommend operating systems more than any other software. Windows vs Mac vs whatever Linux. Android vs iOS. My mother lived in the era when Bill Gates was the villain and refused to update to Win 10 because she expected it to have microtransactions or something.
- xboxnolifes 5y ago
  No PatientCo, I did not enjoy paying my medical bills through your system. I have no intention of recommending your bill payment system to my family. Why is this even asked?
  - Spooky23 5y ago
    I switched doctors when their billing became impossible to deal with. Assuming they gave a shit, that survey response would have been useful to the practice.
actually_a_dog 5y ago
I suspect the concept of NPS is, if not the source of, a major driver of the whole "anything less than 10/10, 5 stars, etc. is basically a fail" meme. There seem to be signifiant toxic effects from this on customer-facing employees, in my experience.
- xraystyle 5y ago
  This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful. One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is essentially giving a rating of zero. If I have what I would consider to be an average interaction with a business, e.g. I just buy something and leave, no need for support, no problems to deal with, etc., that seems average to me. Based on a non-NPS understanding of a 0-10 scale, I'd say that's what, a 7? But the business now considers this as a failure on the part of the person that helped me at that store. This is why sales people and phone reps are constantly now asking you to give them a 10/10 rating if you receive a survey, because even if they literally just took your money and handed you change, their jobs depend on you acting as if you had a heart attack and they saved your life by performing CPR or something. It's honestly a terrible system that produces no meaningful feedback for the company and causes employees to do whatever they can to game the numbers. All you're measuring with NPS is how good your employees are at juking the stats, nothing more.
  - sixdimensional 5y ago
    This, to me, is a general problem with a purely quantified, metrics driven approach to management, not just limited to NPS. I'm not demonizing these approaches, just saying that you have to balance humanity with data. I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking. Qualitative measures are important, as is maintaining as much humanity as possible, to have a balanced and healthy culture. Being solely metrics/data driven can lead to cold, heartless, damaging culture (might be efficient or make profit, but very dehumanizing).
password4321 5y ago
NPS = Net Promoter Score > measures customer experience and predicts business growth
hn_throwaway_99 5y ago
I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is down, let's hire more people to focus on customer 'happiness'", it's instead that NPS can be like a canary in a coal mine - an unexpected drop in NPS is something to investigate and get more information on what went wrong, usually from more detailed/free-form questions in the NPS questionnaire. As a software engineer I love reading our NPS reports, because they always have great tidbits of info that are often difficult to deduce just by looking at standard analytics.
- ekianjo 5y ago
  > primarily that there was a very strong correlation between the calculated NPS score and business performance. There is avsolutely no science linking performance and NPS. Just pseudo science.
- Lazare 5y ago
  > they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. There is no underlying science behind NPS. It was an arbitrary measure with no research or model behind it. Also, it doesn't work very well, in large because because there is no strong correlation between NPS and anything else, because - as this article makes clear - NPS is an extremely noisy number. Look at the graphs - you can have enormous variations in NPS, with zero change in underlying customer sentiment. Obviously that can't correlate with anything useful. I think what you're doing is conflating the idea of "asking how likely people are to recommend your product" with NPS. The former is a decent idea, and it does have some (limited) science behind it! There really was a study done that really did find that question was an excellent predictor of many key results. (...and some others that found it was not, but hey, it's something.) So yes, asking people the question and reading the results is probably a good idea. ...but that's not what NPS is. > an unexpected drop in NPS is something to investigate It really, really isn't. And I would challenge you to point to any evidence to the contrary.
foo4u 5y ago
What’s more interesting to me than the NPS scores are the free form comment section below the 0-10 rating. Never mind that anything below X is considered neutral or a detractor. You asked a user / customer/ etc to rate you on a scale of 1-10 and then tell you why. Want to know what you’re truly screwing up on? Take the feedback on your 1-6 scores seriously and you can find the low hanging fruit to take a product from mediocre to a great user experience. That’s the true NPS value. It’s all about how you handle the feedback.
Gelob 5y ago
I hate NPS. You know what makes it shit? It requires your customers to fill it out. So then your bugging them to complete it
- kqr 5y ago
  How would you (reliably) measure user satisfaction without asking users?
- muzani 5y ago
  I love giving feedback though. If I can't give it direct, it's going through Google Maps or something.
rkk3 5y ago
> (The most irritating — and by far the most common — reason companies seem to measure NPS is that it’s standardized, but more importantly that everyone else does it. Which is fine for one-time-use benchmarking, but not a good basis for internal KPIs.) Except the alternative isn't to stop tracking NPS the alternative is to use unstandardized customer survey metric's, this is worse because you have no true industry comparison & can be gamed more easily since they define the formula!
- itsdrewmiller 5y ago
  https://www.qualtrics.com/experience-management/customer/what-is-csat/ https://www.qualtrics.com/experience-management/customer/wha... is another standardized alternative, that I suspect would suffer less from the problems in the article.
  - rkk3 5y ago
    CSAT is not standardized and there are many ways to calculate it, one of which you have linked. I've seen firsthand many of the largest brands calculate it differently.
    - itsdrewmiller 5y ago
      Yeah, I agree it doesn't have the universal agreement on implementation that NPS does. Here's an alternative that I believe does and is referenced in some of the NPS skeptical research: https://en.wikipedia.org/wiki/American_Customer_Satisfaction_Index https://en.wikipedia.org/wiki/American_Customer_Satisfaction...
      - rkk3 5y ago
        That "measures the satisfaction of consumers across the U.S. economy", very different from Brand's Customer Experience Surveys. Having worked as a software vendor in the CX field, I saw a lot of different implementations of the "same" survey metrics. NPS was only standardized one (I think because its from b school in the 80's).
austenallred 5y ago
Yes, NPS is an imperfect measure, as all are. What I’ve never seen is a great argument for what one should use instead.
- Dave3of5 5y ago
  Company I was working with done an internal study on this I believe CSAT or a simple average were better if your goal is to improve revenue.
nhumrich 5y ago
The real problem with NPS is when a certain score is a goal. After all goodharts law states that is the case with any good measure.
- ketralnis 5y ago
  Yep. I had an advertising-funded app that asked me to fill out a survey about why I rated it poorly in a modal I can’t skip. So I always gave it a good score, until I stopped using it because it interrupted me so often to ask. They lost me and they’ll never know why
Lazare 5y ago
You know what would be vastly better than NPS? Ask people every so often how they're feeling about the product. Let them pick a thumbs up or a thumbs down. If they pick thumbs down, say "we're sorry you hear that. Please let us know what's bothering you", and display a little text box. Carefully read every comment you receive. Asking unhappy users for feedback is a great idea. The 0 to 10 scale and - especially - the bizarre calculation NPS does with it - is statistical malpractice that just isn't mathematically capable of doing what NPS supports claim.
- kqr 5y ago
  > Asking unhappy users for feedback is a great idea. The 0 to 10 scale and - especially - the bizarre calculation NPS does with it - is statistical malpractice I disagree. On the contrary, it is good survey practise to ask yes/no questions with a scale, to get information on intensity of opinion. (Often this is a scale 1--7 ranging from "strongly disagree" through "neutral" to "strongly agree", but there are other alternatives of which NPS is one.) Then you might think only the bottom half of the range would formally be considered "detractors", and that would (as far as the evidence goes) be accurate. However, we're not looking for how many people are detractors today -- we want to know how many people are at risk of becoming detractors in the near future. That's what makes NPS a somewhat leading metric. And that's also why you see the sleight of hand that counts answers up to 6 as detractors. I don't know what "mathematically capable" means in your context, nor what "NPS supports claim" in your experience, but the number absolutely makes sense.
- paulmooreparks 5y ago
  Be careful how you do this. When an app interrupts my normal workflow to ask me to rate the app or ask me how I'm enjoying the app, the answer is always negative. I'm using the app to get work done, not satisfy some VP's KPIs.
joe2784 5y ago
It's not fair to compare how "important" different metrics are to the whole business. A business is composed of and built on top of many sectors. That's why we have Finance team, Sales team, Marketing team, Product and Development team..., and not just one "we're a company" team! Free cash flow is important for business operations. Churn is important to predict sales. NPS, though not directly affecting the "money", might be a good indicator for the marketing team. There are a lot more important marketing metrics than NPS, but for a big corporation's branding team, especially in consumer goods industry, it might be useful to look at the "trend" of their NPS over a long period of time. NPS has noise just because of its nature of being collected with surveys. Like any questionnaires or surveys, if you design it well, you can reduce the noise.
throwawaysea 5y ago
This post is correct about two things - first, that NPS does not give you direction on how to act. Second, that statistical realities make sampling approaches like NPS very noisy. I don’t think either discounts the concept of NPS itself. “Bad statistical work” is going to be bad regardless of what metric we’re talking about. And a metrics being too abstract to action doesn’t make it useless - at least not any more so than something like free cash flow. Yes, connecting the dots to some outcome is difficult. But it’s a pulse on the business, and if done right, that has some inherent value.
Silasdev 5y ago
Why do everyone seem to think that management does not have actual good intentions with measuring NPS as a KPI? The companies I've worked with have all had genuinely good intentions making the best possible experience for the user, because that is what ultimately wins in the end. If someone scores 1-6, you ask them to provide additional feedback, to learn from it - problem solved and everyone has all the info they need to go about their work. There seems to be this idea that KPIs are evil and interpreted in vacuum. It's rarely like that.
- Ozzie_osman 5y ago
  > There seems to be this idea that KPIs are evil and interpreted in vacuum. Not just KPIs, but the Hackernews crowd seems to apply the same to management and corporations as a whole.
- dade_ 5y ago
  20% of companies are knocking it out of the park, and 50% are nowhere near a leaderboard. I expect such folks are correct in their observation, but work for the 50%. Perhaps you work for one that is getting a promoter score. My advice to friends looking for a job, ask them what their NPS is. The leaders in NPS are leaders in employee engagement. If they don’t know, or it’s bad and your role isn’t to make it better, look elsewhere.
kwhitefoot 5y ago
NPS = Net Promoter Score See https://en.wikipedia.org/wiki/Net_promoter_score https://en.wikipedia.org/wiki/Net_promoter_score
deleted 5y ago
[deleted]
meelford 5y ago
Background: I have designed and implemented NPS feedback systems in several companies. I noticed that people are concerned about this when it was not implemented correctly, i.e. as a complimentary question in the long questionnaire or as a follow-up question at the end of the journey. The NPS collection mechanic has many advantages: 1. The mental entry point for the user is very low - just 1 question. 2. It is very useful to track user feedback over their lifetime (ask every 6 months) and actively resolve issues if NPS shows it. 3. Another great thing is not a numerical feedback, but a written answer that gives a lot of really useful ideas for improvement. 4. And finally, you don't need any specific context to show the question, so it can be embedded in almost any stage of the user experience.
- timkam 5y ago
  Agree, the NPS aggregation is mostly BS, but getting a signal from median/mean, e.g. by comparing trends across some dimension, and reading complimentary feedback is quite useful.
thathndude 5y ago
I was unfamiliar with NPS until about 12 hours ago (but not because of this post). NPS has had a strong prediction of stock price (I learned this morning). https://www.marketwatch.com/story/this-surprising-investing-strategy-crushes-the-stock-market-without-examining-a-single-financial-metric-11638985326 https://www.marketwatch.com/story/this-surprising-investing-...? Of course, it’s just opinion. But maybe there’s some use to NPS after all.
dade_ 5y ago
Companies with a promoter level score during the pandemic grew their business. There is a direct correlation between a promoter NPS and revenue growth. The percentage of companies that now provide a promoter level service is now over 20%, up from 13% before the pandemic. The percentage that are rated a detractor level score also grew to nearly 50%. These org’s revenues either shrank or stagnated. This is an existential issue for organizations, not a nuisance KPI. Companies that are leaders in NPS are leaders in employee engagement. This post sounded pretty frustrated with an employer/contract… NPS is a measure of past experiences, it isn’t a GPS navigation system to improvement. It is also 1 question, so it is the most simple to collect. However even this gets screwed up all the time. Interfaces such as a telephone where 10 can be measured as the first digit 1, or PIN pads where they are a nuisance to a customer when in a hurry to pay and missed when paying with NFC. Understanding negative NPS is like understanding a traffic ticket with no other data. Why did I get a traffic ticket? Well, what did or didn’t you do and how fast were you doing it? Were you a jerk to the cop? Was the cop stressed by something else that day? To be a leader in NPS is an organizational effort, not the result of divining an insight from a few survey answers. Ideally, customer journeys maps have been developed and problem areas are self evident before an NPS score arrives.
samsolomon 5y ago
This post misses that the primary benefit of an NPS program is not the metric—that's a nice byproduct—it's the dialogue with customers. As a single metric Net Promoter Score is OK. It's extremely easy for customers to answer. So there's volume in the number of answers. You can learn more by segmenting different ways—Users vs Admins, Enterprise vs SMB, Verticals, etc.. The company I currently work for puts a lot into NPS, not necessarily the metric, but the process. Everyone that leaves feedback gets a follow-up email and interview from someone in Product or User Research. That feedback is then organized in Productboard where it is grouped with similar requests/issues. Overall those interviews heavily weigh what features we refine and build. For some background, I previously designed an NPS tool and have written about NPS pretty extensively. https://solomon.io/understanding-net-promoter-score/ https://solomon.io/understanding-net-promoter-score/

109 points by boraginaceae 5y ago

The point of NPS vs free cash flow or operational metrics is that it is in theory a leading indicator instead of a trailing indicator. It's nice (but a bit too late) to know that your business is screwed because your customers left, it's another to know that your business is about to be screwed because your customers are on the verge of leaving if an opportunity appears. The criticisms of NPS pointed out (that it's a measure of a high-variance metric) are fair, but the conclusion is not.

keiran_cull 5y ago
What exactly would you suggest it's a leading indicator of? Churn? Ultimately if it matters, it should have to be an input to the cash flow time series -- mediated by churn -- no? (Article means to frame free cash flow as a series rather than a point-in-time value). If you're interested in forecasting, as a company, you should know the things that mean your customer is not doing well within the specific context of your specific business. E.g. in software, you should have a customer health score that's built up from product data. Or you could ask simple questions that are easily interpreted, e.g. asking "how would you rate X's value for money," or "how satisfied are you with X," or even "have you recommended X in the last Y months." These things have a cleaner relationship to future business metrics and a tidier interpretation.
- andrewingram 5y ago
  My own understanding is that it’s supposed to be a prediction input organic growth and nothing more. All it’s really telling you is whether a customer is likely to provide you with another customer, which is why only really high values answers matter, because “hell yeahs” are all that really matter. I’m skeptical as to its usefulness even for this, but most companies I’ve worked for use it as a general KPI which is even more aggravating.
  - marcosdumay 5y ago
    If you go into the trouble of collecting it, it's reasonable to use as a KPI, as it carries useful information. I imagine you are annoyed that they optimize for it. And yes, it's not reasonable to optimize for it. Why do people optimize every KPI?
    - Talanes 5y ago
      >Why do people optimize every KPI? Because if it's not worth optimizing around, how key can it be?
      - marcosdumay 5y ago
        You can do a lot of things with a number that are not optimizing. For example, you can satisfy it, you can alert on behavior, or you can use it as a control (but ok, maybe this one makes a two number KPI instead of two KPIs).
- lmeyerov 5y ago
  Hopefully, it should be a leading indicator for negative churn -- per-account and new account growth / virality -- and for more complicated products, a way to slice that across different personas and features. ('bakers at high-end restaurants have high NPS for the new precision thermometer tracker while those at regular shops ignore them and have low NPS.') Esp in early days of B2B products, it's hard to get that because you don't have the volumes and velocity of b2c nor a good way to detect and attribute viral activation. If you are in a startup and not riding the channel of some megacorp, even more so. Alternatives like signups or other activation checkpoints, or say qualitative interviews, are also interesting, but even more spotty. (Ex: startups raising based on GitHub stars.) We don't do NPS as we have our plate full with known funnel holes through less annoying data collection methods, but as soon as we are happy with the baseline funnel, that's the simple next step.
- yuliyp 5y ago
  I'd expect it to be a leading indicator of user growth / churn, yes.
hakfoo 5y ago
One of the most common contexts for NPS scoring is giving them to direct retail customers as a phone/email followup survey. This is the most dangerous context for it, because a customer who isn't familiar with the system, is sort of primed to have the wrong expectations. They're thinking in terms of "how satisfied was I", which is a different scale entirely with "would you recommend the product". But the NPS question, even when explicitly explained, reads close enough that they'll answer it with the "how satisfied am I" answer, NPS might have worked in a context of a professionally proctored focus group where everyone understood the question and discussed it, but I'm worried you're losing a lot of information when you turn it to one digit on a keypad.

mdorazio 5y ago

I didn't see discussed here the main drivers you might want NPS vs. outcome- (and company) specific metrics: 1) it's harder to game and/or bullshit. This is a massive issue with surveying customers and then using their responses to feed back into internal performance goals & rewards. 2) because it has grown to be such a common customer experience metric, it's possible to benchmark against others, which isn't really possible with question types that are very specific to your business I'll agree that NPS is overrated, mostly because the Net Promoter organization and survey companies have spent large amounts of time and money pushing it as The One True Metric for decades, but it still has its merits and its place.

riversflow 5y ago
> it's harder to game and/or bullshit. . . it's possible to benchmark against others, I don't have access to A/B testing of a large business to understand how things affect responses. But I certainly intuit that there is a significance in when/how you ask someone for feedback. e.g. If you ask your question directly after the checkout screen you'll get a significantly different response than if you survey people randomly who visit your website, or email customers a set period after making a purchase/ upon transaction completion. I've seen all these, which brings me to the second part, how do you benchmark against a competitors data, and why would you trust a competitors data?
- mdorazio 5y ago
  Because if you're a big company you don't run your own surveys - you pay one of the big CX companies like Maritz, Medallia, InMoment, etc. to run them for you or for smaller feedback not requiring a full management system you hire one of the 100+ research firms to do it for you. Because these firms run surveys for tons of clients, they generally have standard practices for how & when feedback is collected and are willing to share industry-standard benchmarks for things like NPS. I worked with two of the names above on a 2-year stint with a major auto manufacturer that sent over a million surveys a year and NPS comparisons were one of many tools used to get meaning out of responses.

madeofpalk 5y ago

My "problem" with NPS is when essential services, which I have next to no choice over using, ask me if i would recommend them to others. No, IRS, I would never recommend your service to friends or family. That's not really indicative of anything.

yjftsjthsd-h 5y ago
Ah, like the good old > How likely are you to recommend Windows 10 to a friend or colleague? Please explain why you gave this score. > I need you to understand that people don't have conversations where they randomly recommend operating systems to one another
- chrisseaton 5y ago
  But they do - people ask each other if it's worth updating macOS for example - I've literally been asked if I'd recommend updating many times - I've asked people if they'd recommend a Linux distribution - etc.
- muzani 5y ago
  I think people recommend operating systems more than any other software. Windows vs Mac vs whatever Linux. Android vs iOS. My mother lived in the era when Bill Gates was the villain and refused to update to Win 10 because she expected it to have microtransactions or something.
xboxnolifes 5y ago
No PatientCo, I did not enjoy paying my medical bills through your system. I have no intention of recommending your bill payment system to my family. Why is this even asked?
- Spooky23 5y ago
  I switched doctors when their billing became impossible to deal with. Assuming they gave a shit, that survey response would have been useful to the practice.

actually_a_dog 5y ago

I suspect the concept of NPS is, if not the source of, a major driver of the whole "anything less than 10/10, 5 stars, etc. is basically a fail" meme. There seem to be signifiant toxic effects from this on customer-facing employees, in my experience.

xraystyle 5y ago
This is absolutely a massive issue with it. Apple did this with retail employees back in the day and it was terrible. Employee reviews were based on it, managers had to call customers and apologize if the customer didn't leave a 9 or 10 rating, it was absolutely awful. One of the biggest problems with it is that an average person, unaware of what NPS is, doesn't understand that giving any rating less than a 9 is essentially giving a rating of zero. If I have what I would consider to be an average interaction with a business, e.g. I just buy something and leave, no need for support, no problems to deal with, etc., that seems average to me. Based on a non-NPS understanding of a 0-10 scale, I'd say that's what, a 7? But the business now considers this as a failure on the part of the person that helped me at that store. This is why sales people and phone reps are constantly now asking you to give them a 10/10 rating if you receive a survey, because even if they literally just took your money and handed you change, their jobs depend on you acting as if you had a heart attack and they saved your life by performing CPR or something. It's honestly a terrible system that produces no meaningful feedback for the company and causes employees to do whatever they can to game the numbers. All you're measuring with NPS is how good your employees are at juking the stats, nothing more.
- sixdimensional 5y ago
  This, to me, is a general problem with a purely quantified, metrics driven approach to management, not just limited to NPS. I'm not demonizing these approaches, just saying that you have to balance humanity with data. I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking. Qualitative measures are important, as is maintaining as much humanity as possible, to have a balanced and healthy culture. Being solely metrics/data driven can lead to cold, heartless, damaging culture (might be efficient or make profit, but very dehumanizing).

password4321 5y ago

NPS = Net Promoter Score > measures customer experience and predicts business growth

hn_throwaway_99 5y ago

I see proposals against NPS like this from time to time, but they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. Also, I find that many of these articles have somewhat of a caricaturish view of how NPS is used in the real world. "It's not 'oh no, our NPS score is down, let's hire more people to focus on customer 'happiness'", it's instead that NPS can be like a canary in a coal mine - an unexpected drop in NPS is something to investigate and get more information on what went wrong, usually from more detailed/free-form questions in the NPS questionnaire. As a software engineer I love reading our NPS reports, because they always have great tidbits of info that are often difficult to deduce just by looking at standard analytics.

ekianjo 5y ago
> primarily that there was a very strong correlation between the calculated NPS score and business performance. There is avsolutely no science linking performance and NPS. Just pseudo science.
Lazare 5y ago
> they always tend to ignore the underlying science that went into making NPS in the first place, primarily that there was a very strong correlation between the calculated NPS score and business performance. There is no underlying science behind NPS. It was an arbitrary measure with no research or model behind it. Also, it doesn't work very well, in large because because there is no strong correlation between NPS and anything else, because - as this article makes clear - NPS is an extremely noisy number. Look at the graphs - you can have enormous variations in NPS, with zero change in underlying customer sentiment. Obviously that can't correlate with anything useful. I think what you're doing is conflating the idea of "asking how likely people are to recommend your product" with NPS. The former is a decent idea, and it does have some (limited) science behind it! There really was a study done that really did find that question was an excellent predictor of many key results. (...and some others that found it was not, but hey, it's something.) So yes, asking people the question and reading the results is probably a good idea. ...but that's not what NPS is. > an unexpected drop in NPS is something to investigate It really, really isn't. And I would challenge you to point to any evidence to the contrary.

foo4u 5y ago

What’s more interesting to me than the NPS scores are the free form comment section below the 0-10 rating. Never mind that anything below X is considered neutral or a detractor. You asked a user / customer/ etc to rate you on a scale of 1-10 and then tell you why. Want to know what you’re truly screwing up on? Take the feedback on your 1-6 scores seriously and you can find the low hanging fruit to take a product from mediocre to a great user experience. That’s the true NPS value. It’s all about how you handle the feedback.

Gelob 5y ago

I hate NPS. You know what makes it shit? It requires your customers to fill it out. So then your bugging them to complete it

kqr 5y ago
How would you (reliably) measure user satisfaction without asking users?
muzani 5y ago
I love giving feedback though. If I can't give it direct, it's going through Google Maps or something.

rkk3 5y ago

> (The most irritating — and by far the most common — reason companies seem to measure NPS is that it’s standardized, but more importantly that everyone else does it. Which is fine for one-time-use benchmarking, but not a good basis for internal KPIs.) Except the alternative isn't to stop tracking NPS the alternative is to use unstandardized customer survey metric's, this is worse because you have no true industry comparison & can be gamed more easily since they define the formula!

itsdrewmiller 5y ago
https://www.qualtrics.com/experience-management/customer/what-is-csat/ https://www.qualtrics.com/experience-management/customer/wha... is another standardized alternative, that I suspect would suffer less from the problems in the article.
- rkk3 5y ago
  CSAT is not standardized and there are many ways to calculate it, one of which you have linked. I've seen firsthand many of the largest brands calculate it differently.
  - itsdrewmiller 5y ago
    Yeah, I agree it doesn't have the universal agreement on implementation that NPS does. Here's an alternative that I believe does and is referenced in some of the NPS skeptical research: https://en.wikipedia.org/wiki/American_Customer_Satisfaction_Index https://en.wikipedia.org/wiki/American_Customer_Satisfaction...
    - rkk3 5y ago
      That "measures the satisfaction of consumers across the U.S. economy", very different from Brand's Customer Experience Surveys. Having worked as a software vendor in the CX field, I saw a lot of different implementations of the "same" survey metrics. NPS was only standardized one (I think because its from b school in the 80's).

austenallred 5y ago

Yes, NPS is an imperfect measure, as all are. What I’ve never seen is a great argument for what one should use instead.

Dave3of5 5y ago
Company I was working with done an internal study on this I believe CSAT or a simple average were better if your goal is to improve revenue.

nhumrich 5y ago

The real problem with NPS is when a certain score is a goal. After all goodharts law states that is the case with any good measure.

ketralnis 5y ago
Yep. I had an advertising-funded app that asked me to fill out a survey about why I rated it poorly in a modal I can’t skip. So I always gave it a good score, until I stopped using it because it interrupted me so often to ask. They lost me and they’ll never know why

Lazare 5y ago

You know what would be vastly better than NPS? Ask people every so often how they're feeling about the product. Let them pick a thumbs up or a thumbs down. If they pick thumbs down, say "we're sorry you hear that. Please let us know what's bothering you", and display a little text box. Carefully read every comment you receive. Asking unhappy users for feedback is a great idea. The 0 to 10 scale and - especially - the bizarre calculation NPS does with it - is statistical malpractice that just isn't mathematically capable of doing what NPS supports claim.

kqr 5y ago
> Asking unhappy users for feedback is a great idea. The 0 to 10 scale and - especially - the bizarre calculation NPS does with it - is statistical malpractice I disagree. On the contrary, it is good survey practise to ask yes/no questions with a scale, to get information on intensity of opinion. (Often this is a scale 1--7 ranging from "strongly disagree" through "neutral" to "strongly agree", but there are other alternatives of which NPS is one.) Then you might think only the bottom half of the range would formally be considered "detractors", and that would (as far as the evidence goes) be accurate. However, we're not looking for how many people are detractors today -- we want to know how many people are at risk of becoming detractors in the near future. That's what makes NPS a somewhat leading metric. And that's also why you see the sleight of hand that counts answers up to 6 as detractors. I don't know what "mathematically capable" means in your context, nor what "NPS supports claim" in your experience, but the number absolutely makes sense.
paulmooreparks 5y ago
Be careful how you do this. When an app interrupts my normal workflow to ask me to rate the app or ask me how I'm enjoying the app, the answer is always negative. I'm using the app to get work done, not satisfy some VP's KPIs.

joe2784 5y ago

It's not fair to compare how "important" different metrics are to the whole business. A business is composed of and built on top of many sectors. That's why we have Finance team, Sales team, Marketing team, Product and Development team..., and not just one "we're a company" team! Free cash flow is important for business operations. Churn is important to predict sales. NPS, though not directly affecting the "money", might be a good indicator for the marketing team. There are a lot more important marketing metrics than NPS, but for a big corporation's branding team, especially in consumer goods industry, it might be useful to look at the "trend" of their NPS over a long period of time. NPS has noise just because of its nature of being collected with surveys. Like any questionnaires or surveys, if you design it well, you can reduce the noise.

throwawaysea 5y ago

This post is correct about two things - first, that NPS does not give you direction on how to act. Second, that statistical realities make sampling approaches like NPS very noisy. I don’t think either discounts the concept of NPS itself. “Bad statistical work” is going to be bad regardless of what metric we’re talking about. And a metrics being too abstract to action doesn’t make it useless - at least not any more so than something like free cash flow. Yes, connecting the dots to some outcome is difficult. But it’s a pulse on the business, and if done right, that has some inherent value.

Silasdev 5y ago

Why do everyone seem to think that management does not have actual good intentions with measuring NPS as a KPI? The companies I've worked with have all had genuinely good intentions making the best possible experience for the user, because that is what ultimately wins in the end. If someone scores 1-6, you ask them to provide additional feedback, to learn from it - problem solved and everyone has all the info they need to go about their work. There seems to be this idea that KPIs are evil and interpreted in vacuum. It's rarely like that.

Ozzie_osman 5y ago
> There seems to be this idea that KPIs are evil and interpreted in vacuum. Not just KPIs, but the Hackernews crowd seems to apply the same to management and corporations as a whole.
dade_ 5y ago
20% of companies are knocking it out of the park, and 50% are nowhere near a leaderboard. I expect such folks are correct in their observation, but work for the 50%. Perhaps you work for one that is getting a promoter score. My advice to friends looking for a job, ask them what their NPS is. The leaders in NPS are leaders in employee engagement. If they don’t know, or it’s bad and your role isn’t to make it better, look elsewhere.

kwhitefoot 5y ago

NPS = Net Promoter Score See https://en.wikipedia.org/wiki/Net_promoter_score https://en.wikipedia.org/wiki/Net_promoter_score

deleted 5y ago

[deleted]

meelford 5y ago

Background: I have designed and implemented NPS feedback systems in several companies. I noticed that people are concerned about this when it was not implemented correctly, i.e. as a complimentary question in the long questionnaire or as a follow-up question at the end of the journey. The NPS collection mechanic has many advantages: 1. The mental entry point for the user is very low - just 1 question. 2. It is very useful to track user feedback over their lifetime (ask every 6 months) and actively resolve issues if NPS shows it. 3. Another great thing is not a numerical feedback, but a written answer that gives a lot of really useful ideas for improvement. 4. And finally, you don't need any specific context to show the question, so it can be embedded in almost any stage of the user experience.

timkam 5y ago
Agree, the NPS aggregation is mostly BS, but getting a signal from median/mean, e.g. by comparing trends across some dimension, and reading complimentary feedback is quite useful.

thathndude 5y ago

I was unfamiliar with NPS until about 12 hours ago (but not because of this post). NPS has had a strong prediction of stock price (I learned this morning). https://www.marketwatch.com/story/this-surprising-investing-strategy-crushes-the-stock-market-without-examining-a-single-financial-metric-11638985326 https://www.marketwatch.com/story/this-surprising-investing-...? Of course, it’s just opinion. But maybe there’s some use to NPS after all.

dade_ 5y ago

Companies with a promoter level score during the pandemic grew their business. There is a direct correlation between a promoter NPS and revenue growth. The percentage of companies that now provide a promoter level service is now over 20%, up from 13% before the pandemic. The percentage that are rated a detractor level score also grew to nearly 50%. These org’s revenues either shrank or stagnated. This is an existential issue for organizations, not a nuisance KPI. Companies that are leaders in NPS are leaders in employee engagement. This post sounded pretty frustrated with an employer/contract… NPS is a measure of past experiences, it isn’t a GPS navigation system to improvement. It is also 1 question, so it is the most simple to collect. However even this gets screwed up all the time. Interfaces such as a telephone where 10 can be measured as the first digit 1, or PIN pads where they are a nuisance to a customer when in a hurry to pay and missed when paying with NFC. Understanding negative NPS is like understanding a traffic ticket with no other data. Why did I get a traffic ticket? Well, what did or didn’t you do and how fast were you doing it? Were you a jerk to the cop? Was the cop stressed by something else that day? To be a leader in NPS is an organizational effort, not the result of divining an insight from a few survey answers. Ideally, customer journeys maps have been developed and problem areas are self evident before an NPS score arrives.

samsolomon 5y ago

This post misses that the primary benefit of an NPS program is not the metric—that's a nice byproduct—it's the dialogue with customers. As a single metric Net Promoter Score is OK. It's extremely easy for customers to answer. So there's volume in the number of answers. You can learn more by segmenting different ways—Users vs Admins, Enterprise vs SMB, Verticals, etc.. The company I currently work for puts a lot into NPS, not necessarily the metric, but the process. Everyone that leaves feedback gets a follow-up email and interview from someone in Product or User Research. That feedback is then organized in Productboard where it is grouped with similar requests/issues. Overall those interviews heavily weigh what features we refine and build. For some background, I previously designed an NPS tool and have written about NPS pretty extensively. https://solomon.io/understanding-net-promoter-score/ https://solomon.io/understanding-net-promoter-score/