3 ms·

This is a frustrating piece to read as a startup founder and product manager because I don't think the author understand how much companies already do think abo

This is a frustrating piece to read as a startup founder and product manager because I don't think the author understand how much companies already do think about the things established in the piece. > "If thumbs-ups or ratings on a five-point scale are not automatically useful, what kind of feedback would be? Finely tuned feedback that targets the system it’s meant to regulate will always surpass a barrage of angry or ecstatic reviews. Rather than trumpeting the desirability of all feedback, apps and review sites should pursue only the information that is crucial for making the system work better." It's hard to know what data points correlate to making the system work better until after the fact — so developers generally have to "over ask" so you can back test data to outcomes. Probably the best criticism of the proliferation around feedback is that it doesn't always make the customer's experience better, it just creates better conditions for the company to make more money.

safsexisgoodsex 8y ago
I think the author's point is much more around intent. Asking for feedback is one thing. Having a plan in place to do something with it is another. It's so easy to bake in a feedback loop but that doesn't translate to any actual change. NPS is popular not because it's necessarily useful, but because it's easy to understand and track. I worked in market research for years and our NPS remained flat because we didn't have the right means of sharing and implementing the feedback we gathered. The same themes came up over and over again over the years and yet we never changed our actual product. This is the point the author is touching on.
chiefalchemist 8y ago
re: "doesn't always make the customer's experience better," Does it really if it does? Or is people __believing__ it does good enough? If your comfortable with the latter then such feedback will create confirmation bias. Which the product dev team (also being human) is also going to fall victim to.
taeric 8y ago
"back test data to outcomes" sounds scarily like p-hacking. That is, it seems highly likely you will still not find anything actually meaningful. You are just guaranteeing that you will be able to find something. This is especially true if you have retention numbers, already. Why bother asking if I was happy with a purchase, if I never buy again anyway. Note that I am sympathetic. There is every chance that I will buy something again simply because the exit survey reminded me of your existence. There is also the chance having an exit survey will make me not want to come back. :)
- stuartaxelowen 8y ago
  Not so - the business is looking for real ROI, and thus will not be satisfied with topics that are "in the noise". Also, you create the measurement that establishes a problem before you apply the treatment (create new feature X or fix bug Y), and then verify after treatment that the desired affect was achieved.
  - taeric 8y ago
    If you are scanning historic data looking for correlations, you will almost certainly fall for some of the same traps that are common in p hacking. Now, can you mitigate some by carefully doing an experiment again? Certainly. But you should also not constrain experiments to only things to have been measuring already.
nitwit005 8y ago
The more questions you ask on a survey, the more the results tend to be garbage. Either people will give up on it, or they'll fill in everything with the same value.

safsexisgoodsex 8y ago

I think the author's point is much more around intent. Asking for feedback is one thing. Having a plan in place to do something with it is another. It's so easy to bake in a feedback loop but that doesn't translate to any actual change. NPS is popular not because it's necessarily useful, but because it's easy to understand and track. I worked in market research for years and our NPS remained flat because we didn't have the right means of sharing and implementing the feedback we gathered. The same themes came up over and over again over the years and yet we never changed our actual product. This is the point the author is touching on.

chiefalchemist 8y ago

re: "doesn't always make the customer's experience better," Does it really if it does? Or is people __believing__ it does good enough? If your comfortable with the latter then such feedback will create confirmation bias. Which the product dev team (also being human) is also going to fall victim to.

taeric 8y ago

"back test data to outcomes" sounds scarily like p-hacking. That is, it seems highly likely you will still not find anything actually meaningful. You are just guaranteeing that you will be able to find something. This is especially true if you have retention numbers, already. Why bother asking if I was happy with a purchase, if I never buy again anyway. Note that I am sympathetic. There is every chance that I will buy something again simply because the exit survey reminded me of your existence. There is also the chance having an exit survey will make me not want to come back. :)

stuartaxelowen 8y ago
Not so - the business is looking for real ROI, and thus will not be satisfied with topics that are "in the noise". Also, you create the measurement that establishes a problem before you apply the treatment (create new feature X or fix bug Y), and then verify after treatment that the desired affect was achieved.
- taeric 8y ago
  If you are scanning historic data looking for correlations, you will almost certainly fall for some of the same traps that are common in p hacking. Now, can you mitigate some by carefully doing an experiment again? Certainly. But you should also not constrain experiments to only things to have been measuring already.

nitwit005 8y ago

The more questions you ask on a survey, the more the results tend to be garbage. Either people will give up on it, or they'll fill in everything with the same value.