4 ms·

Do you publish response rates? Things like this always scream selection bias to me. If it were, say 90/7/3, but it took 300 interactions to get 100 responses,

Do you publish response rates? Things like this always scream selection bias to me. If it were, say 90/7/3, but it took 300 interactions to get 100 responses, that seems like a misrepresentation. That being said - I really appreciate the idea, but it only really works if you trust the company already. I have plenty of reason to believe 37s (even applied to be a part of the team being tracked,) but if this were State Farm or Comcast or something like that, it would be a pretty laughable concept.

jasonfried 16y ago
I believe around 15% of our customers respond with a rating. From what I understand that's pretty high as far as these sorts of things go. I think that would be considered a reasonable sample. We don't ask randomly, we ask everyone, so everyone is free to offer up a rating. CORRECTION: Just reviewed the stats again and it looks like the response rate is in the mid 30s.
- alexophile 16y ago
  Right, I didn't mean to imply that this was deliberately misleading, just that I would imagine these things to be far more useful internally than to the casual observer. (i.e. it would be great for spotting consistent sore spots and the like, but not necessarily a statistically viable indicator of customer happiness) My guess would be that the biggest benefit from this is going to be when people hear about how great the customer service was at 37s, stop by to check it out, and see a chart full of smiley faces, each one potentially representing an echo of the great experience they heard about. PS: I would like to add a frowny face for never getting a response to my application :(
- clain 16y ago
  Jason, the issue is not whether the sample is large enough--a 30+% response rate strikes me as very high, and I suspect given your support volume you're getting several hundred responses per day. The issue, instead, is whether the people who choose to respond--to take another step after their support encounter and rate the quality of service--are representative of all users, including the majority who do not rate the experience. That's a tough hypothesis to test, and I suspect it could cut either way: the happiest customers may choose to respond since they so enjoyed the experience, or the angriest may choose to respond since they want you to be aware of their disappointment. Finding out which (if either) of these is occurring is tricky, though. That said, I certainly don't mean to impugn what you're doing here. I love the transparency and I think you probably are getting a fairly good snapshot of overall customer satisfaction. Also, even if there is a response bias, you can safely use these numbers to benchmark over time, which I suspect would be very useful.

jasonfried 16y ago

I believe around 15% of our customers respond with a rating. From what I understand that's pretty high as far as these sorts of things go. I think that would be considered a reasonable sample. We don't ask randomly, we ask everyone, so everyone is free to offer up a rating. CORRECTION: Just reviewed the stats again and it looks like the response rate is in the mid 30s.

alexophile 16y ago
Right, I didn't mean to imply that this was deliberately misleading, just that I would imagine these things to be far more useful internally than to the casual observer. (i.e. it would be great for spotting consistent sore spots and the like, but not necessarily a statistically viable indicator of customer happiness) My guess would be that the biggest benefit from this is going to be when people hear about how great the customer service was at 37s, stop by to check it out, and see a chart full of smiley faces, each one potentially representing an echo of the great experience they heard about. PS: I would like to add a frowny face for never getting a response to my application :(
clain 16y ago
Jason, the issue is not whether the sample is large enough--a 30+% response rate strikes me as very high, and I suspect given your support volume you're getting several hundred responses per day. The issue, instead, is whether the people who choose to respond--to take another step after their support encounter and rate the quality of service--are representative of all users, including the majority who do not rate the experience. That's a tough hypothesis to test, and I suspect it could cut either way: the happiest customers may choose to respond since they so enjoyed the experience, or the angriest may choose to respond since they want you to be aware of their disappointment. Finding out which (if either) of these is occurring is tricky, though. That said, I certainly don't mean to impugn what you're doing here. I love the transparency and I think you probably are getting a fairly good snapshot of overall customer satisfaction. Also, even if there is a response bias, you can safely use these numbers to benchmark over time, which I suspect would be very useful.