3 ms·

consider a total budget for customer service, why not hire 2 or more customer service subcontractors on the following contract: customer questions are randomly

consider a total budget for customer service, why not hire 2 or more customer service subcontractors on the following contract: customer questions are randomly assigned, and money flows proportionally toward the company with the better customer satisfaction scores? any question a subcontractor has not enough information for will be between the customer service contractor and the original company, and they don't need to be afraid of insisting the original company resolve a real fundamental issue with the product as the other subcontractor faces the same issues, which has the side effect of the subcontractors summarizing a large number of complaints for the original companies product engineers etc consider how for a long time many manufacturers didn't want to spend the time and effort to fix some linux driver, not because they couldn't but because they didn't want to plough through 10k complaints or requests, of unknown similarity or difference, and having the original high wage engineers plough through them is uneconomical, so have the subcontractor summarize the issue concisely yet precisely

mlthoughts2018 7y ago
What subcontractor would accept this arrangement? I sure wouldn’t. It’s like an evil genie situation. You’re never going to define customer satisfaction metrics in a way that won’t end up being gamefied to circumvent the spirit of what you’re supposed to do to help customers in order to maximize some metrics. This issues can also be related to the company itself. What if there’s an undisclosed bug in your randomization process based on IP or time zone or something that tilts the satisfaction metrics via an almost impossible to detect skew in the customer distribution routed to subcontractor A instead of B? What if you the company introduce a policy or product change that makes customers unhappy even if the subcontractor is doing exactly what you asked them to do? Or what if subcontractors realize the only way to make people happy is to do something really expensive or unapproved by you?
- DoctorOetker 7y ago
  > .. skew in the customer distribution routed to subcontractor A instead of B? provably rolling a random dice is a solved problem, superstition that fluctuations in a cryptographically verifiable random coin flip would correlate with "pathologically dissatisfied customers" is just that: superstition. > What if you the company introduce a policy or product change that makes customers unhappy even if the subcontractor is doing exactly what you asked them to do? That would make the customers assigned to the other subcontractors equally unhappy, so the customer satisfaction feedback drop will be the same for the other competing subcontractor, and so the budget division will be unaffected, but still tilted proportional to the gross difference in performance between subcontractor A and B > Or what if subcontractors realize the only way to make people happy is to do something really expensive or unapproved by you? What do you mean? reverse engineer a driver and rewrite a better one?
  - mlthoughts2018 7y ago
    > provably rolling a random dice is a solved problem You clearly have no experience with setting up randomized trials in reality. It’s very hard. You think you’re flipping a coin, and totally forget that there are all kinds of things that can creep in to introduce confounding factors that correlate with assignment to one group or the other. Time of day, time of week, operating hours, handling customers waiting in a queue too long by rerouting them, random outage in subcontractor A’s cloud service account, geopolitical event causes problems with payment processing to subcontractor B for a certain time period. How will you isolate the effect on customer satisfaction of each subcontractor, controlling for all effects that could drive the observed satisfaction metric for a given subcontractor that are not in that subcontractor’s control? How will you make certain you’ve done this, as opposed to just making the deeply invalid assumption that some seemingly random assignment mechanism has automatically taken care of it? How will you actually verify the populations of customers sent to each subcontractor are actually probabilistically equivalent across every potential confounding factor? Acting like it’s just going to be a simple coin flip is a woeful lack of imagination.
    - DoctorOetker 7y ago
      there is no random trial to set up? it's just plain simple provably random assignment
      - mlthoughts2018 7y ago
        Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably” random assignment) but you’re not explaining how you will verify your assignment mechanism didn’t result in a lack of probabilistic equivalence (and thus invalidate a simple comparison between the two groups). The key part you’re missing is that it doesn’t matter if you believe you are choosing uniformly between each subcontractor. You might have a perfect random number generator, doesn’t matter. The ability to pick “50/50” between them has pretty much nothing to do with the causes for confounding factors to invalidate the comparison. You might happily go on calling your perfect random number generator and checking if it’s less than 0.5 as your decision rule, and yet still be biasing the population directed to one subcontractor vs the other, due to factors you’re not accounting for.
b_tterc_p 7y ago
Kind of like Thompson sampling Wouldn’t work for the subcontractors though. Margins are too thin to have a volatile demand like that.

mlthoughts2018 7y ago

What subcontractor would accept this arrangement? I sure wouldn’t. It’s like an evil genie situation. You’re never going to define customer satisfaction metrics in a way that won’t end up being gamefied to circumvent the spirit of what you’re supposed to do to help customers in order to maximize some metrics. This issues can also be related to the company itself. What if there’s an undisclosed bug in your randomization process based on IP or time zone or something that tilts the satisfaction metrics via an almost impossible to detect skew in the customer distribution routed to subcontractor A instead of B? What if you the company introduce a policy or product change that makes customers unhappy even if the subcontractor is doing exactly what you asked them to do? Or what if subcontractors realize the only way to make people happy is to do something really expensive or unapproved by you?

DoctorOetker 7y ago
> .. skew in the customer distribution routed to subcontractor A instead of B? provably rolling a random dice is a solved problem, superstition that fluctuations in a cryptographically verifiable random coin flip would correlate with "pathologically dissatisfied customers" is just that: superstition. > What if you the company introduce a policy or product change that makes customers unhappy even if the subcontractor is doing exactly what you asked them to do? That would make the customers assigned to the other subcontractors equally unhappy, so the customer satisfaction feedback drop will be the same for the other competing subcontractor, and so the budget division will be unaffected, but still tilted proportional to the gross difference in performance between subcontractor A and B > Or what if subcontractors realize the only way to make people happy is to do something really expensive or unapproved by you? What do you mean? reverse engineer a driver and rewrite a better one?
- mlthoughts2018 7y ago
  > provably rolling a random dice is a solved problem You clearly have no experience with setting up randomized trials in reality. It’s very hard. You think you’re flipping a coin, and totally forget that there are all kinds of things that can creep in to introduce confounding factors that correlate with assignment to one group or the other. Time of day, time of week, operating hours, handling customers waiting in a queue too long by rerouting them, random outage in subcontractor A’s cloud service account, geopolitical event causes problems with payment processing to subcontractor B for a certain time period. How will you isolate the effect on customer satisfaction of each subcontractor, controlling for all effects that could drive the observed satisfaction metric for a given subcontractor that are not in that subcontractor’s control? How will you make certain you’ve done this, as opposed to just making the deeply invalid assumption that some seemingly random assignment mechanism has automatically taken care of it? How will you actually verify the populations of customers sent to each subcontractor are actually probabilistically equivalent across every potential confounding factor? Acting like it’s just going to be a simple coin flip is a woeful lack of imagination.
  - DoctorOetker 7y ago
    there is no random trial to set up? it's just plain simple provably random assignment
    - mlthoughts2018 7y ago
      Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably” random assignment) but you’re not explaining how you will verify your assignment mechanism didn’t result in a lack of probabilistic equivalence (and thus invalidate a simple comparison between the two groups). The key part you’re missing is that it doesn’t matter if you believe you are choosing uniformly between each subcontractor. You might have a perfect random number generator, doesn’t matter. The ability to pick “50/50” between them has pretty much nothing to do with the causes for confounding factors to invalidate the comparison. You might happily go on calling your perfect random number generator and checking if it’s less than 0.5 as your decision rule, and yet still be biasing the population directed to one subcontractor vs the other, due to factors you’re not accounting for.

b_tterc_p 7y ago

Kind of like Thompson sampling Wouldn’t work for the subcontractors though. Margins are too thin to have a volatile demand like that.