4 ms·

What subcontractor would accept this arrangement? I sure wouldn’t. It’s like an evil genie situation. You’re never going to define customer satisfaction metrics

by mlthoughts2018 7y ago

What subcontractor would accept this arrangement? I sure wouldn’t. It’s like an evil genie situation. You’re never going to define customer satisfaction metrics in a way that won’t end up being gamefied to circumvent the spirit of what you’re supposed to do to help customers in order to maximize some metrics. This issues can also be related to the company itself. What if there’s an undisclosed bug in your randomization process based on IP or time zone or something that tilts the satisfaction metrics via an almost impossible to detect skew in the customer distribution routed to subcontractor A instead of B? What if you the company introduce a policy or product change that makes customers unhappy even if the subcontractor is doing exactly what you asked them to do? Or what if subcontractors realize the only way to make people happy is to do something really expensive or unapproved by you?

DoctorOetker 7y ago
> .. skew in the customer distribution routed to subcontractor A instead of B? provably rolling a random dice is a solved problem, superstition that fluctuations in a cryptographically verifiable random coin flip would correlate with "pathologically dissatisfied customers" is just that: superstition. > What if you the company introduce a policy or product change that makes customers unhappy even if the subcontractor is doing exactly what you asked them to do? That would make the customers assigned to the other subcontractors equally unhappy, so the customer satisfaction feedback drop will be the same for the other competing subcontractor, and so the budget division will be unaffected, but still tilted proportional to the gross difference in performance between subcontractor A and B > Or what if subcontractors realize the only way to make people happy is to do something really expensive or unapproved by you? What do you mean? reverse engineer a driver and rewrite a better one?
- mlthoughts2018 7y ago
  > provably rolling a random dice is a solved problem You clearly have no experience with setting up randomized trials in reality. It’s very hard. You think you’re flipping a coin, and totally forget that there are all kinds of things that can creep in to introduce confounding factors that correlate with assignment to one group or the other. Time of day, time of week, operating hours, handling customers waiting in a queue too long by rerouting them, random outage in subcontractor A’s cloud service account, geopolitical event causes problems with payment processing to subcontractor B for a certain time period. How will you isolate the effect on customer satisfaction of each subcontractor, controlling for all effects that could drive the observed satisfaction metric for a given subcontractor that are not in that subcontractor’s control? How will you make certain you’ve done this, as opposed to just making the deeply invalid assumption that some seemingly random assignment mechanism has automatically taken care of it? How will you actually verify the populations of customers sent to each subcontractor are actually probabilistically equivalent across every potential confounding factor? Acting like it’s just going to be a simple coin flip is a woeful lack of imagination.
  - DoctorOetker 7y ago
    there is no random trial to set up? it's just plain simple provably random assignment
    - mlthoughts2018 7y ago
      Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably” random assignment) but you’re not explaining how you will verify your assignment mechanism didn’t result in a lack of probabilistic equivalence (and thus invalidate a simple comparison between the two groups). The key part you’re missing is that it doesn’t matter if you believe you are choosing uniformly between each subcontractor. You might have a perfect random number generator, doesn’t matter. The ability to pick “50/50” between them has pretty much nothing to do with the causes for confounding factors to invalidate the comparison. You might happily go on calling your perfect random number generator and checking if it’s less than 0.5 as your decision rule, and yet still be biasing the population directed to one subcontractor vs the other, due to factors you’re not accounting for.
      - DoctorOetker 7y ago
        Unlike organizing a randomized trial -where each extra datapoint to gather would cost money and one tries to get the most unbiased result for the low total number of datapoints- in this economical setting of an endless and continuously growing number of processed customers one is not interested in how fast the relative error decreases, but that it converges on a fair value. This is just the law of large numbers: the expectation value of the correlation of pesky clients with the output from a provably fair coin is by definition zero. So yes if 1 out of 10 customers is a pesky (say unhelpable by attitude) customer, and we look at the first ~10 customers, then yes, probably it will result in the unfair assignment where one of the subcontractors got a pesky customer and the other didnt. But this is not a randomized trial with a finite low number of customer tickets. Treat the problem like a poisson process: after a short amount of time the accuracy on gamma is low, but after the millionth ticket the accuracy on gamma is high, and we are effectively discussing fractions of a cent. I assume you know how to throw a provably random dice: for example both subcontractors generate a secret randon number, hash it, publish the hash, then reveal the nonce, sum the nonces: even -> A, odd -> B This is not a medical experiment with a limited budget to study most efficiently with the low number of costly datapoints, this is not about the design of experiments. You seem to believe a lock-in amplifier could never work? That you can't raise a signal from below the noise floor? or is it really just the transient fluctuation that average each other out by the law of large numbers?
- extrapickles 7y ago
  Giving customers steep discounts, free upgrades, etc. You would have to track all the different ways they could give a break to your customers and somehow work that into the metrics without causing them to game them.
  - DoctorOetker 7y ago
    1) Even if a subcontractor, bribes the customer, why would the original company care? why would the customer care if he happily takes the money? Why would the opposing subcontractor care? the bribing subcontractor is basically betting the other company is not as performant as the bribe per customer satisfaction. 2) why on earth do you think the subcontractors would want to bribe the customers? they will just file a new complain to milk another refund, who may end up returning to you, so the first bribe was for naught, or may end up with the other subcontractor, you are basically paying a markov chain to stochastically push the customer to the other subcontractor, which will have a certain value, unless the other is simply better able to serve that customer, and then the other subcontractor will get more from the original company AND you lost the bribe to the customer... so subcontractors won't participate in bribing

DoctorOetker 7y ago

> .. skew in the customer distribution routed to subcontractor A instead of B? provably rolling a random dice is a solved problem, superstition that fluctuations in a cryptographically verifiable random coin flip would correlate with "pathologically dissatisfied customers" is just that: superstition. > What if you the company introduce a policy or product change that makes customers unhappy even if the subcontractor is doing exactly what you asked them to do? That would make the customers assigned to the other subcontractors equally unhappy, so the customer satisfaction feedback drop will be the same for the other competing subcontractor, and so the budget division will be unaffected, but still tilted proportional to the gross difference in performance between subcontractor A and B > Or what if subcontractors realize the only way to make people happy is to do something really expensive or unapproved by you? What do you mean? reverse engineer a driver and rewrite a better one?

mlthoughts2018 7y ago
> provably rolling a random dice is a solved problem You clearly have no experience with setting up randomized trials in reality. It’s very hard. You think you’re flipping a coin, and totally forget that there are all kinds of things that can creep in to introduce confounding factors that correlate with assignment to one group or the other. Time of day, time of week, operating hours, handling customers waiting in a queue too long by rerouting them, random outage in subcontractor A’s cloud service account, geopolitical event causes problems with payment processing to subcontractor B for a certain time period. How will you isolate the effect on customer satisfaction of each subcontractor, controlling for all effects that could drive the observed satisfaction metric for a given subcontractor that are not in that subcontractor’s control? How will you make certain you’ve done this, as opposed to just making the deeply invalid assumption that some seemingly random assignment mechanism has automatically taken care of it? How will you actually verify the populations of customers sent to each subcontractor are actually probabilistically equivalent across every potential confounding factor? Acting like it’s just going to be a simple coin flip is a woeful lack of imagination.
- DoctorOetker 7y ago
  there is no random trial to set up? it's just plain simple provably random assignment
  - mlthoughts2018 7y ago
    Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably” random assignment) but you’re not explaining how you will verify your assignment mechanism didn’t result in a lack of probabilistic equivalence (and thus invalidate a simple comparison between the two groups). The key part you’re missing is that it doesn’t matter if you believe you are choosing uniformly between each subcontractor. You might have a perfect random number generator, doesn’t matter. The ability to pick “50/50” between them has pretty much nothing to do with the causes for confounding factors to invalidate the comparison. You might happily go on calling your perfect random number generator and checking if it’s less than 0.5 as your decision rule, and yet still be biasing the population directed to one subcontractor vs the other, due to factors you’re not accounting for.
    - DoctorOetker 7y ago
      Unlike organizing a randomized trial -where each extra datapoint to gather would cost money and one tries to get the most unbiased result for the low total number of datapoints- in this economical setting of an endless and continuously growing number of processed customers one is not interested in how fast the relative error decreases, but that it converges on a fair value. This is just the law of large numbers: the expectation value of the correlation of pesky clients with the output from a provably fair coin is by definition zero. So yes if 1 out of 10 customers is a pesky (say unhelpable by attitude) customer, and we look at the first ~10 customers, then yes, probably it will result in the unfair assignment where one of the subcontractors got a pesky customer and the other didnt. But this is not a randomized trial with a finite low number of customer tickets. Treat the problem like a poisson process: after a short amount of time the accuracy on gamma is low, but after the millionth ticket the accuracy on gamma is high, and we are effectively discussing fractions of a cent. I assume you know how to throw a provably random dice: for example both subcontractors generate a secret randon number, hash it, publish the hash, then reveal the nonce, sum the nonces: even -> A, odd -> B This is not a medical experiment with a limited budget to study most efficiently with the low number of costly datapoints, this is not about the design of experiments. You seem to believe a lock-in amplifier could never work? That you can't raise a signal from below the noise floor? or is it really just the transient fluctuation that average each other out by the law of large numbers?
extrapickles 7y ago
Giving customers steep discounts, free upgrades, etc. You would have to track all the different ways they could give a break to your customers and somehow work that into the metrics without causing them to game them.
- DoctorOetker 7y ago
  1) Even if a subcontractor, bribes the customer, why would the original company care? why would the customer care if he happily takes the money? Why would the opposing subcontractor care? the bribing subcontractor is basically betting the other company is not as performant as the bribe per customer satisfaction. 2) why on earth do you think the subcontractors would want to bribe the customers? they will just file a new complain to milk another refund, who may end up returning to you, so the first bribe was for naught, or may end up with the other subcontractor, you are basically paying a markov chain to stochastically push the customer to the other subcontractor, which will have a certain value, unless the other is simply better able to serve that customer, and then the other subcontractor will get more from the original company AND you lost the bribe to the customer... so subcontractors won't participate in bribing