3 ms·

> provably rolling a random dice is a solved problem You clearly have no experience with setting up randomized trials in reality. It’s very hard. You think you

by mlthoughts2018 7y ago

> provably rolling a random dice is a solved problem You clearly have no experience with setting up randomized trials in reality. It’s very hard. You think you’re flipping a coin, and totally forget that there are all kinds of things that can creep in to introduce confounding factors that correlate with assignment to one group or the other. Time of day, time of week, operating hours, handling customers waiting in a queue too long by rerouting them, random outage in subcontractor A’s cloud service account, geopolitical event causes problems with payment processing to subcontractor B for a certain time period. How will you isolate the effect on customer satisfaction of each subcontractor, controlling for all effects that could drive the observed satisfaction metric for a given subcontractor that are not in that subcontractor’s control? How will you make certain you’ve done this, as opposed to just making the deeply invalid assumption that some seemingly random assignment mechanism has automatically taken care of it? How will you actually verify the populations of customers sent to each subcontractor are actually probabilistically equivalent across every potential confounding factor? Acting like it’s just going to be a simple coin flip is a woeful lack of imagination.

DoctorOetker 7y ago
there is no random trial to set up? it's just plain simple provably random assignment
- mlthoughts2018 7y ago
  Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably” random assignment) but you’re not explaining how you will verify your assignment mechanism didn’t result in a lack of probabilistic equivalence (and thus invalidate a simple comparison between the two groups). The key part you’re missing is that it doesn’t matter if you believe you are choosing uniformly between each subcontractor. You might have a perfect random number generator, doesn’t matter. The ability to pick “50/50” between them has pretty much nothing to do with the causes for confounding factors to invalidate the comparison. You might happily go on calling your perfect random number generator and checking if it’s less than 0.5 as your decision rule, and yet still be biasing the population directed to one subcontractor vs the other, due to factors you’re not accounting for.
  - DoctorOetker 7y ago
    Unlike organizing a randomized trial -where each extra datapoint to gather would cost money and one tries to get the most unbiased result for the low total number of datapoints- in this economical setting of an endless and continuously growing number of processed customers one is not interested in how fast the relative error decreases, but that it converges on a fair value. This is just the law of large numbers: the expectation value of the correlation of pesky clients with the output from a provably fair coin is by definition zero. So yes if 1 out of 10 customers is a pesky (say unhelpable by attitude) customer, and we look at the first ~10 customers, then yes, probably it will result in the unfair assignment where one of the subcontractors got a pesky customer and the other didnt. But this is not a randomized trial with a finite low number of customer tickets. Treat the problem like a poisson process: after a short amount of time the accuracy on gamma is low, but after the millionth ticket the accuracy on gamma is high, and we are effectively discussing fractions of a cent. I assume you know how to throw a provably random dice: for example both subcontractors generate a secret randon number, hash it, publish the hash, then reveal the nonce, sum the nonces: even -> A, odd -> B This is not a medical experiment with a limited budget to study most efficiently with the low number of costly datapoints, this is not about the design of experiments. You seem to believe a lock-in amplifier could never work? That you can't raise a signal from below the noise floor? or is it really just the transient fluctuation that average each other out by the law of large numbers?

DoctorOetker 7y ago

there is no random trial to set up? it's just plain simple provably random assignment

mlthoughts2018 7y ago
Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably” random assignment) but you’re not explaining how you will verify your assignment mechanism didn’t result in a lack of probabilistic equivalence (and thus invalidate a simple comparison between the two groups). The key part you’re missing is that it doesn’t matter if you believe you are choosing uniformly between each subcontractor. You might have a perfect random number generator, doesn’t matter. The ability to pick “50/50” between them has pretty much nothing to do with the causes for confounding factors to invalidate the comparison. You might happily go on calling your perfect random number generator and checking if it’s less than 0.5 as your decision rule, and yet still be biasing the population directed to one subcontractor vs the other, due to factors you’re not accounting for.
- DoctorOetker 7y ago
  Unlike organizing a randomized trial -where each extra datapoint to gather would cost money and one tries to get the most unbiased result for the low total number of datapoints- in this economical setting of an endless and continuously growing number of processed customers one is not interested in how fast the relative error decreases, but that it converges on a fair value. This is just the law of large numbers: the expectation value of the correlation of pesky clients with the output from a provably fair coin is by definition zero. So yes if 1 out of 10 customers is a pesky (say unhelpable by attitude) customer, and we look at the first ~10 customers, then yes, probably it will result in the unfair assignment where one of the subcontractors got a pesky customer and the other didnt. But this is not a randomized trial with a finite low number of customer tickets. Treat the problem like a poisson process: after a short amount of time the accuracy on gamma is low, but after the millionth ticket the accuracy on gamma is high, and we are effectively discussing fractions of a cent. I assume you know how to throw a provably random dice: for example both subcontractors generate a secret randon number, hash it, publish the hash, then reveal the nonce, sum the nonces: even -> A, odd -> B This is not a medical experiment with a limited budget to study most efficiently with the low number of costly datapoints, this is not about the design of experiments. You seem to believe a lock-in amplifier could never work? That you can't raise a signal from below the noise floor? or is it really just the transient fluctuation that average each other out by the law of large numbers?