3 ms·

Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably”

by mlthoughts2018 7y ago

Huh? The entire A/B test you are proposing is a randomized trial. You are proposing a random assignment mechanism (what you seem to misunderstand as “provably” random assignment) but you’re not explaining how you will verify your assignment mechanism didn’t result in a lack of probabilistic equivalence (and thus invalidate a simple comparison between the two groups). The key part you’re missing is that it doesn’t matter if you believe you are choosing uniformly between each subcontractor. You might have a perfect random number generator, doesn’t matter. The ability to pick “50/50” between them has pretty much nothing to do with the causes for confounding factors to invalidate the comparison. You might happily go on calling your perfect random number generator and checking if it’s less than 0.5 as your decision rule, and yet still be biasing the population directed to one subcontractor vs the other, due to factors you’re not accounting for.

DoctorOetker 7y ago
Unlike organizing a randomized trial -where each extra datapoint to gather would cost money and one tries to get the most unbiased result for the low total number of datapoints- in this economical setting of an endless and continuously growing number of processed customers one is not interested in how fast the relative error decreases, but that it converges on a fair value. This is just the law of large numbers: the expectation value of the correlation of pesky clients with the output from a provably fair coin is by definition zero. So yes if 1 out of 10 customers is a pesky (say unhelpable by attitude) customer, and we look at the first ~10 customers, then yes, probably it will result in the unfair assignment where one of the subcontractors got a pesky customer and the other didnt. But this is not a randomized trial with a finite low number of customer tickets. Treat the problem like a poisson process: after a short amount of time the accuracy on gamma is low, but after the millionth ticket the accuracy on gamma is high, and we are effectively discussing fractions of a cent. I assume you know how to throw a provably random dice: for example both subcontractors generate a secret randon number, hash it, publish the hash, then reveal the nonce, sum the nonces: even -> A, odd -> B This is not a medical experiment with a limited budget to study most efficiently with the low number of costly datapoints, this is not about the design of experiments. You seem to believe a lock-in amplifier could never work? That you can't raise a signal from below the noise floor? or is it really just the transient fluctuation that average each other out by the law of large numbers?

DoctorOetker 7y ago

Unlike organizing a randomized trial -where each extra datapoint to gather would cost money and one tries to get the most unbiased result for the low total number of datapoints- in this economical setting of an endless and continuously growing number of processed customers one is not interested in how fast the relative error decreases, but that it converges on a fair value. This is just the law of large numbers: the expectation value of the correlation of pesky clients with the output from a provably fair coin is by definition zero. So yes if 1 out of 10 customers is a pesky (say unhelpable by attitude) customer, and we look at the first ~10 customers, then yes, probably it will result in the unfair assignment where one of the subcontractors got a pesky customer and the other didnt. But this is not a randomized trial with a finite low number of customer tickets. Treat the problem like a poisson process: after a short amount of time the accuracy on gamma is low, but after the millionth ticket the accuracy on gamma is high, and we are effectively discussing fractions of a cent. I assume you know how to throw a provably random dice: for example both subcontractors generate a secret randon number, hash it, publish the hash, then reveal the nonce, sum the nonces: even -> A, odd -> B This is not a medical experiment with a limited budget to study most efficiently with the low number of costly datapoints, this is not about the design of experiments. You seem to believe a lock-in amplifier could never work? That you can't raise a signal from below the noise floor? or is it really just the transient fluctuation that average each other out by the law of large numbers?