2 ms·
Sure, but that's not really A/B testing, those are more often called staged rollouts or progressive rollouts.
by ratww 4y ago
Sure, but that's not really A/B testing, those are more often called staged rollouts or progressive rollouts.
- tbranyen 4y agoI'm talking about running week or month long tests with control and multi test cells containing new functionality, configuration, or code to determine the viability of a single or combination of changes by analyzing statistical output driven by p-value and pre-determined target metrics. These types of experiments are extremely valuable in uncovering hard-to-find bugs, assuming you have sufficient logging and confidence around your metrics. They let you know a problem exists and roughly where it is in the product. From there you can drill down and investigate your source code until the discrepancy is found.
- EricMausler 4y agoThis makes sense to me. Not the kind of AB testing I had in mind, but fair point. I was thinking more about decision making processes, not operations troubleshooting.
- ratww 4y agoYes, I know this and I agree. I understand they use the same techniques and terminologies, but this is just not the kinda thing the article is criticizing.