3 ms·
But there’s an opportunity cost that needs to be factored in when waiting for a stronger signal.
by renjimen 1y ago
But there’s an opportunity cost that needs to be factored in when waiting for a stronger signal.
- Nevermark 1y agoOne solution is to gradually move instances to you most likely solution. But continue a percentage of A/B/n testing as well. This allows for a balancing of speed vs. certainty
- imachine1980_ 1y agodo you use any tool for this, or simply crunk up slightly the dial each day
- travisjungroth 1y agoThere are multi armed bandit algorithms for this. I don’t know the names of the public tools. This is especially useful for something where the value of the choice is front loaded, like headlines.
- hruk 1y agoWe've used this Python package to do this: https://github.com/bayesianbandits/bayesianbandits https://github.com/bayesianbandits/bayesianbandits
- epgui 1y agoEven if you have to be honest with yourself about how much you care about being right, there’s still a place for balancing priorities. Two things can be true at once. Sometimes someone just has to make imperfect decisions based on incomplete information, or make arbitrary judgment calls. And that’s totally fine… But it shouldn’t be confused with data-driven decisions. The two kinds of decisions need to happen. They can both happen honestly.
- scott_w 1y agoThere is but you can decide that up front. There’s tools that will show you how long it’ll take to get statistical significance. You can then decide if you want to wait that long or have a softer p-value.