Yeah, classical A/B testing only works if you specify the duration of time that the experiment will run in advance and stick to it. There are ways that you can

Yeah, classical A/B testing only works if you specify the duration of time that the experiment will run in advance and stick to it. There are ways that you can continuously monitor how an experiment is performing (Google "sequential Bayesian analysis") but the math is quite a bit more complex and nuanced. Honestly, for most startups a simple multi-armed bandit approach is probably the way to go. Don't worry about statistical significance; just throw some "lite" reinforcement learning on top of your product's aesthetics and enjoy the incremental profit. (Caveat: do not apply MAB to major product changes.)