I'm in a similar position in the sense, but because the turn around on our experiments can be super fast. Instead of using a Chi-square p-value type test, or e

I'm in a similar position in the sense, but because the turn around on our experiments can be super fast. Instead of using a Chi-square p-value type test, or even a Bayesian implementation (like Beta: alpha = 5, beta = 100), that survival analysis gives the profile of the difference. I'm in a similar position because we did choose to pollute our code base with inline experiments (which we shut down and remove). However, ideally I want these experiments to "run" forever once the winner is installed. This is because I can always compare it back to the survival curve of the 'control' group in the experiment (and test group) to make sure it's behaving as expected. So my problem is, how do I escape from inline and yet continue to record everything afterwards? Been thinking a lot about it.