The challenge with A/B experiments is how you design them to have sufficient power and draw a meaningful conclusion out of them. So, you either need a big % dif

The challenge with A/B experiments is how you design them to have sufficient power and draw a meaningful conclusion out of them. So, you either need a big % difference between the test and the control, or you need a big number of samples. LLM apps usually don’t meet either of those two criteria. Have you ran into this with your users?