3 ms·
So pure theoretical thinking now, but: 1) running multiple tests in parallel becomes important where the product/dev frequency exceeds waiting for a single tes
by gojkoa 1mo ago
So pure theoretical thinking now, but:
1) running multiple tests in parallel becomes important where the product/dev frequency exceeds waiting for a single test to finish, which is roughly a week for us. the business is currently relatively stable and doesn't change a lot, and I don't have big ideas that would need measurement so frequently, but if I did, I'd probably look for ways of running things in parallel. So I don't think it's a function of the dev team as much as what you reasonably want to measure. Having said that, I can imagine a larger dev team implying multiple product people, and those might want to have their own experiments/ideas measured in parallel, so there is something there definitely about the size of the dev team as well
2) if we wanted to be very scientific about the whole thing, we'd need multiple tests in parallel, particularly for novelty/primacy holdouts and to reconfirm that certain things actually hold across time. e.g. if I decided that variant A is better than variant B but our traffic changes (which it kind of it did over the last 3-4 months with AI overviews killing clicks from Google), maybe the reduced set of people we get behaves differently than the larger set we measured against a year ago. so I'd probably want to re-run critical experiments and just confirm them.