So one thing we did was for (a/b style) experiments to continue to collect data even after taking a decision to enable or disable[1]. No new sessions/devices wo

So one thing we did was for (a/b style) experiments to continue to collect data even after taking a decision to enable or disable[1]. No new sessions/devices would be added to the experiment, but those already exposed would still be reflected. So if everyone who decided to book because of an experiment ends up cancelling or consistently providing more negative feedback over the months that follow, our automation could still flag that. Whatever we may say about the company and its practices, the engineering team working on the experimentation tooling had their heart in the right place. They worked extremely hard and with tremendous care on trying to make sure the data presented would cause the best possible decisions. [1] Caveat: if an experiment was disabled fully (doesn't work or buggy) or enabled fully, obviously the experiment measurement is now no longer clean because folks in one variant will now see the other. But for trailing metrics like anything that affected past booking decisions, this type of analysis may not be perfect but still carries some meaning as a health check. Edit: formatting fix