At least thinking about it is important NOW, regardless of what stage your startup is. It can be extremely difficult to try to slot a 3rd party A/B testing sol

At least thinking about it is important NOW, regardless of what stage your startup is. It can be extremely difficult to try to slot a 3rd party A/B testing solution into your product (or really hard to roll your own) if your infra doesn't support it from the start. Also, hire a data scientist! (Disclaimers: I am not a data scientist, I just think everyone needs one! I have worked on an experimentation system at $BIG_COMPANY.) I'd suggest thinking about the following BEFORE YOU RUN A SINGLE A/B TEST: 1) Key Metrics: Define these. They are the general, "I don't care what your experiment is about, these numbers are important." Every experiment you run should automatically track these metrics. You should also give the ability to define custom metrics, since an experiment that changes some random button color probably wants to look at how many people clicked the button, which is almost definitely NOT a key metric. 2) Logging Infrastructure: Make sure that you have a easy-to-use, reliable data pipeline set up for logging and processing events. Bad logging == bad experiment results. Also consider streaming vs batch processing for updating experiment results. 3) Population Management: How do your experiments segment users? Are variants calculated in realtime? Batched with some SLA for lag? Are they sticky? 4) Mutual Exclusion: People running experiments often want "their" users excluded from other experiments. 5) Guardrails: Do your experiments automatically shut off if there is a catastrophic decline in one or more key metrics? What safety measures do you have around determining if an experiment is safe/valid? How do you handle cleaning up data when there's a problem? What sorts of actions invalidate an experiement's existing results? Does your entire site break if your A/B Testing service is down for whatever reason? 6) Cleanup/Ownership: Experiments don't run forever (at least they shouldn't!). Cleaning up old features, populations, etc can a pain, especially when the people that wrote the stuff originally no longer work at the company. Make cleanup mandatory and as easy as possible. There's a lot more, but I'm tired now. A/B testing is complex. There are lots of resources out there, though. Look for white papers on the subject, they're surprisingly approachable. Example from Microsoft: https://exp-platform.com/Documents/2017-08%20KDDMetricInterpretationPitfalls.pdf https://exp-platform.com/Documents/2017-08%20KDDMetricInterp...