Every campaign has TWO holdouts. First, the universal holdout who gets nothing from any campaign, and a second additional holdout who doesn't get a specific ca

Every campaign has TWO holdouts. First, the universal holdout who gets nothing from any campaign, and a second additional holdout who doesn't get a specific campaign but gets other campaigns. We have an entire group of people designing the holdout matrix to make sure they are sound and don't cross over. That's the elementry school version of things. The graduate level stuff we run is using decioning engines with adaptive models down to EACH email, switching up not just offers but also ad copy, images, time of delivery etc. that drive VERY complex holdout matrix's which can limit the number of things we can test at once. One of the executive decisions is when or if the drop customers out of holdout populations. Usually keep the holdout going for X period of time as a standard practice, but if a campaign is showing very strong response we might drop half the holdout if we can still make a read on campaign. We have a formal notice of decision that is published when this happens. Holdouts have alot of opportunity cost, but finance doesn't see those numbers. Some of the digital exhaust from our primrary product can be used to solve marketing attribution issues for major brands in an area that has until now lacked attribution. Our product beta is just hitting the market.