Others have done a great job of highlighting some of the challenges, but I wanted to share my own thoughts. Conducting an on/off test can be a viable approach

Others have done a great job of highlighting some of the challenges, but I wanted to share my own thoughts. Conducting an on/off test can be a viable approach in some cases assuming you can properly control for the numerous (understatement) variables that often make your data incredible dirty for such a test. That said, often the nature and number of the variables is such that you would not get meaningful results from an on/off test. This is quite often the case with large brands. As such, other forms of testing are often used individually or in conjunction with each other to get a better read on things. In terms of the end-to-end data, from an online advertising standpoint you often are missing trackable data points around other brand touch points the person might have encountered before purchase, or often, even the end purchase itself. And that's just the start of the data headaches.