4 ms·
The main issue I battle with with A/B testing is backing out far enough to know whether the change is related to the edits made or simply a seasonal/daily/hourl
by josefresco 16y ago
The main issue I battle with with A/B testing is backing out far enough to know whether the change is related to the edits made or simply a seasonal/daily/hourly/freak change.
I manage CPC campaigns for clients, and often times I make A/B changes in the keywords/bids/ads and wait 30-60-90 days before measuring and making changes based on the findings. And even with 90 days and tens of thousands of visits, I find that sometimes my scope is too limited and I get nailed by seasonal/yearly changes that the numbers didn't reveal.
A/B testing in 5K chunks works if everything else in the universe remains constant, but here in the real world that's never the case.
The worst is when you make a change, measure results and implement said change, and then something related to the economy (or even Weather) horks your CAC and suddenly the client isn't so thrilled with your amazing A/B improvements.