I think there's clear danger of having a local maximum problem here. This is because the role of hypothesis in the process seems either misunderstood or misplac

I think there's clear danger of having a local maximum problem here. This is because the role of hypothesis in the process seems either misunderstood or misplaced. The Tim Robinson example in the article recounting the removing of links on the checkout flow illustrates this. He observed something interesting in the data (people clicking on links), but he then skipped straight to a test which removed links and found conversion went up. It's implied that people were being distracted from the main task by the links. But it might equally be that they wanted to click the links to find some missing information about the product. Or even some other reason. Without any qualitative analysis to establish a reason why people were clicking on them, you don't know if he might have raised conversion even higher had he kept the links and addressed a need people had. This is particularly important in a checkout flow where the numbers of people are comparatively low and behavioural elasticity is also comparatively low. So I'm reading between the lines here that because qualitative analysis is hard, people don't do it. Maybe that's fair enough, but that doesn't prevent you from kicking the tyres on hypotheses. Quant analysis and A/B testing will only tell you what happened, not why did.