"For us, there was so much low hanging fruit in watching a bunch of people check out our marketing site. I suspect there's a good amount of diminishing returns

"For us, there was so much low hanging fruit in watching a bunch of people check out our marketing site. I suspect there's a good amount of diminishing returns there, but if you've never done it, the value you get out of it is tremendous." I'd strongly suggest that you keep doing actual tests with customers until you actually see diminishing returns. I've regularly seen people think they've fixed all of the big problems - then see another N rounds of "low hanging fruit" appear ;-) The insights and improvements you get from doing usability testing are very different from the ones that you get through optimisation via a/b testing. Each can inform the other. Keep doing both until you find you get no value. I suspect that you'll find that you keep getting significantly useful results if you keep it up.