You get two groups. One uses the old interface and the other uses the radically new shiny thing that might perform better. Surprise! The new interface performs

You get two groups. One uses the old interface and the other uses the radically new shiny thing that might perform better. Surprise! The new interface performs better in selected metrics. Old interface hits the recycle bin and the new one become the norm. This us what seems to be the standard decision process. I would propose the introduction of a third and maybe fourth group where returning users would also measure those selected metrics.