5 ms·

> Does anyone go back and re-test their biggest wins? Yes, absolutely! We do research first, then come up with simple, well-controlled tests. Once we have a wi

by err4nt 7y ago

> Does anyone go back and re-test their biggest wins? Yes, absolutely! We do research first, then come up with simple, well-controlled tests. Once we have a winner we can either lock it in, but often we continue to research and experiment on the new knowledge we gained. A hefty minority of the tests I implement build on past wins to further flesh out what works and what doesn't with knowledge and the data to back it up.

JohannesH 7y ago
Very interesting. It seems to me that doing incremental work like this might end up in a local minima/maxima. Do you have any advice on how to avoid pitfalls like that? Are you testing radically different ideas along with your incremental improvements?
- orasis 7y ago
  From a workflow perspective, MAB is a bit difficult to find radical improvements from. The radical improvements come from fundamental design changes, of which it would be very expensive to create a bunch of radically different variants. MAB is best used where you can generate a bunch of variants cheaply and hope for a 30% gain.
  - JohannesH 7y ago
    Cool, thanks.
edmundsauto 7y ago
How do you do knowledge transfer when you're documenting what you've learned? I have found even simple changes often don't reproduce on the same site. It feels like a easy trap for an expert practitioner to fall into -- the belief they can predict the outcome of a test based on a different test they ran. Sometimes, yes, but often it's not for the reasons people think. This seems very much in Danny Kahneman land -- we can't trust what our brain is telling us, that this experience will have similar results in this other context. I recently worked at a place that ran > 20 websites, huge traffic numbers, with around 100 simultaneous tests. We would poll each employee about their guess for the test winner. The results were about on par with everyone randomly guessing.

by err4nt 7y ago

JohannesH 7y ago

Very interesting. It seems to me that doing incremental work like this might end up in a local minima/maxima. Do you have any advice on how to avoid pitfalls like that? Are you testing radically different ideas along with your incremental improvements?

orasis 7y ago
From a workflow perspective, MAB is a bit difficult to find radical improvements from. The radical improvements come from fundamental design changes, of which it would be very expensive to create a bunch of radically different variants. MAB is best used where you can generate a bunch of variants cheaply and hope for a 30% gain.
- JohannesH 7y ago
  Cool, thanks.

edmundsauto 7y ago

How do you do knowledge transfer when you're documenting what you've learned? I have found even simple changes often don't reproduce on the same site. It feels like a easy trap for an expert practitioner to fall into -- the belief they can predict the outcome of a test based on a different test they ran. Sometimes, yes, but often it's not for the reasons people think. This seems very much in Danny Kahneman land -- we can't trust what our brain is telling us, that this experience will have similar results in this other context. I recently worked at a place that ran > 20 websites, huge traffic numbers, with around 100 simultaneous tests. We would poll each employee about their guess for the test winner. The results were about on par with everyone randomly guessing.