3 ms·

>Instead try to improve the customer experience, make better products, improve customer service. Without a metric to say what is "better" and a method to measu

by treis 4y ago

>Instead try to improve the customer experience, make better products, improve customer service. Without a metric to say what is "better" and a method to measure it this is empty advice.

josefx 4y ago
But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.
- treis 4y ago
  AB testing is how you isolate a change and measure the impact. It's the only real way to be able to associate cause and effect. Best you can do otherwise is measure something over time while making changes. You can try and correlate changes with outcomes but it's hard to be sure the change is what drove the outcome.
  - oba2311 4y ago
    That sounds pretty accurate. Anyone who's gone through some econometric classes would know that split testing is (almost) the only way to get to (close to) causality - all the rest is black magic called correlation which can also lead you to the conclusion that Nicolas Cage's movies lead to people drowning: https://www.thenationalnews.com/uae/nicolas-cage-movies-linked-to-drownings-and-other-spurious-correlations-1.450759#:~:text=An%20analysis%20of%20data%20by,that%20correlation%20is%20not%20causation https://www.thenationalnews.com/uae/nicolas-cage-movies-link.... https://www.wnycstudios.org/podcasts/otm/articles/spurious-correlations https://www.wnycstudios.org/podcasts/otm/articles/spurious-c...
Kaotique 4y ago
There are many other metrics that do not involve AB testing. You can just survey customer experience before, during and after a purchase for example. I never said to throw out all metrics. With AB testing your are optimising for a specific outcome. Usually higher conversion. As pointed out in the article eventually you'll end up with a bunch of colourful buttons and scary texts that persuade the user to click. A lot of the "only 2 seats/rooms available" are lies to scare the user into a conversion.

by treis 4y ago

>Instead try to improve the customer experience, make better products, improve customer service. Without a metric to say what is "better" and a method to measure it this is empty advice.

josefx 4y ago

But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.

treis 4y ago
AB testing is how you isolate a change and measure the impact. It's the only real way to be able to associate cause and effect. Best you can do otherwise is measure something over time while making changes. You can try and correlate changes with outcomes but it's hard to be sure the change is what drove the outcome.
- oba2311 4y ago
  That sounds pretty accurate. Anyone who's gone through some econometric classes would know that split testing is (almost) the only way to get to (close to) causality - all the rest is black magic called correlation which can also lead you to the conclusion that Nicolas Cage's movies lead to people drowning: https://www.thenationalnews.com/uae/nicolas-cage-movies-linked-to-drownings-and-other-spurious-correlations-1.450759#:~:text=An%20analysis%20of%20data%20by,that%20correlation%20is%20not%20causation https://www.thenationalnews.com/uae/nicolas-cage-movies-link.... https://www.wnycstudios.org/podcasts/otm/articles/spurious-correlations https://www.wnycstudios.org/podcasts/otm/articles/spurious-c...

Kaotique 4y ago

There are many other metrics that do not involve AB testing. You can just survey customer experience before, during and after a purchase for example. I never said to throw out all metrics. With AB testing your are optimising for a specific outcome. Usually higher conversion. As pointed out in the article eventually you'll end up with a bunch of colourful buttons and scary texts that persuade the user to click. A lot of the "only 2 seats/rooms available" are lies to scare the user into a conversion.