4 ms·
Being data-driven is how you get Google's UIs where every color is A/B tested and yet nobody likes the results. Nobody is actually qualified enough to use stati
by astrange 4y ago
Being data-driven is how you get Google's UIs where every color is A/B tested and yet nobody likes the results. Nobody is actually qualified enough to use statistics this way, and nobody who has good taste enough to make the decisions when you don't get good data back is going to put up with the A/B test in the first place.
- lbriner 4y agoAn A/B test with two bad options will just lead to the least worst option but OP correctly points out that to design a new way of doing a checkbox etc. with no problem to solve is a dangerous exercise in vanity. In some cases these unecessary changes get put upon users potentially ruining muscle memory, overload people with sensory disorders and not achieving much more than a simple checkbox. I don't necessarily think we should never try for something better but it should be based on data. It should also recognise the physical metaphor that these controls were based on for a reason. Ask a question on a paper form and a checkbox is simple for cognitive load. That's why I get annoyed at these flat designs where buttons don't look like buttons and tabs don't look like tabs so you end up having to explore to use something that would be 100 times easier in a paper form. Using hover for feedback is perverse both because it requires you interact with controls to work out whether you can interact with them and secondly that it doesn't work instinctively (or at all) on touch devices.
- recroad 4y agoThe thing about an A/B test is that you don't have to put up with it because you don't know you're in it. So I don't think there's much friction in executing one as long as you're bucketing the population in a sane way (e.g., don't show two different things to the same person at two different times). I hear you on the misuse of statistics, but unless you're running multivariate experiments, this isn't much of an issue. Yes, if you can't even describe an experiment in a table/matrix, then it's probably too complicated and the results difficult to parse. I think most A/B tests don’t fall into this category and are far simpler. Let me try to offer a practical A/B test for this checkbox. Hypothesis: We believe that making the checkbox animated and reactive will result in people creating more to-do list items right after using the checkbox because they love it so much. Test: To verify that we will deploy the new checkbox to half the users, and half the users get the normal one. Run this for a duration of two weeks. Metric: We will measure the amount of to-do list items created right after someone completes a to-do list item. Criteria: We are right if people with the new checkbox create 1.2x more to-do list items than the ones with the old one. -- The above is not a difficult experiment to create or analyze.