5 ms·
Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels. My app, 7 Second Meditatio
by orasis 11y ago
Here's what everyone is missing. Don't use bandits to A/B test UI elements, use them to optimize your content / mobile game levels.
My app, 7 Second Meditation, is solid 5 stars, 100+ reviews because I use bandits to optimize my content.
By having the system automatically separate the wheat from the chaff, I am free to just spew out content regardless of its quality. This allows me to let go of perfectionism and just create.
There is an interesting study featured in "Thinking Fast and Slow" where they had two groups in a pottery class. The first group would have the entirety of their grade based on the creativity of a single piece they submit. The second group was graded on only the total number of pounds of clay they threw.
The second group crushed the first group in terms of creativity.
- whatever_dude 11y agoLove that example, even if frequently retold. http://blog.codinghorror.com/quantity-always-trumps-quality/ http://blog.codinghorror.com/quantity-always-trumps-quality/
- deleted 11y ago[deleted]
- mrspeaker 11y agoI don't understand what you mean by "use them to optimize your content" - how are you doing that with your app? Are you serving different messages to different groups of people? How are you grouping/testing/rating them?
- andreasklinger 11y agoi dont know the app but i assume he has a core action (-> which he iterates/churns out) and evaluates if people act (-> used as rating) eg (guesses here) new notification types "think for 9 seconds about a loved one" "what was your favorite vacation, go there" if people act -> reward and reuse if not -> reuse less (only by random chance) alternative to notification could be "completed medidation screen"
- orasis 11y agoMy bandit system generates an ordered list of the content for each individual user. I then will track if the user came back tomorrow or churned. Yes, they may churn due to many other factors, but the signal of the content itself is strong enough. In the past I have used the share rate to optimize, but I've realized that retention is more important.
- searine 11y ago>I am free to just spew out content regardless of its quality. Oh, that's a great goal to have...
- orasis 11y agoBy focusing on volume, I get quality as a side effect. I'm very proud of the the app I've created and the feedback I get from my users: "This app gives a text reminder to do what everyone wants to do: relax, love one's self and others, and bring peace and light into the world. I smile when the alert message appears and feel grateful to the makers of this app for creating a pleasant, quick way to meditate at the most stressful point of my day. I have recommended it to many people" https://itunes.apple.com/us/app/7-second-meditation-daily/id667197203?mt=8 https://itunes.apple.com/us/app/7-second-meditation-daily/id...
- searine 11y agoVolume and efficiency are great, but nobody likes a shit cannon.
- squeaky-clean 11y agoI actually think that's a great idea, because then the users and algorithm decide what's good for you, and it will hide the low quality content automatically. You have to do that sort of "curation" anyways anytime you make something. You have to continually decide whether it's worth it to keep working on something, and then decide if it's good enough to release. People tend to be pretty bad judges of this (especially for more creative tasks. There are many examples of someone's most famous song/book/painting not being their own favorite, or even being their least favorite!). So why not relieve some of that stress, and let the users pick what they really like, instead of you guessing for them?
- searine 11y ago"I promise, there is a diamond somewhere in this bathtub of shit". Efficiency is great, but there still needs to be some quality standards.
- wlesieutre 11y ago> The first group would have the entirety of their grade based on the creativity of a single piece they submit. The second group was graded on only the total number of pounds of clay they threw. I feel like that works partly because an important part of practice the feedback loop between continually practicing and having a sense of whether you did well or not. Your strategy of not evaluating your own work sounds a bit like mushing clay into shapes with a blindfold on and then tossing it in kiln before you even check whether or not it's shaped like a pot. The users can sort through them later! If the end goal is just ending up with a volume of work that's been culled down to the better ones, I guess you still get that. But it's inherently different from the Thinking Fast and Slow example where they're in a class and the goal is to learn and get better, rather than see who's made the nicest pot by the end of the semester.
- orasis 11y agoI get the same effect using bandits. I practice my craft by spewing out volume rather than focusing on quality. The penalty I pay for the bad content is negligible because the bayesian bandits cull them very quickly. I am learning and getting better, but it is because I am not paralyzed by perfectionism.
- kens 11y agoI tried to find the original source of the quality-vs-quantity pottery class story a while back. I think it originates in the book "Art and Fear" but in that book it reads like a parable rather than a factual event. I'm highly suspicious of whether this event actually happened. Anyone have solid evidence?
- orasis 11y agoIt was featured in "Thinking Fast and Slow", and those authors seem quite academically rigorous.
- SonicSoul 11y agothat's an appeal to authority, sounds like @kens actually tried to verify this story. I'd be curious to hear more examples as well. I believe I heard something similar on You're Not So Smart podcast but that might have referenced the same example
- soperj 11y agoI do think the example is bullshit, simply because you could throw ONE very thick pot and have it weight more than 50 pounds. Of course it's a horrible pot and you wouldn't get any better since you only made one piece, but it would get you an A since it would use more clay. Total # of pieces thrown would make more sense than pounds of clay used.
- diab0lic 11y agoForgive me if I've missed something but the original comment says the group that threw the most clay scored higher on creativity, not scored higher on pounds of clay thrown.
- soperj 11y agoGuess you missed something. "The second group was graded on only the total number of pounds of clay they threw."
- tswartz 11y agoGood point! What do you use to do in-app A/B testing?
- hinkley 11y agoYou might want to be careful with your conclusions. We don't know from this study whether the second group was more or less likely to get trapped in the Expert Beginner phase of development. You definitely don't get anywhere without practice, but you are likely to get nowhere fast without theory.
- doug1001 11y agoinsightful analogy. so then "Expert Beginner phase" = local minima, which might be reached quickly compared to a more systematic search (practice informed/infused with theory) and once in that local state, the "expert beginner" might be unable to escape because of over-reliance on small, poorly guided, step size; the real prize anyway is the global minima.
- fapjacks 11y agoThere were two epiphanies hidden in this comment. HN is good for one or two every once in a while, but for me personally, this struck gold. Thanks!!
- spyder 11y agosolid 5 stars, 100+ reviews because I use bandits to optimize my content. It's great app, but were the ratings lower in the beginning before the optimization or how do you know the optimization helped the ratings? I'm asking because it seems the app could have good ratings regardless of the content optimization because it's a "feel good" app. Are there counterexamples of of other meditation apps were the UI is good but it has bad reviews because of their low quality content?