5 ms·
Ask HN: Is anyone using machine learning for A/B tests?
I am starting to apply machine learning in different ways, and I would like to know if someone is using machine learning for A/B tests.
For example, is is saving data and then showing A or B based on that data and improving it with each new case. I am not just talking about if A or B converted (that would be only success %), but more data like date, time, gender. For example:
I just came with the next idea: Maybe on the night, your visitors are tired for the day, so if you put something like "Register now. Only takes 2 minutes", has more conversions because has a better effect on tired people.
I am sorry if I am just divagating (and sorry for my English)
- patio11 14y agoThere existed at least one project to do this, but you essentially can't justify it unless you are an ad network with hundreds of millions of impressions. The statistical confidence math is otherwise prohibitive. Ask if you need elaboration.
- rfergie 14y agoWhen you say "hundreds of millions of impressions" do you mean that literally or are you meaning "a lot of traffic"? If it is the former then I would like some elaboration; my intuition is surprised that the amount of data required would be that large.
- barry-cotter 14y agoHe is talking about Google. Probably Adwords but maybe other Google properties too.
- rfergie 14y agoI'm sure that Google and other large ad networks do this. But I also see small/medium size ecommerce retailers doing "People like you also bought" type stuff. They don't do it as well as Amazon, but sometimes it seems relevant. So I'm curious about whether hundred of millions of impressions are necessary for this type of optimisation (and that the good recommendations I've seen are just a coincidence - quite possible) or if saying "hundreds of millions" actually just means "quite a lot"
- ig1 14y agoWhat makes you say that ? - assuming a reasonable number of variables I would think you'd only end up needing that order of magnitude if you split every combinations of variables into it's own test group to do A/B testing. Presumably the reason that you'd apply a machine learning algorithm in this case is that you could extrapolate from a much smaller test group.
- unreal37 14y agoThat type of business logic is actually quite straightforward. I was involved in a project many years ago, for which we got a patent I think, that involved detecting if time of day, distance from home and other factors could be attributes of ad serving.
- e-dard 14y agoWell, Myna https://mynaweb.com/ https://mynaweb.com/ uses multi-armed bandit algorithms for A/B testing (or A/B/../N testing if you prefer). An n-armed bandit problem is a type of reinforcement learning problem, and the associated algorithms for solving these problems are considered machine learning.