6 ms·

A stevehanov.ca link? Wow, HN is getting classy again. Please more articles with code, equations and / well visualizations, and less upvoting of badly thought o

by wisty 14y ago

A stevehanov.ca link? Wow, HN is getting classy again. Please more articles with code, equations and / well visualizations, and less upvoting of badly thought out infograpics (i.e. pretty numbers which would lose nothing by just being presented in a table) and far less self-help pseudo business articles please. +1 on an article does not mean "I agree". It means "I learnt something".

Swizec 14y ago
I upvoted you because I agree. But I usually upvote stories based on how interesting they are and whether I feel like they were worth attention. Comments are a mix bag of agreeing, interesting and just well thought out arguments that make me think. This is completely meta and kind of offtopic, so it should probably be downvoted, but I felt like saying it anyway.
r00fus 14y ago
I also +1 a comment where I learn something. As in meat-space voting, you should always reward good behavior.. be part of the fitness function.
gbog 14y ago
Sadly upvote on stories really means "more of that".
timr 14y ago
Bandit optimization has been discussed previously on HN: http://news.ycombinator.com/item?id=2831455 http://news.ycombinator.com/item?id=2831455 The problem with this article is that it's a) not very detailed, and b) the conclusion is linkbait. Bandit optimization is a useful tool, but it has drawbacks, and it's not always better than A/B testing. In particular, bandit approaches take longer to converge (on average), and don't give you reliable ways to know when to stop testing (when all you know is that you're using an approach that's optimal in the limit of a large N, your only guarantee is that things get better as N gets large). These techniques also make assumptions that aren't valid for a lot of web experiments: identical "bandit" distributions that are constant over time. Throw a few choices that are optimal at different times of day/week/month/year at a bandit optimizer, and it'll just happily fluctuate between them. Also, there's a lot of variation in performance depending on the parameters of your test -- some of which are completely unknowable. So if you want to really learn about this method, you need to read more than a blog post where the author has concluded that bandit optimization is the new pink. For example, here's a pretty readable paper that does an empirical analysis of the various popular bandit algorithms in different paramterizations: https://docs.google.com/viewer?a=v&q=cache:KgmC8CnPhxwJ:www.cs.mcgill.ca/~vkules/bandits.pdf+&hl=en&gl=us&pid=bl&srcid=ADGEESjCXEToqqEjlrUIYeKyWtVpgBe4edd4wNeBoFqsTouBIiwPSoAIqP4iZCWXZeygjkzhchKEm6lZYwhCY3RMtg2JrD4Zr9Cge6IO9QH9QZ1Lx25Ee5H8OEHOAb2I0g5Z2NpWwe7P&sig=AHIEtbSqosRQf7c82ZCcUOXYgSo0QJ2IJw https://docs.google.com/viewer?a=v&q=cache:KgmC8CnPhxwJ:... This is just one article, but there's tons of literature on this problem. (In fact, if you use the 'softmax' criterion mentioned in that article, you're doing something very similar to simulated annealing, which is a rather elderly optimization technique.)
- pilom 14y ago
  "and don't give you reliable ways to know when to stop testing" Stopping a test when you reach a "statistically significant" result is the wrong way to do A/B testing. In both multi-armed bandit and A/B testing you need to set ahead of time the number of users you are going to run your test against and stop the test at that point regardless of if your result is significant or not.
  - timr 14y ago
    "Stopping a test when you reach a "statistically significant" result is the wrong way to do A/B testing." Nobody said that it was. But when you do regular split testing, you can use power analysis to estimate the length of time you need to run an experiment to get a significant result at a certain precision: http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power You can't do this (at least, not easily) when you're using bandit models, because none of the assumptions are valid.
  - btilly 14y ago
    In theory, yes. In practice, no. See http://elem.com/~btilly/effective-ab-testing/index.html#aslide59 http://elem.com/~btilly/effective-ab-testing/index.html#asli... for part of a presentation that I did where I actually set up some reasonable fake tests, and ran simulations. What I found is that if there is a significant difference, the probability of coming to the wrong conclusion was (as you would expect) higher, but not that high before the underlying difference made mistakes incredibly unlikely. Conversely if there is only a small real difference, the amount of data needed before you have a significant chance of having accidentally come to a erroneous conclusion is very, very long. So avoid accepting any result where you don't have at least a few hundred successes and set your thresholds reasonably high. You will make fewer mistakes than you probably fear, and the ones that you make will almost always be very minor. (Oops, I had a 3% chance of accepting the 1% worse solution as probably better.) Of course if you're aiming to publish academic research, your standards need to be higher. But if you're more interested in getting useful results than publishable ones, you can relax your standards. A lot.
evropamx 14y ago
And I really did, so +1 from me.

by wisty 14y ago

Swizec 14y ago

I upvoted you because I agree. But I usually upvote stories based on how interesting they are and whether I feel like they were worth attention. Comments are a mix bag of agreeing, interesting and just well thought out arguments that make me think. This is completely meta and kind of offtopic, so it should probably be downvoted, but I felt like saying it anyway.

r00fus 14y ago

I also +1 a comment where I learn something. As in meat-space voting, you should always reward good behavior.. be part of the fitness function.

gbog 14y ago

Sadly upvote on stories really means "more of that".

timr 14y ago

Bandit optimization has been discussed previously on HN: http://news.ycombinator.com/item?id=2831455 http://news.ycombinator.com/item?id=2831455 The problem with this article is that it's a) not very detailed, and b) the conclusion is linkbait. Bandit optimization is a useful tool, but it has drawbacks, and it's not always better than A/B testing. In particular, bandit approaches take longer to converge (on average), and don't give you reliable ways to know when to stop testing (when all you know is that you're using an approach that's optimal in the limit of a large N, your only guarantee is that things get better as N gets large). These techniques also make assumptions that aren't valid for a lot of web experiments: identical "bandit" distributions that are constant over time. Throw a few choices that are optimal at different times of day/week/month/year at a bandit optimizer, and it'll just happily fluctuate between them. Also, there's a lot of variation in performance depending on the parameters of your test -- some of which are completely unknowable. So if you want to really learn about this method, you need to read more than a blog post where the author has concluded that bandit optimization is the new pink. For example, here's a pretty readable paper that does an empirical analysis of the various popular bandit algorithms in different paramterizations: https://docs.google.com/viewer?a=v&q=cache:KgmC8CnPhxwJ:www.cs.mcgill.ca/~vkules/bandits.pdf+&hl=en&gl=us&pid=bl&srcid=ADGEESjCXEToqqEjlrUIYeKyWtVpgBe4edd4wNeBoFqsTouBIiwPSoAIqP4iZCWXZeygjkzhchKEm6lZYwhCY3RMtg2JrD4Zr9Cge6IO9QH9QZ1Lx25Ee5H8OEHOAb2I0g5Z2NpWwe7P&sig=AHIEtbSqosRQf7c82ZCcUOXYgSo0QJ2IJw https://docs.google.com/viewer?a=v&q=cache:KgmC8CnPhxwJ:... This is just one article, but there's tons of literature on this problem. (In fact, if you use the 'softmax' criterion mentioned in that article, you're doing something very similar to simulated annealing, which is a rather elderly optimization technique.)

pilom 14y ago
"and don't give you reliable ways to know when to stop testing" Stopping a test when you reach a "statistically significant" result is the wrong way to do A/B testing. In both multi-armed bandit and A/B testing you need to set ahead of time the number of users you are going to run your test against and stop the test at that point regardless of if your result is significant or not.
- timr 14y ago
  "Stopping a test when you reach a "statistically significant" result is the wrong way to do A/B testing." Nobody said that it was. But when you do regular split testing, you can use power analysis to estimate the length of time you need to run an experiment to get a significant result at a certain precision: http://en.wikipedia.org/wiki/Statistical_power http://en.wikipedia.org/wiki/Statistical_power You can't do this (at least, not easily) when you're using bandit models, because none of the assumptions are valid.
- btilly 14y ago
  In theory, yes. In practice, no. See http://elem.com/~btilly/effective-ab-testing/index.html#aslide59 http://elem.com/~btilly/effective-ab-testing/index.html#asli... for part of a presentation that I did where I actually set up some reasonable fake tests, and ran simulations. What I found is that if there is a significant difference, the probability of coming to the wrong conclusion was (as you would expect) higher, but not that high before the underlying difference made mistakes incredibly unlikely. Conversely if there is only a small real difference, the amount of data needed before you have a significant chance of having accidentally come to a erroneous conclusion is very, very long. So avoid accepting any result where you don't have at least a few hundred successes and set your thresholds reasonably high. You will make fewer mistakes than you probably fear, and the ones that you make will almost always be very minor. (Oops, I had a 3% chance of accepting the 1% worse solution as probably better.) Of course if you're aiming to publish academic research, your standards need to be higher. But if you're more interested in getting useful results than publishable ones, you can relax your standards. A lot.

evropamx 14y ago

And I really did, so +1 from me.