3 ms·

Isn't A/B testing pretty much going to behave like a steepest ascent hill climbing? At each micro decision point you take what looks like the 'best' option but

Isn't A/B testing pretty much going to behave like a steepest ascent hill climbing? At each micro decision point you take what looks like the 'best' option but that means you can get stuck in local maxima?

qubex 8y ago
Correct.
Yhippa 8y ago
This is interesting but I don't think I fully understand. Do you mind dumbing it down for me?
- girmad 8y ago
  Global vs local maximum. https://en.wikipedia.org/wiki/Maxima_and_minima https://en.wikipedia.org/wiki/Maxima_and_minima Related: what's the best for one part of the system, may not be the best for the entire system.
- JorgeGT 8y ago
  Typical example: you arrive at a crossroads A where both ways work for you, so you choose the one with less traffic. Then at crossroad B you do the same, and then at C, and finally you arrive at destination D. However, it turns out the heavy traffic at the other A branch was just for a few miles and then it was actually empty after that --- you took optimum local decisions at each step but since you weren't able to look at the big picture, you didn't actually choose the globally optimal route. As others have pointed, this is related to the mathematical concepts of local and global maxima: sometimes your optimization algorithm happily stops when it finds a local maximum, ignoring the much better global maximum because it didn't actually traversed the whole search domain.
jmilloy 8y ago
I think the bigger problem is that A/B only measures certain things and following it blindly can have you descending in important ways that are hard to measure.
Matheus28 8y ago
Yes, but it gets more complicated because there's a time factor too. If you're A/B testing each change for 2 weeks, but the negative impacts of it only happen after a few months (like what the parent post mentioned [1]), then while you're in a local maxima right now, it'll slowly sink, along with all your neighborhood of choices. You can think of it as a function that returns the current value and another function that you have to use for the next time step. Sorta like f(a, b, c, ...) = (y, \a_2 b_2 c_2 ... -> ...). Steepest ascent hill climbing doesn't work well for finding good long term local maxima, since you don't know how long it takes until it stabilizes (or if it ever does). The best you can do is guess it'll stabilize in X amount of time, but if X is too small, you might end up stuck in a really bad local maxima. [1] https://news.ycombinator.com/item?id=19325816 https://news.ycombinator.com/item?id=19325816

qubex 8y ago

Correct.

Yhippa 8y ago

This is interesting but I don't think I fully understand. Do you mind dumbing it down for me?

girmad 8y ago
Global vs local maximum. https://en.wikipedia.org/wiki/Maxima_and_minima https://en.wikipedia.org/wiki/Maxima_and_minima Related: what's the best for one part of the system, may not be the best for the entire system.
JorgeGT 8y ago
Typical example: you arrive at a crossroads A where both ways work for you, so you choose the one with less traffic. Then at crossroad B you do the same, and then at C, and finally you arrive at destination D. However, it turns out the heavy traffic at the other A branch was just for a few miles and then it was actually empty after that --- you took optimum local decisions at each step but since you weren't able to look at the big picture, you didn't actually choose the globally optimal route. As others have pointed, this is related to the mathematical concepts of local and global maxima: sometimes your optimization algorithm happily stops when it finds a local maximum, ignoring the much better global maximum because it didn't actually traversed the whole search domain.

jmilloy 8y ago

I think the bigger problem is that A/B only measures certain things and following it blindly can have you descending in important ways that are hard to measure.

Matheus28 8y ago

Yes, but it gets more complicated because there's a time factor too. If you're A/B testing each change for 2 weeks, but the negative impacts of it only happen after a few months (like what the parent post mentioned [1]), then while you're in a local maxima right now, it'll slowly sink, along with all your neighborhood of choices. You can think of it as a function that returns the current value and another function that you have to use for the next time step. Sorta like f(a, b, c, ...) = (y, \a_2 b_2 c_2 ... -> ...). Steepest ascent hill climbing doesn't work well for finding good long term local maxima, since you don't know how long it takes until it stabilizes (or if it ever does). The best you can do is guess it'll stabilize in X amount of time, but if X is too small, you might end up stuck in a really bad local maxima. [1] https://news.ycombinator.com/item?id=19325816 https://news.ycombinator.com/item?id=19325816