4 ms·

It really feels like some companies like Facebook are flying blind by using A/B testing everywhere and ignoring the long term effects of the changes they do.

edejong 8y ago
Exactly this. Death by a thousand paper cuts, is another.
Kiro 8y ago
Or Hacker News is wrong.
- TomAnthony 8y ago
  You are maybe being downvoted as you stated it so bluntly, but I'm not sure your point is without merit. The HN audience do not necessarily represent the mean/mode user, and Facebook are in a numbers game really. I agree with most of the sentiment above - I wish I could filter posts that are just attachments, 3rd party junk, and tune the algorithm to show me posts from a core set of friends, but I also recognise I don't use FB like a most people, I imagine.
arethuza 8y ago
Isn't A/B testing pretty much going to behave like a steepest ascent hill climbing? At each micro decision point you take what looks like the 'best' option but that means you can get stuck in local maxima?
- qubex 8y ago
  Correct.
- Yhippa 8y ago
  This is interesting but I don't think I fully understand. Do you mind dumbing it down for me?
  - girmad 8y ago
    Global vs local maximum. https://en.wikipedia.org/wiki/Maxima_and_minima https://en.wikipedia.org/wiki/Maxima_and_minima Related: what's the best for one part of the system, may not be the best for the entire system.
  - JorgeGT 8y ago
    Typical example: you arrive at a crossroads A where both ways work for you, so you choose the one with less traffic. Then at crossroad B you do the same, and then at C, and finally you arrive at destination D. However, it turns out the heavy traffic at the other A branch was just for a few miles and then it was actually empty after that --- you took optimum local decisions at each step but since you weren't able to look at the big picture, you didn't actually choose the globally optimal route. As others have pointed, this is related to the mathematical concepts of local and global maxima: sometimes your optimization algorithm happily stops when it finds a local maximum, ignoring the much better global maximum because it didn't actually traversed the whole search domain.
- jmilloy 8y ago
  I think the bigger problem is that A/B only measures certain things and following it blindly can have you descending in important ways that are hard to measure.
- Matheus28 8y ago
  Yes, but it gets more complicated because there's a time factor too. If you're A/B testing each change for 2 weeks, but the negative impacts of it only happen after a few months (like what the parent post mentioned [1]), then while you're in a local maxima right now, it'll slowly sink, along with all your neighborhood of choices. You can think of it as a function that returns the current value and another function that you have to use for the next time step. Sorta like f(a, b, c, ...) = (y, \a_2 b_2 c_2 ... -> ...). Steepest ascent hill climbing doesn't work well for finding good long term local maxima, since you don't know how long it takes until it stabilizes (or if it ever does). The best you can do is guess it'll stabilize in X amount of time, but if X is too small, you might end up stuck in a really bad local maxima. [1] https://news.ycombinator.com/item?id=19325816 https://news.ycombinator.com/item?id=19325816
darkpuma 8y ago
I wonder if they even do their A/B testing right. In my experience (none at facebook) accidental p-hacking is rampant in the tech industry, with trials being cut short or prolonged by the tester who's staring at a graph of the results in real time.
- michaelt 8y ago
  If you have a change that improves your metric initially but damages it in the longer term, and your A/B test only detects the short initial effect, you can execute your A/B test perfectly and still get the wrong result.
  - teleclimber 8y ago
    Interestingly that's exactly how algorithms make things go viral. They pick up on things that get a quick reaction with complete disregard for what happens in the long term. Jaron Lanier explains this in his talks [0]. So features on social media are decided based on short term gains and posts on social media are promoted like that too. It's like an entire industry forgot their parents warnings about thinking about the future. [0] https://www.youtube.com/watch?v=kc_Jq42Og7Q https://www.youtube.com/watch?v=kc_Jq42Og7Q

edejong 8y ago

Exactly this. Death by a thousand paper cuts, is another.

Kiro 8y ago

Or Hacker News is wrong.

TomAnthony 8y ago
You are maybe being downvoted as you stated it so bluntly, but I'm not sure your point is without merit. The HN audience do not necessarily represent the mean/mode user, and Facebook are in a numbers game really. I agree with most of the sentiment above - I wish I could filter posts that are just attachments, 3rd party junk, and tune the algorithm to show me posts from a core set of friends, but I also recognise I don't use FB like a most people, I imagine.

arethuza 8y ago

Isn't A/B testing pretty much going to behave like a steepest ascent hill climbing? At each micro decision point you take what looks like the 'best' option but that means you can get stuck in local maxima?

qubex 8y ago
Correct.
Yhippa 8y ago
This is interesting but I don't think I fully understand. Do you mind dumbing it down for me?
- girmad 8y ago
  Global vs local maximum. https://en.wikipedia.org/wiki/Maxima_and_minima https://en.wikipedia.org/wiki/Maxima_and_minima Related: what's the best for one part of the system, may not be the best for the entire system.
- JorgeGT 8y ago
  Typical example: you arrive at a crossroads A where both ways work for you, so you choose the one with less traffic. Then at crossroad B you do the same, and then at C, and finally you arrive at destination D. However, it turns out the heavy traffic at the other A branch was just for a few miles and then it was actually empty after that --- you took optimum local decisions at each step but since you weren't able to look at the big picture, you didn't actually choose the globally optimal route. As others have pointed, this is related to the mathematical concepts of local and global maxima: sometimes your optimization algorithm happily stops when it finds a local maximum, ignoring the much better global maximum because it didn't actually traversed the whole search domain.
jmilloy 8y ago
I think the bigger problem is that A/B only measures certain things and following it blindly can have you descending in important ways that are hard to measure.
Matheus28 8y ago
Yes, but it gets more complicated because there's a time factor too. If you're A/B testing each change for 2 weeks, but the negative impacts of it only happen after a few months (like what the parent post mentioned [1]), then while you're in a local maxima right now, it'll slowly sink, along with all your neighborhood of choices. You can think of it as a function that returns the current value and another function that you have to use for the next time step. Sorta like f(a, b, c, ...) = (y, \a_2 b_2 c_2 ... -> ...). Steepest ascent hill climbing doesn't work well for finding good long term local maxima, since you don't know how long it takes until it stabilizes (or if it ever does). The best you can do is guess it'll stabilize in X amount of time, but if X is too small, you might end up stuck in a really bad local maxima. [1] https://news.ycombinator.com/item?id=19325816 https://news.ycombinator.com/item?id=19325816

darkpuma 8y ago

I wonder if they even do their A/B testing right. In my experience (none at facebook) accidental p-hacking is rampant in the tech industry, with trials being cut short or prolonged by the tester who's staring at a graph of the results in real time.

michaelt 8y ago
If you have a change that improves your metric initially but damages it in the longer term, and your A/B test only detects the short initial effect, you can execute your A/B test perfectly and still get the wrong result.
- teleclimber 8y ago
  Interestingly that's exactly how algorithms make things go viral. They pick up on things that get a quick reaction with complete disregard for what happens in the long term. Jaron Lanier explains this in his talks [0]. So features on social media are decided based on short term gains and posts on social media are promoted like that too. It's like an entire industry forgot their parents warnings about thinking about the future. [0] https://www.youtube.com/watch?v=kc_Jq42Og7Q https://www.youtube.com/watch?v=kc_Jq42Og7Q