3 ms·

I'm surprised no one on the team was familiar with multivariate analysis. That's the wrong way to say it. Let me rephrase more carefully... I'm surprised no on

I'm surprised no one on the team was familiar with multivariate analysis. That's the wrong way to say it. Let me rephrase more carefully... I'm surprised no one used more sophisticated analysis techniques to allow an experiment that changes many explanatory factors simultaneously and estimates the effects of not only each factor independently but also their interactions. It's relatively straightforward statistics and would have been taught to clever undergrads or first-year grad students who specialized in math, economics, machine learning, industrial engineering, etc.

nostrademons 11y ago
There were stats Ph.Ds on the team that were certainly familiar with it, and I knew at least one former Amazonian manager that held that we were too afraid of having multiple experiments running at once and too afraid of confounding factors. (Amazon's A/B testing is much more traditional split-testing, where they show one variant to half the population and one to another half and then pick the best one.) The biggest problem is that many of the folks involved had encountered very surprising and counter-intuitive interactions between features, and because of that, they didn't trust the textbook models. For example, at one point we had a very convenient embedded-Python experimentation framework that let us rapidly try things out. Our main data-scientist on the project decreed that we were not to use it, because a number of the externalities that framework introduced - higher latency, and potential error conditions - could confound the metrics in ways that were not well-understood at the time. Data is useless unless you can trust that it actually represents reality. Stats textbooks all make certain assumptions about reality when they present the models, and if reality doesn't conform to the assumptions, the models won't conform to reality. Certain factors were well-studied enough that we could build useful models from them - that was the point of all the experimentation around color, after all. (I wonder if those conclusions still hold, anyway: data has an expiration date, and I recall personally running an experiment that directly contradicted an experiment Marissa Mayer had run 6-8 years previously.) But that requires a lot of hypothesis testing and verification: it's never a matter of "well, the model says so, therefore it must be", but rather "well, the model says so, let's run an experiment to see if the model's predictions line up with reality, and if they do, we have a pretty strong - but not ironclad - indication that we can use it for future calculations." tl;dr: Largest and best-funded social science program I've ever seen.
- hueving 11y ago
  >Largest and best-funded social science program I've ever seen. You should check out Facebook. There emotional manipulation studies are horrifying, but they certainly have access to many more person hours of time and collected data than Google does on end users.
- paganel 11y ago
  > tl;dr: Largest and best-funded social science program I've ever seen. I don't want to sound like a hater and I'm very well aware of the fact that some design choices might have affected the company's revenues, but all this discussion involving very smart people discussing what, from a very "outsider-ish" perspective, seem like trivial issues (the color of the letter G, for example) reminds me of Herman Hesse's "Glass Bead Game". (https://en.wikipedia.org/wiki/The_Glass_Bead_Game#Plot https://en.wikipedia.org/wiki/The_Glass_Bead_Game#Plot)

nostrademons 11y ago

There were stats Ph.Ds on the team that were certainly familiar with it, and I knew at least one former Amazonian manager that held that we were too afraid of having multiple experiments running at once and too afraid of confounding factors. (Amazon's A/B testing is much more traditional split-testing, where they show one variant to half the population and one to another half and then pick the best one.) The biggest problem is that many of the folks involved had encountered very surprising and counter-intuitive interactions between features, and because of that, they didn't trust the textbook models. For example, at one point we had a very convenient embedded-Python experimentation framework that let us rapidly try things out. Our main data-scientist on the project decreed that we were not to use it, because a number of the externalities that framework introduced - higher latency, and potential error conditions - could confound the metrics in ways that were not well-understood at the time. Data is useless unless you can trust that it actually represents reality. Stats textbooks all make certain assumptions about reality when they present the models, and if reality doesn't conform to the assumptions, the models won't conform to reality. Certain factors were well-studied enough that we could build useful models from them - that was the point of all the experimentation around color, after all. (I wonder if those conclusions still hold, anyway: data has an expiration date, and I recall personally running an experiment that directly contradicted an experiment Marissa Mayer had run 6-8 years previously.) But that requires a lot of hypothesis testing and verification: it's never a matter of "well, the model says so, therefore it must be", but rather "well, the model says so, let's run an experiment to see if the model's predictions line up with reality, and if they do, we have a pretty strong - but not ironclad - indication that we can use it for future calculations." tl;dr: Largest and best-funded social science program I've ever seen.

hueving 11y ago
>Largest and best-funded social science program I've ever seen. You should check out Facebook. There emotional manipulation studies are horrifying, but they certainly have access to many more person hours of time and collected data than Google does on end users.
paganel 11y ago
> tl;dr: Largest and best-funded social science program I've ever seen. I don't want to sound like a hater and I'm very well aware of the fact that some design choices might have affected the company's revenues, but all this discussion involving very smart people discussing what, from a very "outsider-ish" perspective, seem like trivial issues (the color of the letter G, for example) reminds me of Herman Hesse's "Glass Bead Game". (https://en.wikipedia.org/wiki/The_Glass_Bead_Game#Plot https://en.wikipedia.org/wiki/The_Glass_Bead_Game#Plot)