14 ms·

The what and why of product experimentation at Twitter

13 points by squarecog 11y ago

meatysnapper 11y ago
hahahaha oh wow. yeah, no. EDIT: It is the game of thrones version of product development, except no dwarf or sexy people.
crabasa 11y ago
This is an odd blog post for a company people consider to be utterly paralyzed and incapable of shipping new features.
illicium 11y ago
Sounds like analysis paralysis to me.
fideloper 11y ago
The only thing Twitter is likely considering for new features is "Can we sell ads on this?"
wslh 11y ago
Give me an API and I will move the world... I can't find API in that page.
yabatopia 11y ago
Apparently there's a framework for experimentation at Twitter, but it looks like few experiments actually succeed to turn into new features. What goes wrong there? Or is Twitter really a prime example of analysis paralysis, as others suggest here?
- squarecog 11y ago
  Hi, author of the post here. A few brief points: 1. The post talks about most experiments not moving core metrics in a large way. This does not mean the experiments don't ship, it just means core metrics are very hard to move. So "few experiments actually succeed to turn into new features" is not entirely correct -- quite a few of them do ship because they improve a particular feature, or because the team believes they are the right thing to do as long as they don't do any harm, or even an acceptably small amount of harm. 2. If the majority of the experiments worked, we wouldn't need to experiment, would we? :-). 3. Core metrics being hard to move is not a Twitter-specific thing. If you find them easy to (positively) move, you are either so tiny that any change is a large change, or you are measuring something incorrectly. LinkedIn, Bing, and Amazon experimenters contributed to a paper, linked from the post, which states as "rule of thumb #2" that "changes rarely have a big positive result on key metrics." Anecdotally, the same is true for Google, EBay, and Pinterest. 4. Analysis paralysis, according to wikipedia, is "an anti-pattern, the state of over-analyzing (or over-thinking) a situation so that a decision or action is never taken, in effect paralyzing the outcome". Deciding to not ship something is not analysis paralysis. It's a valid outcome. If everything you try works, you are probably not trying hard enough. Now, if you ran an experiment and kept saying there is not enough data to make a decision, and kept analyzing the data and collecting more inputs -- that would be analysis paralysis.

13 points by squarecog 11y ago

hahahaha oh wow. yeah, no. EDIT: It is the game of thrones version of product development, except no dwarf or sexy people.

crabasa 11y ago

This is an odd blog post for a company people consider to be utterly paralyzed and incapable of shipping new features.

illicium 11y ago

Sounds like analysis paralysis to me.

fideloper 11y ago

The only thing Twitter is likely considering for new features is "Can we sell ads on this?"

wslh 11y ago

Give me an API and I will move the world... I can't find API in that page.

yabatopia 11y ago

Apparently there's a framework for experimentation at Twitter, but it looks like few experiments actually succeed to turn into new features. What goes wrong there? Or is Twitter really a prime example of analysis paralysis, as others suggest here?

squarecog 11y ago
Hi, author of the post here. A few brief points: 1. The post talks about most experiments not moving core metrics in a large way. This does not mean the experiments don't ship, it just means core metrics are very hard to move. So "few experiments actually succeed to turn into new features" is not entirely correct -- quite a few of them do ship because they improve a particular feature, or because the team believes they are the right thing to do as long as they don't do any harm, or even an acceptably small amount of harm. 2. If the majority of the experiments worked, we wouldn't need to experiment, would we? :-). 3. Core metrics being hard to move is not a Twitter-specific thing. If you find them easy to (positively) move, you are either so tiny that any change is a large change, or you are measuring something incorrectly. LinkedIn, Bing, and Amazon experimenters contributed to a paper, linked from the post, which states as "rule of thumb #2" that "changes rarely have a big positive result on key metrics." Anecdotally, the same is true for Google, EBay, and Pinterest. 4. Analysis paralysis, according to wikipedia, is "an anti-pattern, the state of over-analyzing (or over-thinking) a situation so that a decision or action is never taken, in effect paralyzing the outcome". Deciding to not ship something is not analysis paralysis. It's a valid outcome. If everything you try works, you are probably not trying hard enough. Now, if you ran an experiment and kept saying there is not enough data to make a decision, and kept analyzing the data and collecting more inputs -- that would be analysis paralysis.