47 ms·

They don't seem to have fallen off, are still the most widely used platforms in the world with billions of users in total. It is popular to think that A/B testi

by vasco 5y ago

They don't seem to have fallen off, are still the most widely used platforms in the world with billions of users in total. It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The only thing I've seen work is something like Pixar's brain trust - you do testing but you also rely on the good judgement of a small group of people with a strong sense of vision. Judging good judgement is pretty hard though, and you won't know if they're just full of it or winging it unless you try and trust. And so everyone does A/B testing which to be honest sounds much better than your average PM making decisions that are pulled out of their ass.

deleted 5y ago
[deleted]
HWR_14 5y ago
> It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? I mean, if corporate leadership isn't able to better direct the path of a product than the results of A/B tests, what value do they really have?
piva00 5y ago
> It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The alternative involves creativity to create a vision to achieve. A/B testing are pretty good tools to evaluate differing opinions on some vision, as the only deciding tool for what is worth to keep or not it's pretty short-sighted. Is it really that much better? I lived through the rise of A/B testing, professionally I've been responsible for implementing platforms for experimentation in at least 5 different companies and over the years I saw it transition from a tool to a crutch. Everything now is "data-based" which means incessant A/B testing, it still depends on your average PM deciding the parameters of the experiment, leading to the current state of affairs: a bunch of uninspired features that are solely judged on the data produced, no soul, no vision behind most of it. And it feels exactly like that, I can feel how tech products just became that, an incessant stream of features being tested, implemented or tossed. I'd much rather have data-supported intuition than this statistical machine in motion.
fisf 5y ago
The problem is not only that it could be a local Maxima. It might also measure the wrong metric all together. Is the platform widely used and the business successful because content makes users happy and improves their lives? Or is engagement high, because users have become accustomed to whatever a company is feeding them and look for something shallow to fill up their time. If you only measure engagement as metric, without any deeper reflection and judgment, then heroin is a perfect product in terms of A/B testing.

by vasco 5y ago

deleted 5y ago

[deleted]

HWR_14 5y ago

> It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? I mean, if corporate leadership isn't able to better direct the path of a product than the results of A/B tests, what value do they really have?

piva00 5y ago

> It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The alternative involves creativity to create a vision to achieve. A/B testing are pretty good tools to evaluate differing opinions on some vision, as the only deciding tool for what is worth to keep or not it's pretty short-sighted. Is it really that much better? I lived through the rise of A/B testing, professionally I've been responsible for implementing platforms for experimentation in at least 5 different companies and over the years I saw it transition from a tool to a crutch. Everything now is "data-based" which means incessant A/B testing, it still depends on your average PM deciding the parameters of the experiment, leading to the current state of affairs: a bunch of uninspired features that are solely judged on the data produced, no soul, no vision behind most of it. And it feels exactly like that, I can feel how tech products just became that, an incessant stream of features being tested, implemented or tossed. I'd much rather have data-supported intuition than this statistical machine in motion.

fisf 5y ago

The problem is not only that it could be a local Maxima. It might also measure the wrong metric all together. Is the platform widely used and the business successful because content makes users happy and improves their lives? Or is engagement high, because users have become accustomed to whatever a company is feeding them and look for something shallow to fill up their time. If you only measure engagement as metric, without any deeper reflection and judgment, then heroin is a perfect product in terms of A/B testing.