4 ms·

This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free. I've seen a nontrivial number of smart engineers

This list mentions A/B testing a few times and it's worth noting that A/B testing is great but it's not free. I've seen a nontrivial number of smart engineers get bogged down in wanting to A/B test everything that they spend more time building and maintaining the experiment framework than actually shipping more product and then realizing the A/B testing was useless because they only had a few hundred data points. Data-driven decisions are definitely valuable but you also have to recognize when you have no data to drive the decisions in the first place. Overall, I agree with a lot of the list but I've seen that trap one too many times when people take the advice too superficially.

nightpool 2y ago
It probably helps that one of PostHog's core products is an A/B testing framework, so it's much easier for them to iterate on it internally for what they need to A/B PostHog. Even when you already have a best in class A/B testing framework though, I agree—A/B testing too much or waiting too long for "more data" to make decisions can slow down momentum badly for features that should be no-brainers.
abxyz 2y ago
A/B testing is also high risk: a good test produces valuable data, a bad test produces harmful data. A company that does no testing is better off than a company that does bad testing. Many people treat product decisions made based on test results as unimpeachable, whereas they treat product decisions made on a hunch with a healthy skepticism that leads to better outcomes.
pkaler 2y ago
Agree! Most orgs should just be shipping features. Before starting an Experiment Program teams should be brainstorming a portfolio of experiments. Just create a spreadsheet where the first column is a one-line hypothesis of the experiment. Eg. "Removing step X from the funnel will increase metric Y while reducing metric Z". And the RICE (Reach-Impact-Confidence-Estimation) score your portfolio. If the team can't come up with a portfolio of 10s to 100s of experiments then the team should just be shipping stuff. And then Experiment Buildout should be standardized. Have standardized XRD (Experiment Requirements Doc). Standardize Eligibility and Enrollment criteria. Which population sees this experiment? When do they see it? How do you test that bucketing is happening correctly? What events do analysts need? When do we do readouts? That's just off the top of my head. Most orgs should just be shipping features.
bluGill 2y ago
Most A/B tests should not be done in a production like way. Grab some post-it notes and and sketch out both A and B: then watch users work with it. For a lot of what you want to know the above will give better information than a 100% polished A/B test in production. When people see a polished product they won't give you the same type of feedback as they will for an obvious quick sketch. The UI industry has gone wrong by making A/B in production too easy, even though the above has been known for many decades. (A/B in production isn't all bad, but it is the last step, and often not worth it)
- marcosdumay 2y ago
  You are pushing for bit of too much of name-misuse. A/B tests are something what the entire point is that you run them on the real world, and gather how real users react. Design-time user testing has been a thing for much longer than A/B tests. They are a different thing. I mean, your point stands. But you can't do A/B tests on anything that is not production, those your are recommending are a different kind of tests.
  - bluGill 2y ago
    I'll accept your definition correction. However I think my point still stands: there are better things than A/B testing to get the information you need.
simonw 2y ago
I think A/B testing is one of the most expensive ways of getting feedback on a product feature. - You have to make good decisions about what you're going to test - You have to build the feature twice - You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them. - You have to test both versions of the feature AND the tracking and test selection mechanisms really well, because bugs in any of those invalidate the test - You have to run it in production for several weeks (or you won't get statistically significant results) - and ensure it doesn't overlap with other tests in a way that could bias the results - You'd better be good at statistics. I've seen plenty of A/B test results presented in ways that did not feel statistically sound to me. ... and after all of that, my experience is that a LOT of the tests you run don't show a statistically significant result one way or the other - so all of that effort really didn't teach you much that was useful. The problem is that talking people out of running an A/B test is really hard! No-one ever got fired for suggesting an A/B test - it feels like the "safe" option. Want to do something much cheaper than that which results in a much higher level of information? Run usability tests. Recruit 3-5 testers and watch them use your new feature over screen sharing and talk through what they're doing. This is an order of magnitude cheaper than A/B testing and will probably teach you a whole lot more.
- light_triad 2y ago
  Some teams think they can A/B test their way to a great product. It can become a socially acceptable mechanism to avoid having opinions and reduce friction. Steve Blank's quote about validating assumptions: "Lean was designed to inform the founders’ vision while they operated frugally at speed. It was not built as a focus group for consensus for those without deep convictions" Is the Lean Startup Dead? (2018) https://medium.com/@sgblank/is-the-lean-startup-dead-71e0517294a0 https://medium.com/@sgblank/is-the-lean-startup-dead-71e0517... Discussed on HN at the time: https://news.ycombinator.com/item?id=17917479 https://news.ycombinator.com/item?id=17917479
  - eddythompson80 2y ago
    Any sort of political/PR fallout in any organization can be greatly limited or eliminated if you just explain a change as an "experiment" rather than something deliberate. "We were just running an experiment; we do lots of those. We'll stop that particular experiment. No harm no foul" is much more palatable than "We thought we'd make that change. We will revert it. Sorry about that". With the former people think: "Those guys are always experimenting with new stuff. With experimentations comes hiccups, but experimentation is generally good" With the later; now people would wanna know more about your decision-making process. How and why that decision was made. What were the underlying reasons? What was your end goal with such a change? Do you actually have a plan or are you just stumbling in the dark?
baxtr 2y ago
You need to have a lot of traffic for it to make sense. And then the effect size needs to be big enough to notice. Not easy.

nightpool 2y ago

It probably helps that one of PostHog's core products is an A/B testing framework, so it's much easier for them to iterate on it internally for what they need to A/B PostHog. Even when you already have a best in class A/B testing framework though, I agree—A/B testing too much or waiting too long for "more data" to make decisions can slow down momentum badly for features that should be no-brainers.

abxyz 2y ago

A/B testing is also high risk: a good test produces valuable data, a bad test produces harmful data. A company that does no testing is better off than a company that does bad testing. Many people treat product decisions made based on test results as unimpeachable, whereas they treat product decisions made on a hunch with a healthy skepticism that leads to better outcomes.

pkaler 2y ago

Agree! Most orgs should just be shipping features. Before starting an Experiment Program teams should be brainstorming a portfolio of experiments. Just create a spreadsheet where the first column is a one-line hypothesis of the experiment. Eg. "Removing step X from the funnel will increase metric Y while reducing metric Z". And the RICE (Reach-Impact-Confidence-Estimation) score your portfolio. If the team can't come up with a portfolio of 10s to 100s of experiments then the team should just be shipping stuff. And then Experiment Buildout should be standardized. Have standardized XRD (Experiment Requirements Doc). Standardize Eligibility and Enrollment criteria. Which population sees this experiment? When do they see it? How do you test that bucketing is happening correctly? What events do analysts need? When do we do readouts? That's just off the top of my head. Most orgs should just be shipping features.

bluGill 2y ago

Most A/B tests should not be done in a production like way. Grab some post-it notes and and sketch out both A and B: then watch users work with it. For a lot of what you want to know the above will give better information than a 100% polished A/B test in production. When people see a polished product they won't give you the same type of feedback as they will for an obvious quick sketch. The UI industry has gone wrong by making A/B in production too easy, even though the above has been known for many decades. (A/B in production isn't all bad, but it is the last step, and often not worth it)

marcosdumay 2y ago
You are pushing for bit of too much of name-misuse. A/B tests are something what the entire point is that you run them on the real world, and gather how real users react. Design-time user testing has been a thing for much longer than A/B tests. They are a different thing. I mean, your point stands. But you can't do A/B tests on anything that is not production, those your are recommending are a different kind of tests.
- bluGill 2y ago
  I'll accept your definition correction. However I think my point still stands: there are better things than A/B testing to get the information you need.

simonw 2y ago

I think A/B testing is one of the most expensive ways of getting feedback on a product feature. - You have to make good decisions about what you're going to test - You have to build the feature twice - You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them. - You have to test both versions of the feature AND the tracking and test selection mechanisms really well, because bugs in any of those invalidate the test - You have to run it in production for several weeks (or you won't get statistically significant results) - and ensure it doesn't overlap with other tests in a way that could bias the results - You'd better be good at statistics. I've seen plenty of A/B test results presented in ways that did not feel statistically sound to me. ... and after all of that, my experience is that a LOT of the tests you run don't show a statistically significant result one way or the other - so all of that effort really didn't teach you much that was useful. The problem is that talking people out of running an A/B test is really hard! No-one ever got fired for suggesting an A/B test - it feels like the "safe" option. Want to do something much cheaper than that which results in a much higher level of information? Run usability tests. Recruit 3-5 testers and watch them use your new feature over screen sharing and talk through what they're doing. This is an order of magnitude cheaper than A/B testing and will probably teach you a whole lot more.

light_triad 2y ago
Some teams think they can A/B test their way to a great product. It can become a socially acceptable mechanism to avoid having opinions and reduce friction. Steve Blank's quote about validating assumptions: "Lean was designed to inform the founders’ vision while they operated frugally at speed. It was not built as a focus group for consensus for those without deep convictions" Is the Lean Startup Dead? (2018) https://medium.com/@sgblank/is-the-lean-startup-dead-71e0517294a0 https://medium.com/@sgblank/is-the-lean-startup-dead-71e0517... Discussed on HN at the time: https://news.ycombinator.com/item?id=17917479 https://news.ycombinator.com/item?id=17917479
- eddythompson80 2y ago
  Any sort of political/PR fallout in any organization can be greatly limited or eliminated if you just explain a change as an "experiment" rather than something deliberate. "We were just running an experiment; we do lots of those. We'll stop that particular experiment. No harm no foul" is much more palatable than "We thought we'd make that change. We will revert it. Sorry about that". With the former people think: "Those guys are always experimenting with new stuff. With experimentations comes hiccups, but experimentation is generally good" With the later; now people would wanna know more about your decision-making process. How and why that decision was made. What were the underlying reasons? What was your end goal with such a change? Do you actually have a plan or are you just stumbling in the dark?

baxtr 2y ago

You need to have a lot of traffic for it to make sense. And then the effect size needs to be big enough to notice. Not easy.