4 ms·

I feel like that example is missing some context - if signups did increase then their experiment was successful - we aren’t here to make pretty pages, we’re her

I feel like that example is missing some context - if signups did increase then their experiment was successful - we aren’t here to make pretty pages, we’re here to make money.

shubhamjain 2y ago
The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Most A/B tests focus on the former, very few on the latter. We had a free plan, and most users who signed up never made a single API request. So, assuming that the increase in signups is driving more business is just foolhardy.
- blackoil 2y ago
  You can always track signup/paying-users ratio. Purpose of landing/pricing page is to get the users to sign-up. Unless some dark pattern or misinformation is used to confuse users into sign-up, more users is a positive thing.
  - TeMPOraL 2y ago
    The problem with A/B test optimization is that, unless you're extremely careful, they naturally lead you to apply dark patterns and misinformation.
    - foldr 2y ago
      Ok, but the example we're discussing is one where the signup button was simply moved to a different position on the page. That's not a 'dark pattern'.
- lelanthran 2y ago
  > We had a free plan, and most users who signed up never made a single API request. That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?
- albedoa 2y ago
  > The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Okay? The A/B test sought to measure which of two options A and B led to more signups. > So, assuming that the increase in signups is driving more business is just foolhardy. Your "A/B test enthusiast" was not testing for or trying to prove a causal relationship between increased signups and more business. If he made the claim separately, then that is the context that is missing from now multiple comments.

I feel like that example is missing some context - if signups did increase then their experiment was successful - we aren’t here to make pretty pages, we’re here to make money.

shubhamjain 2y ago

The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Most A/B tests focus on the former, very few on the latter. We had a free plan, and most users who signed up never made a single API request. So, assuming that the increase in signups is driving more business is just foolhardy.

blackoil 2y ago
You can always track signup/paying-users ratio. Purpose of landing/pricing page is to get the users to sign-up. Unless some dark pattern or misinformation is used to confuse users into sign-up, more users is a positive thing.
- TeMPOraL 2y ago
  The problem with A/B test optimization is that, unless you're extremely careful, they naturally lead you to apply dark patterns and misinformation.
  - foldr 2y ago
    Ok, but the example we're discussing is one where the signup button was simply moved to a different position on the page. That's not a 'dark pattern'.
lelanthran 2y ago
> We had a free plan, and most users who signed up never made a single API request. That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?
albedoa 2y ago
> The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Okay? The A/B test sought to measure which of two options A and B led to more signups. > So, assuming that the increase in signups is driving more business is just foolhardy. Your "A/B test enthusiast" was not testing for or trying to prove a causal relationship between increased signups and more business. If he made the claim separately, then that is the context that is missing from now multiple comments.