10 ms·

Don't build growth teams

201 points by jasim 7y ago

jdawg77 7y ago
Title changed in between login and viewing this...why is that? Seems like the article is a self congratulating puff piece. Seeing the bio, I'm unimpresed. Zero public exits. Why are you qualified on this topic again?
cousin_it 7y ago
Good article, the author is clearly well informed. In the past I've made the mistake of staying on a growth team for too long, well after the gains mostly ran out. Toward the end, I was doing the best work I could, but it didn't have much impact. Now I'm on another growth team still in the honeymoon phase, grabbing 10% here and 20% there, and thinking what to do next when the gains run out.
cyborgx7 7y ago
>At 95% certainty, you have 19 people saying “yes” and 1 person saying “no.” >At 99% certainty, you have 99 people saying “yes” and 1 person saying “no.” The article was worth the read to me just for this part. I thought I had a pretty decent intuitive understanding of probabilities but this put things in new perspective for me. The rest of the article is marketing gobbledygook to me, though.
- computerex 7y ago
  I didn't understand this part. Why is the author changing scales? Shouldn't it be that at: At 95% certainty, you have 95 people saying "yes" and 5 people saying "no"? So it's easier to make an apples to apples comparison? What point is the author trying to make with changing the scale?
  - tempguy9999 7y ago
    Though obviously equivalent, I think as a percentage 95/5 and as a reduced fraction 19/1, they both have value in different ways to the non-mathematician. Giving 2 intuitive handles on a thing instead of one is helpful.
  - meigwilym 7y ago
    Especially as he goes on to state: > It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect. I had to stop reading here.
    - Marazan 7y ago
      But that's the point. Ratios are really hard to reason about. Your mind sees the 5% going to 1% (or the 95% going to 99%) and thinks it's a small difference. When in actuality it's a big change.
    - UserIsUnused 7y ago
      Depends on how you frame it. There is 80 of difference when you think about "how many users say yes for each No" So, what do you want to know, "yes for every 100" or "yes for every no" ? It matters.
    - Ensorceled 7y ago
      When you thought about it some more, did you come back?
  - arnvald 7y ago
    The point is to show that with 95% for every 1 person saying "no" you have 19 people saying "yes". With 99% for every 1 person saying "no" you have 99 people saying "yes". The common denominator is one "no".
    - ashelmire 7y ago
      That’s not how this works though. The same number of people are landing on your site. You aren’t getting 80 more signups as suggested.
      - kryogen1c 7y ago
        > That’s not how this works though. What youre struggling with is the counterintuitive nature of applied statistics vs pure math, and this is the point TFA was trying to make. > You aren’t getting 80 more signups as suggested TFA isnt saying you "get" more, but just illustrating how different 95 and 99% actually are. Its restating the potato paradox, linked elsewhere in the thread https://en.m.wikipedia.org/wiki/Potato_paradox https://en.m.wikipedia.org/wiki/Potato_paradox
        bobwaycott 7y ago
        Genuinely curious aside—you used TFA twice. It means “the fucking article” here, right? When you use TFA, your otherwise helpful comment reads as if it’s angry, exasperated, or some otherwise negative feeling that stands in opposition to the rest of your comment—which reads as a genuine attempt to be helpful and explain. Why do use TFA?
        lixtra 7y ago
        TFA can also stand for The Fine Article. By HN rules you should assume the best; in this case for you, he was talking about the fine article.
        bobwaycott 7y ago
  - samastur 7y ago
    I'm not the author of the article, but I sometimes use the same approach because in my experience saying 5 people out of 100 does not have the same effect as 1 in 20 even though mathematically they are same. It provides a mental check if you are still ok with your original perception.
  - dkersten 7y ago
    Comparing A:B against C:D is much less intuitive for us puny humans than comparing A:B against C:B.
  - mytailorisrich 7y ago
    He is not changing scale. He is just illustrating what the probability means in terms of the size of the set you need to expect the event to occur once, which has a more practical value than a probability.
  - Ensorceled 7y ago
    > So it's easier to make an apples to apples comparison? That's exactly the point. How many people said yes for every one who said no: 50% -> 1 yes for every no 95% -> 19 yeses for every no 99% -> 99 yeses for every no 99.9% -> 999 yeses for every no
  - bigiain 7y ago
    In the context of A/B Testing, if you've decided to stop the test at "95% significance" then you'll stop at the 19 yes, 1 no spot (or maybe 38:2). If you're testing for 99% significance, you won't stop unless you get to 99:1 (or more likely, you'll need to test way past that because "no" number 2 will arrive before "yes" number 99, so you'll need to test to 198:2 or 297:3 or further. This is why you should never stop an A/B test once you've "hit your statistical significance". Always choose the number of tests you'd need to prove the significance before you start, and let it run even if it's "obviously winning" (or losing).
    - krumpet 7y ago
      Can you elaborate on this? When an A/B test has hit the desired significance, what is the value obtained in testing further?
- brianberns 7y ago
  Here's a little puzzle that really puts this in perspective: I put 100 lb. of potatoes in the sun. The potatoes were 99% water by weight. After drying a while, the potatoes were 98% water. How much did they weigh at that point? https://en.wikipedia.org/wiki/Potato_paradox https://en.wikipedia.org/wiki/Potato_paradox
  - vishnugupta 7y ago
    In my college days, we used a similar trick question to harass (or ragging as known in India) juniors. Beginning with $100 how much will you be left with if you lose 50% and gain 50%?
    - maneesh 7y ago
      It's $75, right? Don't leave us in suspense !
      - cityzen 7y ago
        That’s how they rag on juniors, they never tell them the answer!
      - vishnugupta 7y ago
        Haha! It's indeed $75 ️.
    - Normal_gaussian 7y ago
      This is a technique often used in gambling games (poker apps minigames etc.) They offer a series of actions that let you lose or gain x% n many times in a row, with an equal chance of each. The fact that +x% doesn't cancel out a -x% compounds on the difference in lay person expectations to produce horrific odds for microtransactions.
  - sundbry 7y ago
    51 pounds?
- ncphillips 7y ago
  It’s actually nonsense. Certainty has nothing to do with how many people said yes or no, it’s about how many people are in the experiment. If you sample size of 2, and both like the new UI, then you can’t say you have 100% certainty. I.e. 3/4 people isn’t the same certainty as 75/100 I think the point he’s trying to make is it’s easier to get the statement “95% of people prefer X” from a study then to get “99% of people”. The first requires >20 people, the later >100. But he’s tying together the concepts of sample size, confidence, and the outcome of the experiment in a way hat doesn’t make sense
  - Nimitz14 7y ago
    That's not at all what he's saying. Maybe understand the article first before calling it nonsense. Kind of funny how all you people are making his point for him!
- 127 7y ago
  To make it match with human brain, this would be more helpful indeed: >At 5% uncertainty >At 1% uncertainty
- benschulz 7y ago
  I actually thought it was the worst part of the article because it fails to put the statements into context. Let me quote from "0 And 1 Are Not Probabilities"[1]. > In probabilities, 0.9999 and 0.99999 seem to be only 0.00009 apart, so that 0.502 is much further away from 0.503 than 0.9999 is from 0.99999. To get to probability 1 from probability 0.99999, it seems like you should need to travel a distance of merely 0.00001. > But when you transform to odds ratios, 0.502 and 0.503 go to 1.008 and 1.012, and 0.9999 and 0.99999 go to 9,999 and 99,999. And when you transform to log odds, 0.502 and 0.503 go to 0.03 decibels and 0.05 decibels, but 0.9999 and 0.99999 go to 40 decibels and 50 decibels. > When you work in log odds, the distance between any two degrees of uncertainty equals the amount of evidence you would need to go from one to the other. That is, the log odds gives us a natural measure of spacing among degrees of confidence. [1]: https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-ar... Edit: formatting
  - mattkrause 7y ago
    The quoted part is sensible, but the title is goofy. Zero and one have to be probabilities. A lot of the underlying math doesn't work out otherwise, but you can also believe something always/never happens (if you're a Bayesian), or observe the same thing happening over and over again (for the frequentists). Your example shows that (non-linear) transformations aren't linear, and that our gut feels about likelihood aren't either, which is true enough. However, you wouldn't say "101 ˚C isn't a temperature" because it takes much less energy to heat a wet thing from 97 to 99˚ than it does to bring it from 99 to 101˚ because of the phase change.
    - marcosdumay 7y ago
      You can not ever discover in a statistical test that some probability is 0 or 1. Those are impossible values. You can get arbitrarily close to them, but you can never reach those values. Of course, they are valid values for the set of probabilities, just like 0K is a valid temperature, and c is a valid speed for massive things. You just will never see any of those.
d--b 7y ago
Wow, amazing article. Note for those who scan through, recommendations about what to do instead are at the bottom.
seddona 7y ago
Very useful content and perspective. I was certain it was all building to the conclusion you should hire the author's growth agency to get that 2x conversion gain and then leave the funnel alone, but the ask never came!
- reggieband 7y ago
  I was expecting that too but it could also be a savvy attempt at gaining credibility. The real thing I was thinking was that productizing the activities of a growth team would be ideal. If you could build a software product that efficiently managed the process of the growth team you could make a good value proposition. e.g. one person ($150k/year) using one tool ($100k/year) is significantly cheaper than the $650k/year he quotes.
cobbzilla 7y ago
I liked this part: “I have a rule-of-thumb for picking A/B test winners: Whichever version doesn’t make any sense or seems like it would never work, bet on that. More often than I like to admit, the dumber or weirder version wins. This is actually how I tell if a company is truly optimizing their funnel. From a branding or UX perspective, it should feel a little “off.” If it’s polished and everything makes sense, they haven’t pushed that hard on optimization.” ...but if you take this too far you end up with something like Amazon.com, where everything is “a little off”?
- UserIsUnused 7y ago
  And it works. Amazon is the leader. Are you sure it's despite its UX or did UX contributed?
  - cobbzilla 7y ago
    I suppose the cynical response is “there’s no such thing as taking it too far”, hence Amazon.
  - s3r3nity 7y ago
    This. I always use Amazon and eBay as go-to examples of Optimization that works as a counter-force to those that want massive redesigns every 6 months because some other new app or competitor has some sexy slick UI.
- stdbrouw 7y ago
  That's actually the part that I found a little bit ironic (though I very much enjoyed the post), considering that the author links to and praises a study that points out that many A/B tests result in illusory wins because growth teams end up testing a zillion different variations and stop tests early whenever one alternative seems promising. In such an environment, you're bound to end up with all kinds of weird "winners" but there's no guarantee that (1) they're actually better than the alternatives you tested and (2) even if they are, that the advantage is stable and not just a temporary novelty effect.
- jefftk 7y ago
C1sc0cat 7y ago
I think the big take away is that still today the decision makers don't really understand digital and are still working in the same way that Mad Men did in the 60's Won't mention the brand but I have in 2019 had in the context of a new website) comments of don't want to much text we will let images lead - which is just not going to work. This is supposedly the digital native generation who are 20 years younger than me edited to correct naïve to native
- chrisweekly 7y ago
  naïve (ignorant) -> native (fluent / expert) not notpicking; those words have opposite meanings here
  - C1sc0cat 7y ago
    Oops a frieduan slip there (or an autocorrect gone wrong) you are of course quite correct.
- KaoruAoiShiho 7y ago
  Was there data to show that they're wrong?
- cercatrova 7y ago
  Seems correct to me. People don't read text, but they'll look at pictures.
  - C1sc0cat 7y ago
    Google doesn't :-) which is the point here
tacone 7y ago
The issue is hyper-optimization is very easy to break and very hard to maintain. In a way it's an anti-pattern you should do only if there's no other way to grow. That is: if you want a car to go faster you may improve the engine, or focus to make the car lighter. If you hyper-optimize the engine the smallest variation in the fuel will affect performance. If you hyper-optimize the weight you better watch out on your own diet :) otherwise the car won't be as fast as intended. All in all it often goes back to following the Pareto principle. If you need to hyper-optimize you landing pages you probably need to focus elsewhere. You need good landing pages, not perfect ones.
- ohduran 7y ago
  Agreed. Relying on growth tactics while ignoring a well-thought marketing strategy is doomed to fail.
- aaronblohowiak 7y ago
  You are approaching a discussion of one of my favorite topics "adaptation vs adaptability"[1]. Basically, there is a spectrum between perfect efficiency and being completely flexible. we see this over and over again, from software (where flexibility might show as generality or abstraction) to organisms/ecosystems to statistical methods. https://people.clas.ufl.edu/ulan/files/Conrad.pdf https://people.clas.ufl.edu/ulan/files/Conrad.pdf
  - celticmusic 7y ago
    It's the basis of evolution in general. Something can be perfectly optimized for an environment, but when that environment changes they may go extinct from it. so instead don't necessary want perfectly optimized, we want good enough, because inside that good enough is where adaptability to changes comes from.
    - aaronblohowiak 7y ago
      yes, and the article i linked to explores this in terms of ecosystems and the flows of material and energy between different organisms within the ecosystem.
KaoruAoiShiho 7y ago
Can someone shed some light onto the executives hated homepage that was discussed? Since kissmetrics is an analytics company, doing this kind of thing is their core competency. How could executives change it to a loser immediately? There must be more to the story or some kind of other side to the story. Shooting from the hip here possible problems with the "overly optimized" homepage: 1. Long term brand damage? If people go to the site and see something not as "polished" or "professional" looking, the bounces won't leave with just a neutral impression but a straight up negative impression. Sometimes even without a conversion a landing page can sell a visitor on a company and lead to recommendations or return visits down the line. 2. Lower value conversions? Is it possible that people who would immediately sign up without a long sales page end up being less valuable? This is more easily tracked so I'm sure OP would've accounted for this but it's still hard to tell without a ton of data.
- wjossey 7y ago
  I can’t speak to this group specifically, just to say that being an exec in an industry doesn’t make you an expert in that industry. When I worked in ad tech, I met plenty of execs who couldn’t run a facebook campaign to save their life. If you’re a founder of an analytics company hiring your CRO, it’d be great to find someone with an analytics background, but more than likely they’re coming in having worked in tech, but not that specific vertical.
  - KaoruAoiShiho 7y ago
    It's specifically that vertical. Their product is designed for growth hacking, if they discard the results it's like saying their own product doesn't work.
- mlyle 7y ago
  This is really important, I think, and overlooked. A/B testing micro-optimizes an outcome you're searching for. It doesn't tell you the long term trajectory of customers, because that takes way too long to get feedback. An example: "dark patterns". Facebook is deeply damaged, in part by making choices from well-run A/B trials. Worse, A/B testing searches out local maxima. Maybe you get locked into an approach that precludes you from making the changes that would really drive the visit ultimately. Performance indicators and properly using quantitative information are really important-- and many executives aren't versed at this. But, conversely, you can't use A/B testing to decide who you are as a company and define your relationship with the customer.
harryf 7y ago
> I have yet to come across a designer, engineer, or marketer that intuitively understood probability on day one. Sad but true. And it’s not even about understanding the more in depth math. It’s about having an intuition about the 80/20 rule or Venn diagrams. “If I work on this bug I can make 80% of our users happier but if I work on this bug, which affects no one but our CMO who always complains loudly...” ... is a type of discussion I’ve found happens too rarely
airnomad 7y ago
There's always something to work on, so the problem is sometimes that growth teams are ofte too narrowly focused. You're done with landing page, then move on AdWords. You're done with AdWords, move on improving sales team. Making forms work better is not everything.
Ididntdothis 7y ago
“9. Growth teams have limited revenue potential.” This one is important for almost all optimization efforts. Once the low hanging fruit is gone further efforts often don’t produce much but can cause harm instead. Stack ranking is a good example. It may make sense to identify and lay off the bottom 10% for one or two years. The you should stop. But if you keep going you get all kinds of weird dynamics that harm morale instead of improving performance.
vincent-toups 7y ago
"Take 95% certainty compared to 99%. Because 95 is pretty close to 99, it feels like the difference should be minimal. In reality, there’s a gulf between those two benchmarks: At 95% certainty, you have 19 people saying “yes” and 1 person saying “no.” At 99% certainty, you have 99 people saying “yes” and 1 person saying “no.” It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect." I'm a data scientist, I feel like this is a really wildly confusing way to put this.
- enraged_camel 7y ago
  Reminds me of when I was shopping for a windbreaker. A brand had two models: one that blocked 98% of the wind and another that blocked 99%. To a casual observer, it wouldn’t be immediately obvious why the second one was significantly more expensive. After all, it’s just a 1% difference! (Of course, the answer is obvious: the second one was twice as effective at blocking the wind.)
  - eximius 7y ago
    That is a questionable interpretation. It means the 98% one lets twice as much air in, but it doesn't block twice as much wind because the amount of ambient wind remains constant. To be more explicit, if there are 100 units of wind and one blocks 98 units, the other blocks 99 units. So one lets twice as much air in than the other, but I wouldn't consider that saying that one was 'twice as effective'. The relative improvement is (99-98)/98 ~ 1%.
    - enraged_camel 7y ago
      Your comment illustrates the point I'm making. If windbreaker A let's in 2 units of wind out of 100 and windbreaker B lets in 1 unit of wind out of 100, that means windbreaker B is twice as effective at blocking wind, because it lets in half as much wind as windbreaker A.
      - kilburn 7y ago
encoderer 7y ago
He doesn't directly address that relentless A/B testing can be a pathology that removes the soul and flexibility from your product and can wall in future growth. This is because you have to optimize on one (or selected) KPIs but the true health of your business is more nuanced than that. Large analytics operations try to solve this by creating ever more complex KPI models but that is not attainable for most. As a crude example if you have a community site like Reddit you could harm the community by over-optimizing your conversion funnel for Reddit Gold, while still substantially growing the Reddit Gold business. For a time. He touches on this by admitting that the super-optimized KISSMetrics homepage was not the right strategy but I would have loved read more.
- everythingswan 7y ago
  I believe there will be plenty more stories like this coming out and landing well. Some of the testing I've managed hasn't helped end-to-end conversion rate _at all_ and did nothing short-term results even when it looked great at first glance. I feel pretty jaded about "performance marketing" and "growth marketing" at this point. There are legitimate growth marketers out there but it's become a cop out for The Hard Things in marketing. Essentially, it's become really easy to get leads for cheap and incredibly expensive to get them to convert. The question becomes, which master do you serve? I think marketing leaders need to delicately balance the KPIs with the vision of the company or product. In High Output Management, Andy Grove talked about negative indicators for KPIs as well (forgive me, don't have book within reach). I don't think any time gets spent on that in my network. Granted, it is hard to do with small teams and you might say it's the wrong thing to focus on for small teams. Eventually, you reach the point where the numbers look too good to be true and they usually are--meaning you won't convert these people because they're not ready to buy what you're selling. They're just ready to get the free info you're offering them in two clicks of their time. I appreciate the candidness about failure even if it wasn't detailed. Can't wait for more similar stories.
- jasonless 7y ago
  If one claims to have growth experience. Enough to create a blog post about it. And one can't figure out the appropriate strategy to make a return on your head count - then stop pretending . A growth team should be designed to meet the business requirements and make a return (on top of your head count). Because one can't create an effective strategy to produce a return, doesn't mean you toss out the entire idea of creating Growth Teams. Because you failed - it doesn't mean its bad. It wasn't bad for Facebook.
aaronblohowiak 7y ago
I highly recommend considering https://medium.com/convoy-tech/the-power-of-bayesian-a-b-testing-f859d2219d5 https://medium.com/convoy-tech/the-power-of-bayesian-a-b-tes... as an alternative to p-value hacking / early stopping.

201 points by jasim 7y ago

Title changed in between login and viewing this...why is that? Seems like the article is a self congratulating puff piece. Seeing the bio, I'm unimpresed. Zero public exits. Why are you qualified on this topic again?

cousin_it 7y ago

Good article, the author is clearly well informed. In the past I've made the mistake of staying on a growth team for too long, well after the gains mostly ran out. Toward the end, I was doing the best work I could, but it didn't have much impact. Now I'm on another growth team still in the honeymoon phase, grabbing 10% here and 20% there, and thinking what to do next when the gains run out.

cyborgx7 7y ago

>At 95% certainty, you have 19 people saying “yes” and 1 person saying “no.” >At 99% certainty, you have 99 people saying “yes” and 1 person saying “no.” The article was worth the read to me just for this part. I thought I had a pretty decent intuitive understanding of probabilities but this put things in new perspective for me. The rest of the article is marketing gobbledygook to me, though.

computerex 7y ago
I didn't understand this part. Why is the author changing scales? Shouldn't it be that at: At 95% certainty, you have 95 people saying "yes" and 5 people saying "no"? So it's easier to make an apples to apples comparison? What point is the author trying to make with changing the scale?
- tempguy9999 7y ago
  Though obviously equivalent, I think as a percentage 95/5 and as a reduced fraction 19/1, they both have value in different ways to the non-mathematician. Giving 2 intuitive handles on a thing instead of one is helpful.
- meigwilym 7y ago
  Especially as he goes on to state: > It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect. I had to stop reading here.
  - Marazan 7y ago
    But that's the point. Ratios are really hard to reason about. Your mind sees the 5% going to 1% (or the 95% going to 99%) and thinks it's a small difference. When in actuality it's a big change.
  - UserIsUnused 7y ago
    Depends on how you frame it. There is 80 of difference when you think about "how many users say yes for each No" So, what do you want to know, "yes for every 100" or "yes for every no" ? It matters.
  - Ensorceled 7y ago
    When you thought about it some more, did you come back?
- arnvald 7y ago
  The point is to show that with 95% for every 1 person saying "no" you have 19 people saying "yes". With 99% for every 1 person saying "no" you have 99 people saying "yes". The common denominator is one "no".
  - ashelmire 7y ago
    That’s not how this works though. The same number of people are landing on your site. You aren’t getting 80 more signups as suggested.
    - kryogen1c 7y ago
      > That’s not how this works though. What youre struggling with is the counterintuitive nature of applied statistics vs pure math, and this is the point TFA was trying to make. > You aren’t getting 80 more signups as suggested TFA isnt saying you "get" more, but just illustrating how different 95 and 99% actually are. Its restating the potato paradox, linked elsewhere in the thread https://en.m.wikipedia.org/wiki/Potato_paradox https://en.m.wikipedia.org/wiki/Potato_paradox
      - bobwaycott 7y ago
        Genuinely curious aside—you used TFA twice. It means “the fucking article” here, right? When you use TFA, your otherwise helpful comment reads as if it’s angry, exasperated, or some otherwise negative feeling that stands in opposition to the rest of your comment—which reads as a genuine attempt to be helpful and explain. Why do use TFA?
        lixtra 7y ago
        TFA can also stand for The Fine Article. By HN rules you should assume the best; in this case for you, he was talking about the fine article.
        bobwaycott 7y ago
- samastur 7y ago
  I'm not the author of the article, but I sometimes use the same approach because in my experience saying 5 people out of 100 does not have the same effect as 1 in 20 even though mathematically they are same. It provides a mental check if you are still ok with your original perception.
- dkersten 7y ago
  Comparing A:B against C:D is much less intuitive for us puny humans than comparing A:B against C:B.
- mytailorisrich 7y ago
  He is not changing scale. He is just illustrating what the probability means in terms of the size of the set you need to expect the event to occur once, which has a more practical value than a probability.
- Ensorceled 7y ago
  > So it's easier to make an apples to apples comparison? That's exactly the point. How many people said yes for every one who said no: 50% -> 1 yes for every no 95% -> 19 yeses for every no 99% -> 99 yeses for every no 99.9% -> 999 yeses for every no
- bigiain 7y ago
  In the context of A/B Testing, if you've decided to stop the test at "95% significance" then you'll stop at the 19 yes, 1 no spot (or maybe 38:2). If you're testing for 99% significance, you won't stop unless you get to 99:1 (or more likely, you'll need to test way past that because "no" number 2 will arrive before "yes" number 99, so you'll need to test to 198:2 or 297:3 or further. This is why you should never stop an A/B test once you've "hit your statistical significance". Always choose the number of tests you'd need to prove the significance before you start, and let it run even if it's "obviously winning" (or losing).
  - krumpet 7y ago
    Can you elaborate on this? When an A/B test has hit the desired significance, what is the value obtained in testing further?
brianberns 7y ago
Here's a little puzzle that really puts this in perspective: I put 100 lb. of potatoes in the sun. The potatoes were 99% water by weight. After drying a while, the potatoes were 98% water. How much did they weigh at that point? https://en.wikipedia.org/wiki/Potato_paradox https://en.wikipedia.org/wiki/Potato_paradox
- vishnugupta 7y ago
  In my college days, we used a similar trick question to harass (or ragging as known in India) juniors. Beginning with $100 how much will you be left with if you lose 50% and gain 50%?
  - maneesh 7y ago
    It's $75, right? Don't leave us in suspense !
    - cityzen 7y ago
      That’s how they rag on juniors, they never tell them the answer!
    - vishnugupta 7y ago
      Haha! It's indeed $75 ️.
  - Normal_gaussian 7y ago
    This is a technique often used in gambling games (poker apps minigames etc.) They offer a series of actions that let you lose or gain x% n many times in a row, with an equal chance of each. The fact that +x% doesn't cancel out a -x% compounds on the difference in lay person expectations to produce horrific odds for microtransactions.
- sundbry 7y ago
  51 pounds?
ncphillips 7y ago
It’s actually nonsense. Certainty has nothing to do with how many people said yes or no, it’s about how many people are in the experiment. If you sample size of 2, and both like the new UI, then you can’t say you have 100% certainty. I.e. 3/4 people isn’t the same certainty as 75/100 I think the point he’s trying to make is it’s easier to get the statement “95% of people prefer X” from a study then to get “99% of people”. The first requires >20 people, the later >100. But he’s tying together the concepts of sample size, confidence, and the outcome of the experiment in a way hat doesn’t make sense
- Nimitz14 7y ago
  That's not at all what he's saying. Maybe understand the article first before calling it nonsense. Kind of funny how all you people are making his point for him!
127 7y ago
To make it match with human brain, this would be more helpful indeed: >At 5% uncertainty >At 1% uncertainty
benschulz 7y ago
I actually thought it was the worst part of the article because it fails to put the statements into context. Let me quote from "0 And 1 Are Not Probabilities"[1]. > In probabilities, 0.9999 and 0.99999 seem to be only 0.00009 apart, so that 0.502 is much further away from 0.503 than 0.9999 is from 0.99999. To get to probability 1 from probability 0.99999, it seems like you should need to travel a distance of merely 0.00001. > But when you transform to odds ratios, 0.502 and 0.503 go to 1.008 and 1.012, and 0.9999 and 0.99999 go to 9,999 and 99,999. And when you transform to log odds, 0.502 and 0.503 go to 0.03 decibels and 0.05 decibels, but 0.9999 and 0.99999 go to 40 decibels and 50 decibels. > When you work in log odds, the distance between any two degrees of uncertainty equals the amount of evidence you would need to go from one to the other. That is, the log odds gives us a natural measure of spacing among degrees of confidence. [1]: https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-are-not-probabilities https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-ar... Edit: formatting
- mattkrause 7y ago
  The quoted part is sensible, but the title is goofy. Zero and one have to be probabilities. A lot of the underlying math doesn't work out otherwise, but you can also believe something always/never happens (if you're a Bayesian), or observe the same thing happening over and over again (for the frequentists). Your example shows that (non-linear) transformations aren't linear, and that our gut feels about likelihood aren't either, which is true enough. However, you wouldn't say "101 ˚C isn't a temperature" because it takes much less energy to heat a wet thing from 97 to 99˚ than it does to bring it from 99 to 101˚ because of the phase change.
  - marcosdumay 7y ago
    You can not ever discover in a statistical test that some probability is 0 or 1. Those are impossible values. You can get arbitrarily close to them, but you can never reach those values. Of course, they are valid values for the set of probabilities, just like 0K is a valid temperature, and c is a valid speed for massive things. You just will never see any of those.

d--b 7y ago

Wow, amazing article. Note for those who scan through, recommendations about what to do instead are at the bottom.

seddona 7y ago

Very useful content and perspective. I was certain it was all building to the conclusion you should hire the author's growth agency to get that 2x conversion gain and then leave the funnel alone, but the ask never came!

reggieband 7y ago
I was expecting that too but it could also be a savvy attempt at gaining credibility. The real thing I was thinking was that productizing the activities of a growth team would be ideal. If you could build a software product that efficiently managed the process of the growth team you could make a good value proposition. e.g. one person ($150k/year) using one tool ($100k/year) is significantly cheaper than the $650k/year he quotes.

cobbzilla 7y ago

I liked this part: “I have a rule-of-thumb for picking A/B test winners: Whichever version doesn’t make any sense or seems like it would never work, bet on that. More often than I like to admit, the dumber or weirder version wins. This is actually how I tell if a company is truly optimizing their funnel. From a branding or UX perspective, it should feel a little “off.” If it’s polished and everything makes sense, they haven’t pushed that hard on optimization.” ...but if you take this too far you end up with something like Amazon.com, where everything is “a little off”?

UserIsUnused 7y ago
And it works. Amazon is the leader. Are you sure it's despite its UX or did UX contributed?
- cobbzilla 7y ago
  I suppose the cynical response is “there’s no such thing as taking it too far”, hence Amazon.
- s3r3nity 7y ago
  This. I always use Amazon and eBay as go-to examples of Optimization that works as a counter-force to those that want massive redesigns every 6 months because some other new app or competitor has some sexy slick UI.
stdbrouw 7y ago
That's actually the part that I found a little bit ironic (though I very much enjoyed the post), considering that the author links to and praises a study that points out that many A/B tests result in illusory wins because growth teams end up testing a zillion different variations and stop tests early whenever one alternative seems promising. In such an environment, you're bound to end up with all kinds of weird "winners" but there's no guarantee that (1) they're actually better than the alternatives you tested and (2) even if they are, that the advantage is stable and not just a temporary novelty effect.
jefftk 7y ago

C1sc0cat 7y ago

I think the big take away is that still today the decision makers don't really understand digital and are still working in the same way that Mad Men did in the 60's Won't mention the brand but I have in 2019 had in the context of a new website) comments of don't want to much text we will let images lead - which is just not going to work. This is supposedly the digital native generation who are 20 years younger than me edited to correct naïve to native

chrisweekly 7y ago
naïve (ignorant) -> native (fluent / expert) not notpicking; those words have opposite meanings here
- C1sc0cat 7y ago
  Oops a frieduan slip there (or an autocorrect gone wrong) you are of course quite correct.
KaoruAoiShiho 7y ago
Was there data to show that they're wrong?
cercatrova 7y ago
Seems correct to me. People don't read text, but they'll look at pictures.
- C1sc0cat 7y ago
  Google doesn't :-) which is the point here

tacone 7y ago

The issue is hyper-optimization is very easy to break and very hard to maintain. In a way it's an anti-pattern you should do only if there's no other way to grow. That is: if you want a car to go faster you may improve the engine, or focus to make the car lighter. If you hyper-optimize the engine the smallest variation in the fuel will affect performance. If you hyper-optimize the weight you better watch out on your own diet :) otherwise the car won't be as fast as intended. All in all it often goes back to following the Pareto principle. If you need to hyper-optimize you landing pages you probably need to focus elsewhere. You need good landing pages, not perfect ones.

ohduran 7y ago
Agreed. Relying on growth tactics while ignoring a well-thought marketing strategy is doomed to fail.
aaronblohowiak 7y ago
You are approaching a discussion of one of my favorite topics "adaptation vs adaptability"[1]. Basically, there is a spectrum between perfect efficiency and being completely flexible. we see this over and over again, from software (where flexibility might show as generality or abstraction) to organisms/ecosystems to statistical methods. https://people.clas.ufl.edu/ulan/files/Conrad.pdf https://people.clas.ufl.edu/ulan/files/Conrad.pdf
- celticmusic 7y ago
  It's the basis of evolution in general. Something can be perfectly optimized for an environment, but when that environment changes they may go extinct from it. so instead don't necessary want perfectly optimized, we want good enough, because inside that good enough is where adaptability to changes comes from.
  - aaronblohowiak 7y ago
    yes, and the article i linked to explores this in terms of ecosystems and the flows of material and energy between different organisms within the ecosystem.

KaoruAoiShiho 7y ago

Can someone shed some light onto the executives hated homepage that was discussed? Since kissmetrics is an analytics company, doing this kind of thing is their core competency. How could executives change it to a loser immediately? There must be more to the story or some kind of other side to the story. Shooting from the hip here possible problems with the "overly optimized" homepage: 1. Long term brand damage? If people go to the site and see something not as "polished" or "professional" looking, the bounces won't leave with just a neutral impression but a straight up negative impression. Sometimes even without a conversion a landing page can sell a visitor on a company and lead to recommendations or return visits down the line. 2. Lower value conversions? Is it possible that people who would immediately sign up without a long sales page end up being less valuable? This is more easily tracked so I'm sure OP would've accounted for this but it's still hard to tell without a ton of data.

wjossey 7y ago
I can’t speak to this group specifically, just to say that being an exec in an industry doesn’t make you an expert in that industry. When I worked in ad tech, I met plenty of execs who couldn’t run a facebook campaign to save their life. If you’re a founder of an analytics company hiring your CRO, it’d be great to find someone with an analytics background, but more than likely they’re coming in having worked in tech, but not that specific vertical.
- KaoruAoiShiho 7y ago
  It's specifically that vertical. Their product is designed for growth hacking, if they discard the results it's like saying their own product doesn't work.
mlyle 7y ago
This is really important, I think, and overlooked. A/B testing micro-optimizes an outcome you're searching for. It doesn't tell you the long term trajectory of customers, because that takes way too long to get feedback. An example: "dark patterns". Facebook is deeply damaged, in part by making choices from well-run A/B trials. Worse, A/B testing searches out local maxima. Maybe you get locked into an approach that precludes you from making the changes that would really drive the visit ultimately. Performance indicators and properly using quantitative information are really important-- and many executives aren't versed at this. But, conversely, you can't use A/B testing to decide who you are as a company and define your relationship with the customer.

harryf 7y ago

> I have yet to come across a designer, engineer, or marketer that intuitively understood probability on day one. Sad but true. And it’s not even about understanding the more in depth math. It’s about having an intuition about the 80/20 rule or Venn diagrams. “If I work on this bug I can make 80% of our users happier but if I work on this bug, which affects no one but our CMO who always complains loudly...” ... is a type of discussion I’ve found happens too rarely

airnomad 7y ago

There's always something to work on, so the problem is sometimes that growth teams are ofte too narrowly focused. You're done with landing page, then move on AdWords. You're done with AdWords, move on improving sales team. Making forms work better is not everything.

Ididntdothis 7y ago

“9. Growth teams have limited revenue potential.” This one is important for almost all optimization efforts. Once the low hanging fruit is gone further efforts often don’t produce much but can cause harm instead. Stack ranking is a good example. It may make sense to identify and lay off the bottom 10% for one or two years. The you should stop. But if you keep going you get all kinds of weird dynamics that harm morale instead of improving performance.

vincent-toups 7y ago

"Take 95% certainty compared to 99%. Because 95 is pretty close to 99, it feels like the difference should be minimal. In reality, there’s a gulf between those two benchmarks: At 95% certainty, you have 19 people saying “yes” and 1 person saying “no.” At 99% certainty, you have 99 people saying “yes” and 1 person saying “no.” It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect." I'm a data scientist, I feel like this is a really wildly confusing way to put this.

enraged_camel 7y ago
Reminds me of when I was shopping for a windbreaker. A brand had two models: one that blocked 98% of the wind and another that blocked 99%. To a casual observer, it wouldn’t be immediately obvious why the second one was significantly more expensive. After all, it’s just a 1% difference! (Of course, the answer is obvious: the second one was twice as effective at blocking the wind.)
- eximius 7y ago
  That is a questionable interpretation. It means the 98% one lets twice as much air in, but it doesn't block twice as much wind because the amount of ambient wind remains constant. To be more explicit, if there are 100 units of wind and one blocks 98 units, the other blocks 99 units. So one lets twice as much air in than the other, but I wouldn't consider that saying that one was 'twice as effective'. The relative improvement is (99-98)/98 ~ 1%.
  - enraged_camel 7y ago
    Your comment illustrates the point I'm making. If windbreaker A let's in 2 units of wind out of 100 and windbreaker B lets in 1 unit of wind out of 100, that means windbreaker B is twice as effective at blocking wind, because it lets in half as much wind as windbreaker A.
    - kilburn 7y ago

encoderer 7y ago

He doesn't directly address that relentless A/B testing can be a pathology that removes the soul and flexibility from your product and can wall in future growth. This is because you have to optimize on one (or selected) KPIs but the true health of your business is more nuanced than that. Large analytics operations try to solve this by creating ever more complex KPI models but that is not attainable for most. As a crude example if you have a community site like Reddit you could harm the community by over-optimizing your conversion funnel for Reddit Gold, while still substantially growing the Reddit Gold business. For a time. He touches on this by admitting that the super-optimized KISSMetrics homepage was not the right strategy but I would have loved read more.

everythingswan 7y ago
I believe there will be plenty more stories like this coming out and landing well. Some of the testing I've managed hasn't helped end-to-end conversion rate _at all_ and did nothing short-term results even when it looked great at first glance. I feel pretty jaded about "performance marketing" and "growth marketing" at this point. There are legitimate growth marketers out there but it's become a cop out for The Hard Things in marketing. Essentially, it's become really easy to get leads for cheap and incredibly expensive to get them to convert. The question becomes, which master do you serve? I think marketing leaders need to delicately balance the KPIs with the vision of the company or product. In High Output Management, Andy Grove talked about negative indicators for KPIs as well (forgive me, don't have book within reach). I don't think any time gets spent on that in my network. Granted, it is hard to do with small teams and you might say it's the wrong thing to focus on for small teams. Eventually, you reach the point where the numbers look too good to be true and they usually are--meaning you won't convert these people because they're not ready to buy what you're selling. They're just ready to get the free info you're offering them in two clicks of their time. I appreciate the candidness about failure even if it wasn't detailed. Can't wait for more similar stories.
jasonless 7y ago
If one claims to have growth experience. Enough to create a blog post about it. And one can't figure out the appropriate strategy to make a return on your head count - then stop pretending . A growth team should be designed to meet the business requirements and make a return (on top of your head count). Because one can't create an effective strategy to produce a return, doesn't mean you toss out the entire idea of creating Growth Teams. Because you failed - it doesn't mean its bad. It wasn't bad for Facebook.

aaronblohowiak 7y ago

I highly recommend considering https://medium.com/convoy-tech/the-power-of-bayesian-a-b-testing-f859d2219d5 https://medium.com/convoy-tech/the-power-of-bayesian-a-b-tes... as an alternative to p-value hacking / early stopping.