8 ms·

Four pitfalls of hill climbing

131 points by charlieegan3 11y ago

xlayn 11y ago
TL;DR: Incremental approach could prevent disruption benefits.
- btowngar 11y ago
  There's three to four minutes of reading in this article, possibly less if you're familiar with the concepts. It would be great if users were required to click on the actual article and wait a certain length of time before commenting.
  - xlayn 11y ago
    I did read it. Probably wording didn't help. I see it as the "Newton's method" for finding roots that is know to have the problem of local maximum; you can see different initial values for finding root as R&D.
claar 11y ago
I really liked the animations in this article. I found them very useful in visualizing the authors points about the potential pitfalls of various product development strategies, even if they exaggerate the effects a bit. Also neat that they're SVGs instead of the usual GIFs.
astazangasta 11y ago
Haha, if we are going to use terrible reward function analogies to measure success, why stop at A/B testing? Introduce your features as mutations to your product! Start them off with one or two customers. Those features that make your users quit in a huff will be eliminated. Those that make them recommend your app to a friend will propagate. Survival of the fittest feature!
- natosaichek 11y ago
  I love this terrible idea. I can totally imagine some terrified product manager turning up the mutation rate to try to find some good set of features while people leave because they don't want a mutating product.
  - techdragon 11y ago
    We sometimes call this "pivoting your business" ;-)
ThrustVectoring 11y ago
There's another one I've seen: climbing the wrong hill. Consider a news website doing A/B testing on headlines, and optimizing for getting more clicks on an individual headline compared to other candidates in the slot. The problem is conflating "people want to click on a headline after visiting our site" with "people want to visit our site", and optimizing for the former at the expense of the latter. The end result is clickbait headlines that alienate visitors while they fight to grab attention.
- abraae 11y ago
  A "wrong hill" example in our industry (recruitment software) is a focus on getting as many as possible candidates to find and apply for jobs. If I sell widgets, then yes, I want as many as possible people to come and buy them, and I really only care about the colour of their money. However when as a widget company I am advertising jobs, I really want only qualified candidates to apply. Otherwise I am drowning in people I just need to sort through, and reject - and when I reject them, they may stop buying my widgets. In this situation, bounce rates (e.g. people leaving the site after viewing a job) and a lot of other common metrics are meaningless without far more context.
  - danieltillett 11y ago
    The equivalent in widget selling is time-wasting leads. When you are advertising a job you are looking for leads and unqualified people are just bad leads. Having said this there is little you can do about this problem since any well advertised position will attract the unqualified since the cost is so low to just mass-blast your cv. I have always wanted to have a fee (say $500) to apply for a position and refunded $1000 to the candidates that actually have the skills they claim and which match the job requirements. Real candidates would be rewarded for applying (easy $500) paid for by all the time wasters and dunning-kruger types. While this appeals to me I doubt this idea would be viable in the real world :)
    - ThrustVectoring 11y ago
      I think what you're looking for is something like a preliminary interview round that is cheap for you to administer, combined with a financial reward for those you select to go to the next level of the process. For incentive reasons, the budget for this must come out of the hiring company's recruiting budget. Otherwise, it's a great way for unethical folks to extract money out of the applicant pool. An alternative way is to set up some sort of competition with prizes, and then recruit from those who do well at it.
cialowicz 11y ago
This is a great little read, and really resonated with me. Like the author mentioned, there is some value in A/B testing and an incremental approach to product development. But too often organizations can become obsessed with this path forward, and continuing with the same approach becomes like trying to squeeze blood from a stone -- there are just no more significant gains to be had. Unfortunately the company I'm at right now feels like this. The product team and CEO are obsessed with A/B testing, and don't have a more holistic approach to product development. This means that our competitors are out-innovating us in certain areas, and it's disappointing to watch.
- avn2109 11y ago
  >> "...don't have a more holistic approach to product development..." Interestingly, the article's author works at Twitter.
sopooneo 11y ago
I often wonder if an agile development approach that only plans in the range of very short iterations could run into the same problem.
namenotrequired 11y ago
> Bold changes require lots of resources. Maybe it’s mostly the success-bound companies who have enough resources to afford the bold changes. And even when they do get the causality right, maybe big changes in production are likely to cause big changes in outcome - providing both greater potential reward and greater risk of losing everything. Any sufficiently lucky contrarian is indistinguishable from genius. Edit for clarity
cshimmin 11y ago
Those javascript animations had my CPU running at 100% while I was reading the article. I'm not sure I like this new web... can't we just go back to GIFs for simple stuff like that?
- restalis 11y ago
  The only reason GIFs are a better solution for you (i.e. not taking 100% of the CPU resource) is because it's about a quality GIF render. What you really need is quality of the software that powers the alternative solutions.
llogiq 11y ago
Note that this has little to do with hill-climbing algorithms and much with A/B testing. Though there is an overlap in the problems both face. I wonder if algorithmic solutions like randome restart or simulated annealing could improve the outcome of A/B testing, too.

131 points by charlieegan3 11y ago

xlayn 11y ago

TL;DR: Incremental approach could prevent disruption benefits.

btowngar 11y ago
There's three to four minutes of reading in this article, possibly less if you're familiar with the concepts. It would be great if users were required to click on the actual article and wait a certain length of time before commenting.
- xlayn 11y ago
  I did read it. Probably wording didn't help. I see it as the "Newton's method" for finding roots that is know to have the problem of local maximum; you can see different initial values for finding root as R&D.

claar 11y ago

I really liked the animations in this article. I found them very useful in visualizing the authors points about the potential pitfalls of various product development strategies, even if they exaggerate the effects a bit. Also neat that they're SVGs instead of the usual GIFs.

astazangasta 11y ago

Haha, if we are going to use terrible reward function analogies to measure success, why stop at A/B testing? Introduce your features as mutations to your product! Start them off with one or two customers. Those features that make your users quit in a huff will be eliminated. Those that make them recommend your app to a friend will propagate. Survival of the fittest feature!

natosaichek 11y ago
I love this terrible idea. I can totally imagine some terrified product manager turning up the mutation rate to try to find some good set of features while people leave because they don't want a mutating product.
- techdragon 11y ago
  We sometimes call this "pivoting your business" ;-)

ThrustVectoring 11y ago

There's another one I've seen: climbing the wrong hill. Consider a news website doing A/B testing on headlines, and optimizing for getting more clicks on an individual headline compared to other candidates in the slot. The problem is conflating "people want to click on a headline after visiting our site" with "people want to visit our site", and optimizing for the former at the expense of the latter. The end result is clickbait headlines that alienate visitors while they fight to grab attention.

abraae 11y ago
A "wrong hill" example in our industry (recruitment software) is a focus on getting as many as possible candidates to find and apply for jobs. If I sell widgets, then yes, I want as many as possible people to come and buy them, and I really only care about the colour of their money. However when as a widget company I am advertising jobs, I really want only qualified candidates to apply. Otherwise I am drowning in people I just need to sort through, and reject - and when I reject them, they may stop buying my widgets. In this situation, bounce rates (e.g. people leaving the site after viewing a job) and a lot of other common metrics are meaningless without far more context.
- danieltillett 11y ago
  The equivalent in widget selling is time-wasting leads. When you are advertising a job you are looking for leads and unqualified people are just bad leads. Having said this there is little you can do about this problem since any well advertised position will attract the unqualified since the cost is so low to just mass-blast your cv. I have always wanted to have a fee (say $500) to apply for a position and refunded $1000 to the candidates that actually have the skills they claim and which match the job requirements. Real candidates would be rewarded for applying (easy $500) paid for by all the time wasters and dunning-kruger types. While this appeals to me I doubt this idea would be viable in the real world :)
  - ThrustVectoring 11y ago
    I think what you're looking for is something like a preliminary interview round that is cheap for you to administer, combined with a financial reward for those you select to go to the next level of the process. For incentive reasons, the budget for this must come out of the hiring company's recruiting budget. Otherwise, it's a great way for unethical folks to extract money out of the applicant pool. An alternative way is to set up some sort of competition with prizes, and then recruit from those who do well at it.

cialowicz 11y ago

This is a great little read, and really resonated with me. Like the author mentioned, there is some value in A/B testing and an incremental approach to product development. But too often organizations can become obsessed with this path forward, and continuing with the same approach becomes like trying to squeeze blood from a stone -- there are just no more significant gains to be had. Unfortunately the company I'm at right now feels like this. The product team and CEO are obsessed with A/B testing, and don't have a more holistic approach to product development. This means that our competitors are out-innovating us in certain areas, and it's disappointing to watch.

avn2109 11y ago
>> "...don't have a more holistic approach to product development..." Interestingly, the article's author works at Twitter.

sopooneo 11y ago

I often wonder if an agile development approach that only plans in the range of very short iterations could run into the same problem.

namenotrequired 11y ago

> Bold changes require lots of resources. Maybe it’s mostly the success-bound companies who have enough resources to afford the bold changes. And even when they do get the causality right, maybe big changes in production are likely to cause big changes in outcome - providing both greater potential reward and greater risk of losing everything. Any sufficiently lucky contrarian is indistinguishable from genius. Edit for clarity

cshimmin 11y ago

Those javascript animations had my CPU running at 100% while I was reading the article. I'm not sure I like this new web... can't we just go back to GIFs for simple stuff like that?

restalis 11y ago
The only reason GIFs are a better solution for you (i.e. not taking 100% of the CPU resource) is because it's about a quality GIF render. What you really need is quality of the software that powers the alternative solutions.

llogiq 11y ago

Note that this has little to do with hill-climbing algorithms and much with A/B testing. Though there is an overlap in the problems both face. I wonder if algorithmic solutions like randome restart or simulated annealing could improve the outcome of A/B testing, too.