5 ms·

"The gist is that it’s tough to get unbiased feedback during customer validation." - I wish this was taught along with lean startup content. Too many times I ha

"The gist is that it’s tough to get unbiased feedback during customer validation." - I wish this was taught along with lean startup content. Too many times I have heard founders say they validated the idea when all they have done is gotten biased feedback while asking self-fulfilling questions.

ben_jones 7y ago
I mean, iterate on the product until it sells like hotcakes that's your validation. Then you'll spend the rest of your days trying to STOP the torrent of complaints, feature requests, etc. Better yet have actual experience in the industry that the product is targeting rather then expecting that industry to go out of its way to tell you what to build. Ideally you have orders for the product before you start building it, don't put all your eggs in if I build it they will come.
- matwood 7y ago
  > hotcakes that's your validation Exactly. The best validation is people paying for a product.
  - tschwimmer 7y ago
    I would go so far as to say it’s the only unambiguous validation you can get.
    - ambicapter 7y ago
      Not really that unambiguous depending on the market and/or product. You could make the argument that these HIV drugs are miraculously helpful! As you jack up the price. People keep buying it at any price, must means its fantastically effective, right?
      - TeMPOraL 7y ago
        Even in less extreme cases, it may be just that you have extremely good salespeople / marketing team. With good enough sales, you can sell any kind of garbage, and in many markets it might be unlikely that complaints of existing users will reach or discourage new ones.
AznHisoka 7y ago
Yep, it is extremely hard. Lean Startup makes it sound trivial when it's almost impossible. Almost nobody is going to clear up their day to talk to you about their problems, answer your questions, give you feedback, be your guinea pig, etc. The ideal way to validate it is to be so immerse in the industry/problem such that you understand most of the pain points. Do the job that your ideal customer is doing. Attend the conferences they're going to. Visit the forums they're visiting. Read the websites they're visiting. Analyze the reviews they're posting in TrustRadius/G2Crowd. Look at their Linkedin profiles to see what their day to day involves.
- mooreds 7y ago
  > The ideal way to validate it is to be so immerse in the industry/problem such that you understand most of the pain points. Do the job that your ideal customer is doing. I think this is a great reason to have a co-founder. I co-founded a company with a domain expert, and her ability to get in front of and understand our target audience was way more important to our startup success than my development ability (and the equity shares reflected that). I mean, we needed each other, but she was far more unique than I was. The alternative for a developer is you immerse yourself into a different domain, or you build tools for developers. Both of which are good paths, but the former is far more time commitment and latter is more common (and thus more competitive).
- davidivadavid 7y ago
  You don't need them to clear up a whole day, and people are surprisingly willing to talk about their job and their frustrations.
- rpedela 7y ago
  I agree overall but it's not that hard to get people to talk to you if done a certain way. If you are trying to sell them while talking, yes that will be hard/impossible. However if you approach it as learning from experts, then I find people are much more receptive because most people like to help and feel important/useful. I always say I want to learn from them and offer to pay for their time which shows them I understand they are busy. I still get some NOs but that's okay. BTW I've never had someone want me to pay them, but I would if they did.
jessmartin 7y ago
This was the post that made the lights come on for me: https://customerdevlabs.com/2013/11/05/how-i-interview-customers/ https://customerdevlabs.com/2013/11/05/how-i-interview-custo... After doing 10-20 of those interviews, I got pretty good at coming up with an internal "pain scale" - where this particular problem sits in terms of overall pain the customer is experiencing, and thus how much they are willing to commit to solving the problem.
iovrthoughtthis 7y ago
I say it every time, to every founding team I meet. Read “The Mum Test”[1]. It will make you aware that you to get unbiased feedback you need to remove your identity from the idea so that people aren’t nice to protect you. [1] http://momtestbook.com/ http://momtestbook.com/
- davidivadavid 7y ago
  I also keep recommending this book to everyone. The TL;DR is: NEVER ask questions about hypotheticals. Ask people about what they do, how much they pay, how much time it takes, and so on. Facts, not answers to "Would you use this if...?"
  - sdenton4 7y ago
    Indeed! One of the things I found frustrating about the article is that he talks about people 'lying' constantly... when if fact he had just asked meaningless questions.
    - SlowRobotAhead 7y ago
      He used the word lying. They didn't know they were lying. They were miswanting. And it's a HUGE glossed over fact when talking to customers.
      - davidivadavid 7y ago
        Yes, it's the customer development version of the XY problem. https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem
  - jnwatson 7y ago
    Interestingly enough, that’s also a principle of behavioral interviewing. Hypothetical questions allow the bullshitters to shine, but specific targeted lines of question about what the interviewee did in a specific situation is frequently enlightening.
wpietri 7y ago
Huh. I think this is often taught. Indeed, I think it's at the core of the LS approach. The build-measure-learn loop is the heart of the process, meaning that whatever ideas you have, you test them as soon as possible with customer behavior. As an example, I mentored at Lean Startup Machine. We'd have them form up into teams around ideas. The first thing they'd ask for was verbal feedback. But they were quickly supposed to move to increasing levels of demonstration of commitment. One team, for example, had the notion that restaurants needed an easier way to get insight into customer experiences. They went out and found restaurant managers, and first confirmed that they had the hypothesized problem. But they quickly moved beyond words, asking if the managers would be willing to let them trial the system in the restaurant. Since enough of them said yes, they build something simple and put those little printed cardboard triangles on a bunch of tables. They then hacked together reports for the managers based on the data collected. The managers liked the reports. They then asked the managers how many would be willing to pay, right then, some modest monthly charge for ongoing use of the system. If I recall rightly, this team discovered that when actual money was on the line, the reports were just a nice-to-have. In that situation, it's still reasonable to persevere, as you may have hypotheses about what really will get them to cross the line. But it's also reasonable to kill the idea. There's a big difference, though, between what is taught and what people do. I was an early adopter of both the Agile movement (XP in particular) and Lean Startup approaches, and in both cases I'm kinda horrified by what it turned into in practice. With the Agile movement, it was quickly captured by companies who wanted to feel like they were doing something different while still retaining most of their top-down pathologies and high-debt code bases. With the Lean Startup approach it's harder to say how it turned out, as early-stage startups rarely produce reports like this. But my general impression is that, as with Agile, people pick the parts that seem fun and easy, and avoid the hard parts. I think this was made worse by the flood of VC money, especially around Uber-for-X startups. Who wants a method that will kill your probably-bad idea early if you can get tens or hundreds of millions? The bad idea might work out or you might thrash your way to a good one. And if it still failed? Well, now you're a seasoned founder of a venture-backed startup. By some measures, although not mine, that's way better than having a quiet, cheap failure.

ben_jones 7y ago

I mean, iterate on the product until it sells like hotcakes that's your validation. Then you'll spend the rest of your days trying to STOP the torrent of complaints, feature requests, etc. Better yet have actual experience in the industry that the product is targeting rather then expecting that industry to go out of its way to tell you what to build. Ideally you have orders for the product before you start building it, don't put all your eggs in if I build it they will come.

matwood 7y ago
> hotcakes that's your validation Exactly. The best validation is people paying for a product.
- tschwimmer 7y ago
  I would go so far as to say it’s the only unambiguous validation you can get.
  - ambicapter 7y ago
    Not really that unambiguous depending on the market and/or product. You could make the argument that these HIV drugs are miraculously helpful! As you jack up the price. People keep buying it at any price, must means its fantastically effective, right?
    - TeMPOraL 7y ago
      Even in less extreme cases, it may be just that you have extremely good salespeople / marketing team. With good enough sales, you can sell any kind of garbage, and in many markets it might be unlikely that complaints of existing users will reach or discourage new ones.

AznHisoka 7y ago

Yep, it is extremely hard. Lean Startup makes it sound trivial when it's almost impossible. Almost nobody is going to clear up their day to talk to you about their problems, answer your questions, give you feedback, be your guinea pig, etc. The ideal way to validate it is to be so immerse in the industry/problem such that you understand most of the pain points. Do the job that your ideal customer is doing. Attend the conferences they're going to. Visit the forums they're visiting. Read the websites they're visiting. Analyze the reviews they're posting in TrustRadius/G2Crowd. Look at their Linkedin profiles to see what their day to day involves.

mooreds 7y ago
> The ideal way to validate it is to be so immerse in the industry/problem such that you understand most of the pain points. Do the job that your ideal customer is doing. I think this is a great reason to have a co-founder. I co-founded a company with a domain expert, and her ability to get in front of and understand our target audience was way more important to our startup success than my development ability (and the equity shares reflected that). I mean, we needed each other, but she was far more unique than I was. The alternative for a developer is you immerse yourself into a different domain, or you build tools for developers. Both of which are good paths, but the former is far more time commitment and latter is more common (and thus more competitive).
davidivadavid 7y ago
You don't need them to clear up a whole day, and people are surprisingly willing to talk about their job and their frustrations.
rpedela 7y ago
I agree overall but it's not that hard to get people to talk to you if done a certain way. If you are trying to sell them while talking, yes that will be hard/impossible. However if you approach it as learning from experts, then I find people are much more receptive because most people like to help and feel important/useful. I always say I want to learn from them and offer to pay for their time which shows them I understand they are busy. I still get some NOs but that's okay. BTW I've never had someone want me to pay them, but I would if they did.

jessmartin 7y ago

This was the post that made the lights come on for me: https://customerdevlabs.com/2013/11/05/how-i-interview-customers/ https://customerdevlabs.com/2013/11/05/how-i-interview-custo... After doing 10-20 of those interviews, I got pretty good at coming up with an internal "pain scale" - where this particular problem sits in terms of overall pain the customer is experiencing, and thus how much they are willing to commit to solving the problem.

iovrthoughtthis 7y ago

I say it every time, to every founding team I meet. Read “The Mum Test”[1]. It will make you aware that you to get unbiased feedback you need to remove your identity from the idea so that people aren’t nice to protect you. [1] http://momtestbook.com/ http://momtestbook.com/

davidivadavid 7y ago
I also keep recommending this book to everyone. The TL;DR is: NEVER ask questions about hypotheticals. Ask people about what they do, how much they pay, how much time it takes, and so on. Facts, not answers to "Would you use this if...?"
- sdenton4 7y ago
  Indeed! One of the things I found frustrating about the article is that he talks about people 'lying' constantly... when if fact he had just asked meaningless questions.
  - SlowRobotAhead 7y ago
    He used the word lying. They didn't know they were lying. They were miswanting. And it's a HUGE glossed over fact when talking to customers.
    - davidivadavid 7y ago
      Yes, it's the customer development version of the XY problem. https://en.wikipedia.org/wiki/XY_problem https://en.wikipedia.org/wiki/XY_problem
- jnwatson 7y ago
  Interestingly enough, that’s also a principle of behavioral interviewing. Hypothetical questions allow the bullshitters to shine, but specific targeted lines of question about what the interviewee did in a specific situation is frequently enlightening.

wpietri 7y ago

Huh. I think this is often taught. Indeed, I think it's at the core of the LS approach. The build-measure-learn loop is the heart of the process, meaning that whatever ideas you have, you test them as soon as possible with customer behavior. As an example, I mentored at Lean Startup Machine. We'd have them form up into teams around ideas. The first thing they'd ask for was verbal feedback. But they were quickly supposed to move to increasing levels of demonstration of commitment. One team, for example, had the notion that restaurants needed an easier way to get insight into customer experiences. They went out and found restaurant managers, and first confirmed that they had the hypothesized problem. But they quickly moved beyond words, asking if the managers would be willing to let them trial the system in the restaurant. Since enough of them said yes, they build something simple and put those little printed cardboard triangles on a bunch of tables. They then hacked together reports for the managers based on the data collected. The managers liked the reports. They then asked the managers how many would be willing to pay, right then, some modest monthly charge for ongoing use of the system. If I recall rightly, this team discovered that when actual money was on the line, the reports were just a nice-to-have. In that situation, it's still reasonable to persevere, as you may have hypotheses about what really will get them to cross the line. But it's also reasonable to kill the idea. There's a big difference, though, between what is taught and what people do. I was an early adopter of both the Agile movement (XP in particular) and Lean Startup approaches, and in both cases I'm kinda horrified by what it turned into in practice. With the Agile movement, it was quickly captured by companies who wanted to feel like they were doing something different while still retaining most of their top-down pathologies and high-debt code bases. With the Lean Startup approach it's harder to say how it turned out, as early-stage startups rarely produce reports like this. But my general impression is that, as with Agile, people pick the parts that seem fun and easy, and avoid the hard parts. I think this was made worse by the flood of VC money, especially around Uber-for-X startups. Who wants a method that will kill your probably-bad idea early if you can get tens or hundreds of millions? The bad idea might work out or you might thrash your way to a good one. And if it still failed? Well, now you're a seasoned founder of a venture-backed startup. By some measures, although not mine, that's way better than having a quiet, cheap failure.