6 ms·

I’ve heard this a couple times but it still makes no sense to me. How do you test your product hypothesis without a product? You’re essentially testing the viab

I’ve heard this a couple times but it still makes no sense to me. How do you test your product hypothesis without a product? You’re essentially testing the viability of an idea with what you’re proposing, which is vastly different than having an actual product with features that you can iterate on when given feedback. I think for very niche categories, a landing page might be the only thing you need but I honestly believe that if you’re pitching a product with features, people want to see that product, not a page with text.

rfrey 6y ago
I believe you are entirely correct: it is testing an idea rather than testing and iterating on features. That's the point: people who love programming might skip directly to testing features without testing the idea. Testing the idea is essential and can be done 10x faster and cheaper than testing the first feature.
- AlchemistCamp 6y ago
  Okay, let's take Roblox as an example since they're about to IPO. They've clearly made an incredible product and built a developer ecosystem around it. When they were starting several years ago, how would have tested "the idea"?
  - cinntaile 6y ago
    That's just survivorship bias though. A lot more ready-coded ideas fail which means a lot more wasted time (if making money is the point). I agree that not all ideas can be tested that way, some ideas just have to be experienced. For the ones that can be tested in their idea phase, it seems like a good way to prevent doing too much when the interest just isn't there. At the same time you'll probably miss out on some ideas that need a bit of a runway first, but you're minimizing for downside risk instead of maximizing for upside potential.
    - AlchemistCamp 6y ago
      How is my question survivorship bias? Other than the sense that any question asked is one of a multitude of candidate questions that "survived" the decision process and made it out into the world, I don't see how asking how a company's "idea" may have been tested is survivorship bias.
      - cinntaile 6y ago
        You are right, that sentence and the one after that are irrelevant. My mistake!
        AlchemistCamp 6y ago
        No worries!
  - fny 6y ago
    They could have put out a site with videos of the product and experience akin to what Nikola did with their trucks. Vaporware is an extremely common tactic in sales.
  - tbran 6y ago
    Instead of asking, "How do I test Roblox?", I'd ask: "Is my idea testable with the time/runway I have?" Assuming the context of a solo dev that just wants to make money: info products, simple apis, or simple apps should be easier to test. I think that doing lots of small projects and launching them is probably better than working on "the one thing" - until one of them becomes "the one thing". This gives you more opportunity to test ideas and test marketing (which is probably more important than your idea). The canonical example of this is probably Pieter Levels [1]. Let's call this the "Test with Teeny MVP" [2] method as a opposed to the "Test with Landing Page" method. Important note - I suspect that testing with a teeny MVP is easier than a landing page when you don't have an audience. Interesting tweet on that from Rob Walling [3]: "Out of nearly 1600 applicants to @tinyseedfund we chose 23 exceptional companies to fund. Of those 23, one (maybe two if you stretch) built an audience before launching their SaaS. Audience helps, but so much less in SaaS. I've been beating this drum since 2012." [1]: https://levels.io/12-startups-12-months/ https://levels.io/12-startups-12-months/ [2]: I'm sure that someone will point out that "Teeny" is redundant here. :) [3]: https://twitter.com/robwalling/status/1306591312498405376 https://twitter.com/robwalling/status/1306591312498405376
    - AlchemistCamp 6y ago
      > Instead of asking, "How do I test Roblox?", I'd ask: > "Is my idea testable with the time/runway I have?" I'm already ramen profitable with what I'm doing. The vast majority of my ideas are not ideas for new businesses, though a few are. The idea I'm testing in this thread is, "looking at successful businesses, how would rigid Lean Startup practices have worked in their early days?" Tinyseed isn't terribly interesting to me in this context since they have a very narrow SaaS focus, none of them have had an IPO and Rob has a large podcast he uses to promote the companies. It's a great strategy on his part, but it makes it much harder to figure to know how much the success of companies they invest in is due to their ideas. Something similar could be said of Jason Calicanis's launch.co, though I bet it will have some massive wins in the next few years.
  - chris11 6y ago
    Roblox's history actually goes back decades. In 1989 Dave Baszucki cofounded Knowledge Revolution and created Interactive Physics, an educational physics simulator. Seeing how students used the software inspired him to cofound Roblox in 2004.
  - AznHisoka 6y ago
    Lots of market research, surveys, and analysis of market trends. And then you build and hope that analysis translates to customers. Most people here have the right intention on testing your idea/concept before building, but they forget the world is 100x more competitive than it was 10 years ago. All the low hanging fruit ideas are gone. Yes you can obviously test some ideas easily but most of them were executed already. To succeed now, you need to tackle the “high hanging” fruit ideas/problems and a lot of those simply involve more complexity/building and are not as easily testable via a prototype. Instead you need to invest more into market research before you build a huge chunk of it (while continually doing more market research along the way)
KarimDaghari 6y ago
Not GP, but here's my 2c: > How do you test your product hypothesis without a product? You can't. Actually... it depends on how you define a product. The way I think of it is: a product is merely a vehicle for the added value. With that understanding it becomes easier to decouple the value from the actual product. Therefore the MVP is no longer a stripped down version of the product but rather the simplest tool you can build to provide [that] value.
fuball63 6y ago
This is right up there with the "to gain users, build a product that users like by talking to users" paradox.
- scollet 6y ago
  It seems strange to me that you would build without already having a problem to solve.
andi999 6y ago
I think I read about it in Eric Ries 'Lean Startup' or so. He narrated his experience of building a product, making a signup page for that product. Nobody pressed the sign-up button. In this case the signup page with the button (and then transfering to 'get notified when it is ready') would have been enough.
WJW 6y ago
I mean, you just make up features and ask customers if they want them before writing a line of code. Check out https://buffer.com/resources/idea-to-paying-customers-in-7-weeks-how-we-did-it/ https://buffer.com/resources/idea-to-paying-customers-in-7-w... from buffer.com for an example. It was literally the landing and pricing pages first, functionality came a few weeks later and only after people were clicking on the (non-functional) pricing plans because they wanted to buy the product.
- halfmatthalfcat 6y ago
  How do you translate product efficacy by an anonymous person clicking a buy link on a pricing page? All you are doing is gauging this ephemeral, uninformed click which has no direct correlation to what the product will actually _do_. It seems more of a crap shoot to gauge interest off of random clicks on a buy page versus having an actual product with actual user feedback that is actionable.
  - WJW 6y ago
    Well, obviously. But there is more to it than just those two in isolation. First off, one anonymous person clicking a link is just noise. But if you have hundreds clicking a link per day that gives you a much better indication you might be onto something. Further, how much time do you spend on making a basic landing and pricing page? Two hours? Four maybe? The original comment was talking about programmers working for 6+ months on a MVP product before showing it to people. You could launch hundreds of landing pages in that time, only limited by how many ideas you have. Even though the chance of a "hit" per landing page is probably lower, the sheer volume can more than make up for it. The landing plus pricing page model also immediately informs you how much people are actually willing to spend on it, which (if higher than zero) is a much higher indicator for product success than "I think this is a super cool feature". Finally, I notice that you are talking about "product efficacy" while the original comment is interested in making money. The two are related but not the same, since there are tons of crappy products making a boatload of profit (comcast anyone?) and also tons of great products that struggle to make money (see the ongoing troubles of open source developers on this very forum). Either goal is fine, but optimizing for one while thinking you are then also optimizing for the other is a recipe for disappointment.
schwartzworld 6y ago
I did a little focus group work years ago. Companies pay people to listen to a pitch or watch a demo and give them feedback.
gamerDude 6y ago
At the end of the road, you still have to make a landing page showcasing what your product does, what problem it solves and why people should sign up. Usually that landing page doesn't actually have any features, just shows people what features they will get. So build the landing page first. Showcase the features you are going to build and see if you can get people to sign up.
PragmaticPulp 6y ago
It's a false dichotomy. In the real world you can't fully test your product hypothesis before building anything, but you also can't build a complete product before testing the market. Instead, it needs to be an iterative process. Build a little bit of product, tease it to the market, gauge interest, engage with customers, adjust trajectory as necessary, and continue to iterate. Testing your business hypthesis before building anything is prone to generating false signal. Some times your customers don't realize they need what you're building until they see it and use it. Many times your customers will praise an idea, right up until you ask them to use it or pay for it. A good example might be the recruiting company started by a famous HN commenter around 2015 that tried to use programming games as a recruiting filter. The idea received huge amounts of praise on HN and large numbers of waiting list signups before they released anything. The demand and hypothesis appeared to be validated. Then they switched to the other extreme, building enormously complex systems for years without actually selling anything to their customers. Eventually they shut down because they realized the market didn't actually want what they were building. The initial signal was misleading, but going heads-down to build a product according to the initial signal was also misleading. A better approach would have been to start recruiting up front and slowly iterate on improving it with programming games, rather than going off in the weeds to build a product that no one actually wanted.
- mushbino 6y ago
  This is not true. There are many many mockup tools that you can use to test without building anything. It's a good way to separate the technology decisions from the business decisions.
  - PragmaticPulp 6y ago
    Mockups are part of the iterative process. Building a mockup to test is part of what I was describing. The point is that you can't simply make a mockup, assume it proves your hypothesis, and then go heads-down building the exact product shown in the mockups for 6-24 months and expect great results. Instead, you need to iterate both the mockups and the product in parallel, engaging with the market along the way. As in the example I provided, sometimes customers will claim to love your mockups but then decline to pay for the product when it arrives. It's one of the first things every product manager learns in the real world. > It's a good way to separate the technology decisions from the business decisions. Trying to separate technology from the business decisions is a mistake. At the start of the process, you need technology, product, sales, and business to be tightly intertwined. Dividing the labor and sending different roles in different directions is a mistake at an early stage company. You need to get everyone working together to iterate over and over again.

rfrey 6y ago

I believe you are entirely correct: it is testing an idea rather than testing and iterating on features. That's the point: people who love programming might skip directly to testing features without testing the idea. Testing the idea is essential and can be done 10x faster and cheaper than testing the first feature.

AlchemistCamp 6y ago
Okay, let's take Roblox as an example since they're about to IPO. They've clearly made an incredible product and built a developer ecosystem around it. When they were starting several years ago, how would have tested "the idea"?
- cinntaile 6y ago
  That's just survivorship bias though. A lot more ready-coded ideas fail which means a lot more wasted time (if making money is the point). I agree that not all ideas can be tested that way, some ideas just have to be experienced. For the ones that can be tested in their idea phase, it seems like a good way to prevent doing too much when the interest just isn't there. At the same time you'll probably miss out on some ideas that need a bit of a runway first, but you're minimizing for downside risk instead of maximizing for upside potential.
  - AlchemistCamp 6y ago
    How is my question survivorship bias? Other than the sense that any question asked is one of a multitude of candidate questions that "survived" the decision process and made it out into the world, I don't see how asking how a company's "idea" may have been tested is survivorship bias.
    - cinntaile 6y ago
      You are right, that sentence and the one after that are irrelevant. My mistake!
      - AlchemistCamp 6y ago
        No worries!
- fny 6y ago
  They could have put out a site with videos of the product and experience akin to what Nikola did with their trucks. Vaporware is an extremely common tactic in sales.
- tbran 6y ago
  Instead of asking, "How do I test Roblox?", I'd ask: "Is my idea testable with the time/runway I have?" Assuming the context of a solo dev that just wants to make money: info products, simple apis, or simple apps should be easier to test. I think that doing lots of small projects and launching them is probably better than working on "the one thing" - until one of them becomes "the one thing". This gives you more opportunity to test ideas and test marketing (which is probably more important than your idea). The canonical example of this is probably Pieter Levels [1]. Let's call this the "Test with Teeny MVP" [2] method as a opposed to the "Test with Landing Page" method. Important note - I suspect that testing with a teeny MVP is easier than a landing page when you don't have an audience. Interesting tweet on that from Rob Walling [3]: "Out of nearly 1600 applicants to @tinyseedfund we chose 23 exceptional companies to fund. Of those 23, one (maybe two if you stretch) built an audience before launching their SaaS. Audience helps, but so much less in SaaS. I've been beating this drum since 2012." [1]: https://levels.io/12-startups-12-months/ https://levels.io/12-startups-12-months/ [2]: I'm sure that someone will point out that "Teeny" is redundant here. :) [3]: https://twitter.com/robwalling/status/1306591312498405376 https://twitter.com/robwalling/status/1306591312498405376
  - AlchemistCamp 6y ago
    > Instead of asking, "How do I test Roblox?", I'd ask: > "Is my idea testable with the time/runway I have?" I'm already ramen profitable with what I'm doing. The vast majority of my ideas are not ideas for new businesses, though a few are. The idea I'm testing in this thread is, "looking at successful businesses, how would rigid Lean Startup practices have worked in their early days?" Tinyseed isn't terribly interesting to me in this context since they have a very narrow SaaS focus, none of them have had an IPO and Rob has a large podcast he uses to promote the companies. It's a great strategy on his part, but it makes it much harder to figure to know how much the success of companies they invest in is due to their ideas. Something similar could be said of Jason Calicanis's launch.co, though I bet it will have some massive wins in the next few years.
- chris11 6y ago
  Roblox's history actually goes back decades. In 1989 Dave Baszucki cofounded Knowledge Revolution and created Interactive Physics, an educational physics simulator. Seeing how students used the software inspired him to cofound Roblox in 2004.
- AznHisoka 6y ago
  Lots of market research, surveys, and analysis of market trends. And then you build and hope that analysis translates to customers. Most people here have the right intention on testing your idea/concept before building, but they forget the world is 100x more competitive than it was 10 years ago. All the low hanging fruit ideas are gone. Yes you can obviously test some ideas easily but most of them were executed already. To succeed now, you need to tackle the “high hanging” fruit ideas/problems and a lot of those simply involve more complexity/building and are not as easily testable via a prototype. Instead you need to invest more into market research before you build a huge chunk of it (while continually doing more market research along the way)

KarimDaghari 6y ago

Not GP, but here's my 2c: > How do you test your product hypothesis without a product? You can't. Actually... it depends on how you define a product. The way I think of it is: a product is merely a vehicle for the added value. With that understanding it becomes easier to decouple the value from the actual product. Therefore the MVP is no longer a stripped down version of the product but rather the simplest tool you can build to provide [that] value.

fuball63 6y ago

This is right up there with the "to gain users, build a product that users like by talking to users" paradox.

scollet 6y ago
It seems strange to me that you would build without already having a problem to solve.

andi999 6y ago

I think I read about it in Eric Ries 'Lean Startup' or so. He narrated his experience of building a product, making a signup page for that product. Nobody pressed the sign-up button. In this case the signup page with the button (and then transfering to 'get notified when it is ready') would have been enough.

WJW 6y ago

I mean, you just make up features and ask customers if they want them before writing a line of code. Check out https://buffer.com/resources/idea-to-paying-customers-in-7-weeks-how-we-did-it/ https://buffer.com/resources/idea-to-paying-customers-in-7-w... from buffer.com for an example. It was literally the landing and pricing pages first, functionality came a few weeks later and only after people were clicking on the (non-functional) pricing plans because they wanted to buy the product.

halfmatthalfcat 6y ago
How do you translate product efficacy by an anonymous person clicking a buy link on a pricing page? All you are doing is gauging this ephemeral, uninformed click which has no direct correlation to what the product will actually _do_. It seems more of a crap shoot to gauge interest off of random clicks on a buy page versus having an actual product with actual user feedback that is actionable.
- WJW 6y ago
  Well, obviously. But there is more to it than just those two in isolation. First off, one anonymous person clicking a link is just noise. But if you have hundreds clicking a link per day that gives you a much better indication you might be onto something. Further, how much time do you spend on making a basic landing and pricing page? Two hours? Four maybe? The original comment was talking about programmers working for 6+ months on a MVP product before showing it to people. You could launch hundreds of landing pages in that time, only limited by how many ideas you have. Even though the chance of a "hit" per landing page is probably lower, the sheer volume can more than make up for it. The landing plus pricing page model also immediately informs you how much people are actually willing to spend on it, which (if higher than zero) is a much higher indicator for product success than "I think this is a super cool feature". Finally, I notice that you are talking about "product efficacy" while the original comment is interested in making money. The two are related but not the same, since there are tons of crappy products making a boatload of profit (comcast anyone?) and also tons of great products that struggle to make money (see the ongoing troubles of open source developers on this very forum). Either goal is fine, but optimizing for one while thinking you are then also optimizing for the other is a recipe for disappointment.

schwartzworld 6y ago

I did a little focus group work years ago. Companies pay people to listen to a pitch or watch a demo and give them feedback.

gamerDude 6y ago

At the end of the road, you still have to make a landing page showcasing what your product does, what problem it solves and why people should sign up. Usually that landing page doesn't actually have any features, just shows people what features they will get. So build the landing page first. Showcase the features you are going to build and see if you can get people to sign up.

PragmaticPulp 6y ago

It's a false dichotomy. In the real world you can't fully test your product hypothesis before building anything, but you also can't build a complete product before testing the market. Instead, it needs to be an iterative process. Build a little bit of product, tease it to the market, gauge interest, engage with customers, adjust trajectory as necessary, and continue to iterate. Testing your business hypthesis before building anything is prone to generating false signal. Some times your customers don't realize they need what you're building until they see it and use it. Many times your customers will praise an idea, right up until you ask them to use it or pay for it. A good example might be the recruiting company started by a famous HN commenter around 2015 that tried to use programming games as a recruiting filter. The idea received huge amounts of praise on HN and large numbers of waiting list signups before they released anything. The demand and hypothesis appeared to be validated. Then they switched to the other extreme, building enormously complex systems for years without actually selling anything to their customers. Eventually they shut down because they realized the market didn't actually want what they were building. The initial signal was misleading, but going heads-down to build a product according to the initial signal was also misleading. A better approach would have been to start recruiting up front and slowly iterate on improving it with programming games, rather than going off in the weeds to build a product that no one actually wanted.

mushbino 6y ago
This is not true. There are many many mockup tools that you can use to test without building anything. It's a good way to separate the technology decisions from the business decisions.
- PragmaticPulp 6y ago
  Mockups are part of the iterative process. Building a mockup to test is part of what I was describing. The point is that you can't simply make a mockup, assume it proves your hypothesis, and then go heads-down building the exact product shown in the mockups for 6-24 months and expect great results. Instead, you need to iterate both the mockups and the product in parallel, engaging with the market along the way. As in the example I provided, sometimes customers will claim to love your mockups but then decline to pay for the product when it arrives. It's one of the first things every product manager learns in the real world. > It's a good way to separate the technology decisions from the business decisions. Trying to separate technology from the business decisions is a mistake. At the start of the process, you need technology, product, sales, and business to be tightly intertwined. Dividing the labor and sending different roles in different directions is a mistake at an early stage company. You need to get everyone working together to iterate over and over again.