2 ms·

I've been itching to try this for some time. I would appreciate any war stories HNers could share.

by teej 17y ago

dawie 17y ago
Like I said in my blog post, it worked pretty good for me. From here, I basically went on and created a Minimal Viable product.
patio11 17y ago
Email addresses don't covert on a 1 to 1 basis to sales. (If. Only.) After the autoresponder runs out (6 days in my case) they've only got only a moderately higher chance of conversion than my average trial user. (~5% vs. 2.8%) I really, really don't like the Put Up Fake AdWords Ad, See Who Gets Suckered minimum viable product. Aside from misleading customers and hurting the AdWords ecosystem (they don't actually get offered what they clicked to get offered, which Google will come down on you like a ton of bricks for if they detect, incidentally), it doesn't test your product. It tests your ability to write AdWords copy. I could get 14% conversion to stabbing toothpicks under your fingernails, but what does that tell you? The world has weird people in it, and I'm good at qualifying non-weird people out with my ad text and keyword selection? Whee? Similarly, if I delivered an AdWords campaign that was awful (low CTRs, low conversions), would that tell you the product is awful? Because I've done that before, too -- the first draft of my AdWords campaign for Bingo Card Creator was terrible. It took me almost a year before I figured out how to write AdWords decently. That was not an indication that the product was terrible (I sell it quite successfully), it was just an indication that I needed to adjust to a new way of copywriting. If it can't tell you if an idea is good, and you can't trust it if it says the idea is bad -- what is the point, again?
- tocomment 17y ago
  Those are really good points. Is there another type of market testing you recommend?
- sgk284 17y ago
  I think you've hit on some good points, but missed a big one. Running ad words drives traffic to a site that currently has no traffic, better yet that traffic has shown some interest in the product you're claiming to have. If some of those people try to buy (or express interest in) this non-existent product, that says something. You're not measuring how much traffic ad words brings in. You're using ad words as a traffic generator (ideally generating interested traffic), and of this traffic that comes to your site, you can determine if people are clicking through. For ~$100 you can determine if there is any interest at all. This can be powerful for people who really have no idea what the market will do. It's not perfect, but not useless.
  - patio11 17y ago
    This can be powerful for people who really have no idea what the market will do. These people should take their hands off the keyboard and not put it back on until they have spoken to at least one Real Person who has the problem that they are trying to solve. After they have done this, they will know if there is any interest at all. This produces actionable insights: if they tell you they have a problem, then at least some people have the problem. If they tell you that the industry's favorite way for solving the problem costs $1,000, that immediately establishes that there is a viable market for solving the problem. Compare this to the AdWords campaign: can you tell me what numbers you can get that say "This is a good idea -- do this" and what numbers you can get that say "This is not a good idea -- don't do this"? No -- subjectively "good" results don't imply purchasing intent, subjectively "bad" results don't imply lack of it. You'll get noise -- very precise noise, noise you can measure down to the click, but still noise.
    - sgk284 17y ago
      Suppose I came up with an idea that I absolutely knew was brilliant, let's say edible shoelaces. Unfortunately these edible shoelaces aren't cheap to produce. I'm a one man shop and have almost no money. I make a fully functional website to sell this product, only I don't have the product yet. When people browse the site and have gone through the pricing page, etc... and finally click "buy" I simply tell them that we're out of stock. Now I want to see if people are actually trying to buy this thing, so I start marketing but have a limited budget. I throw $1000 at ad words. If I get 100 people trying to buy my edible shoelaces at $20 each then I know that maybe it's worth investing in manufacturing the product. How is this not useful data? Instead of going all out and spending money on manufacturing and a website, you only pay for the website and hold off on the manufacturing expense until you're confident it'll pay off.
- pbhjpbhj 17y ago
  Because I've done that before, too -- the first draft of my AdWords campaign for Bingo Card Creator was terrible. It took me almost a year before I figured out how to write AdWords decently. An SEO specialising in ad-words and PPC would have saved you a lot of time ... I thought all this SEO stuff was obvious! ;0)>
- netsp 17y ago
  A minimum viable product experiment is good at measuring whatever a minimum viable product experiment measures. You definitely can't make exact projections, but it is a piece of info that is useful. At the very least it can tell you not to build something. It might also be good to test several ideas with this method to decide which one to pursue. Again, they shouldn't be the only factor but all else equal, better fake conversion is better. It isn't an accurate projection, but it is info. It is also a chance to get an understanding of the adwords environment, click costs, etc.
- lupin_sansei 17y ago
  > If it can't tell you if an idea is good, and you can't trust it if it says the idea is bad -- what is the point, again? The point is not to prove the idea is good, but to prove that the idea isn't bad. In other words you can't prove that email addresses will convert to sales, but you can prove that no email addresses will convert to no sales. It's useful to know if your idea will produce no click-throughs, which will produce no sales.
jmathes 17y ago
I worked at a software company where we should have run an experiment like this. There's an upgrade on user accounts that we charged a one-time $10 for, and people buy it all the time. We decided to try it as a subscription. We spent a month getting the subscription code ready, then deployed it to some % of users, only to discover that nobody wanted it. What we should have done was build the front-end only, get people to buy subscriptions, and then just grant them the one-time permanent version.

I've been itching to try this for some time. I would appreciate any war stories HNers could share.

by teej 17y ago

I've been itching to try this for some time. I would appreciate any war stories HNers could share.

dawie 17y ago

Like I said in my blog post, it worked pretty good for me. From here, I basically went on and created a Minimal Viable product.

patio11 17y ago

Email addresses don't covert on a 1 to 1 basis to sales. (If. Only.) After the autoresponder runs out (6 days in my case) they've only got only a moderately higher chance of conversion than my average trial user. (~5% vs. 2.8%) I really, really don't like the Put Up Fake AdWords Ad, See Who Gets Suckered minimum viable product. Aside from misleading customers and hurting the AdWords ecosystem (they don't actually get offered what they clicked to get offered, which Google will come down on you like a ton of bricks for if they detect, incidentally), it doesn't test your product. It tests your ability to write AdWords copy. I could get 14% conversion to stabbing toothpicks under your fingernails, but what does that tell you? The world has weird people in it, and I'm good at qualifying non-weird people out with my ad text and keyword selection? Whee? Similarly, if I delivered an AdWords campaign that was awful (low CTRs, low conversions), would that tell you the product is awful? Because I've done that before, too -- the first draft of my AdWords campaign for Bingo Card Creator was terrible. It took me almost a year before I figured out how to write AdWords decently. That was not an indication that the product was terrible (I sell it quite successfully), it was just an indication that I needed to adjust to a new way of copywriting. If it can't tell you if an idea is good, and you can't trust it if it says the idea is bad -- what is the point, again?

tocomment 17y ago
Those are really good points. Is there another type of market testing you recommend?
sgk284 17y ago
I think you've hit on some good points, but missed a big one. Running ad words drives traffic to a site that currently has no traffic, better yet that traffic has shown some interest in the product you're claiming to have. If some of those people try to buy (or express interest in) this non-existent product, that says something. You're not measuring how much traffic ad words brings in. You're using ad words as a traffic generator (ideally generating interested traffic), and of this traffic that comes to your site, you can determine if people are clicking through. For ~$100 you can determine if there is any interest at all. This can be powerful for people who really have no idea what the market will do. It's not perfect, but not useless.
- patio11 17y ago
  This can be powerful for people who really have no idea what the market will do. These people should take their hands off the keyboard and not put it back on until they have spoken to at least one Real Person who has the problem that they are trying to solve. After they have done this, they will know if there is any interest at all. This produces actionable insights: if they tell you they have a problem, then at least some people have the problem. If they tell you that the industry's favorite way for solving the problem costs $1,000, that immediately establishes that there is a viable market for solving the problem. Compare this to the AdWords campaign: can you tell me what numbers you can get that say "This is a good idea -- do this" and what numbers you can get that say "This is not a good idea -- don't do this"? No -- subjectively "good" results don't imply purchasing intent, subjectively "bad" results don't imply lack of it. You'll get noise -- very precise noise, noise you can measure down to the click, but still noise.
  - sgk284 17y ago
    Suppose I came up with an idea that I absolutely knew was brilliant, let's say edible shoelaces. Unfortunately these edible shoelaces aren't cheap to produce. I'm a one man shop and have almost no money. I make a fully functional website to sell this product, only I don't have the product yet. When people browse the site and have gone through the pricing page, etc... and finally click "buy" I simply tell them that we're out of stock. Now I want to see if people are actually trying to buy this thing, so I start marketing but have a limited budget. I throw $1000 at ad words. If I get 100 people trying to buy my edible shoelaces at $20 each then I know that maybe it's worth investing in manufacturing the product. How is this not useful data? Instead of going all out and spending money on manufacturing and a website, you only pay for the website and hold off on the manufacturing expense until you're confident it'll pay off.
pbhjpbhj 17y ago
Because I've done that before, too -- the first draft of my AdWords campaign for Bingo Card Creator was terrible. It took me almost a year before I figured out how to write AdWords decently. An SEO specialising in ad-words and PPC would have saved you a lot of time ... I thought all this SEO stuff was obvious! ;0)>
netsp 17y ago
A minimum viable product experiment is good at measuring whatever a minimum viable product experiment measures. You definitely can't make exact projections, but it is a piece of info that is useful. At the very least it can tell you not to build something. It might also be good to test several ideas with this method to decide which one to pursue. Again, they shouldn't be the only factor but all else equal, better fake conversion is better. It isn't an accurate projection, but it is info. It is also a chance to get an understanding of the adwords environment, click costs, etc.
lupin_sansei 17y ago
> If it can't tell you if an idea is good, and you can't trust it if it says the idea is bad -- what is the point, again? The point is not to prove the idea is good, but to prove that the idea isn't bad. In other words you can't prove that email addresses will convert to sales, but you can prove that no email addresses will convert to no sales. It's useful to know if your idea will produce no click-throughs, which will produce no sales.

jmathes 17y ago

I worked at a software company where we should have run an experiment like this. There's an upgrade on user accounts that we charged a one-time $10 for, and people buy it all the time. We decided to try it as a subscription. We spent a month getting the subscription code ready, then deployed it to some % of users, only to discover that nobody wanted it. What we should have done was build the front-end only, get people to buy subscriptions, and then just grant them the one-time permanent version.