8 ms·

How we validated our SaaS product without building it

120 points by bvanvugt 13y ago

patrickdavey 13y ago
What's the story with using logos for integration (SendGrid, AWS etc.) without actually supporting them? Is it not possible to get into legal issues there as it's quite an effective boost to the site to make it _look_ like you're associated with very reputable sites? Or.. just something to worry about when people come knocking? :)
- mrmch 13y ago
  if they come knocking... ;)
ultimatedelman 13y ago
this sounds like the most frustrating experience for a user ever
- bvanvugt 13y ago
  In a way that's a good thing - having a user work through the frustrating UI and still be interested in the product is amazing validation. Early adopters can self select.
  - minimaxir 13y ago
    It's an amazing biased validation, though. You won't be able to determine the usage patterns of the average user, and that's what's important.
    - ThomPete 13y ago
      How does that follow? If someone is willing to use it even when it's absurdly cumbersome then they are going to turn into customers. Once you build it you can iterate to find the best patterns.
  - mkr-hn 13y ago
    It makes them resent you and long for a competitor.
    - mrmch 13y ago
      Our experience was to the contrary. It's not shown in the blog post, but users were thanked for their feedback immediately. The reaction was overwhelmingly positive; people were excited that we were building something in this space.
    - kmnc 13y ago
      If a good one exists already that is. And if that is the case why are they checking broken demos? Why are they given their email addresses to broken demos? Either way you slice it, if something like this gains interest it certainly illustrates opportunities.
    - teej 13y ago
      I think it's natural to think that people will react this way. It turns out that people react completely differently. In my experience employing this tactic across several different verticals (SaaS, ecommerce, games), I have found people to react positively (yes, please make this!!) or neutral/mildly disappointed (darn, I wanted to use that). Never resentful.
- ThomPete 13y ago
  which sounds like the most valid form of market validation.
rubiquity 13y ago
I think this approach is very interesting and may give it a try on a product in the future. How did you go about directing people to the demo though?
- bvanvugt 13y ago
  Excellent question, and probably grounds for another blog post. We tried to use a variety of channels, including twitter, marketer communities, and even some google adwords.
KiwiCoder 13y ago
> We spent the next 24 hours getting as many people as possible to view and interact with the experiment OP - could you elaborate on how you got these 803 unique visitors within 24 hours?
- bvanvugt 13y ago
  Sure - that's probably worth another blog post too. We used a combination of twitter, adwords, marketing forums, and personal network.
  - lzecon 13y ago
    If you're relying heavily on your personal network I'd be worried that those 803 people were not true customers. Don't get me wrong, I think this is a great idea, but I'm wondering how you know that a 803 person sample size is big enough. Also - could/would people click on more than one button once they found out they were non-functional? It could be that some features would be best developed in tandem with others.
    - bvanvugt 13y ago
      We intentionally diversified the traffic as much as possible, exactly for this reason. Only a small portion of the traffic was from personal networks. Repeat clicks are definitely interesting. It makes sense that once a button was clicked and found to be non-functional, that may prevent future interaction. But that first click still holds value. It'd be interesting to pull stats on how many unique visitors generated those click events.
unfunco 13y ago
We knew that transactional email was a problem worth solving Really? Hasn't this been solved already, by many businesses (SendGrid, Mandrill, SES, MailJet) – I'm happy you've managed to build something and are seeing some success, but I don't think you're solving a problem that hasn't yet found a solution.
- EC1 13y ago
  "but I don't think you're solving a problem that hasn't yet found a solution." - eliminates like 95% of companies.
- bvanvugt 13y ago
  SendGrid/Mandrill/Mailgun are great! We've built a much needed layer on top of those services, that focuses on email content and customer experience.
- davidw 13y ago
  Altavista worked ok, but I'm glad Google came along. I think that your comment isn't a bad one, but should go more in depth. Some markets really are unassailable - others aren't. The reasons are many and sometimes complex, so analyzing one is often quite interesting.
  - chc 13y ago
    I don't think unfunco is suggesting that the market is unassailable. The point seemed to be that this problem wasn't one that needed to be solved (because there were already solutions), not that there isn't room for another solution-provider.
alexsilver 13y ago
We are going through the same approach with our current beta customers (and will be making our demos available to a wide audience in the future too) and I can attest to the validity of this approach. We make it very clear to our customers that this is a demo and their changes will go away if they refresh the page. We don't have pop-ups integrated since we collect emails on another page. I think event tracking should be sufficient but we'll have to see what the data says in a little while. We actually also use these semi-live demos instead of mock-ups (since we'll be implementing the same pages afterward anyway) since only one of us is proficient with Photoshop but everyone is comfortable with Bootstrap. It serves as a testing ground for various approaches for JS code too.
xg15 13y ago
The apparent origin of the idea, the "button to nowhere" that they linked to in the story is also a good read: https://medium.com/design-ux/77d911517318 https://medium.com/design-ux/77d911517318 From a user perspective, I think there are both good and bad ways how this technique can be used: I actually like the way Sendwithus used it - as part of a very very early "pre-alpha" version where you're not yet interested in providing a reliant service and just want to get a feel for the demand. As a user you're informed early-on that you're looking at a version in development - so you will not be surprised by missing or incomplete features and will maybe even motivated to provide additional feedback. The downside of this approach is of course that the data you're gathering is really only good for that - an early overview. Your test group will likely be very different from the actual people who will later use the service. The usage patterns will be as well. The second way to use the technique is how Nick Kishfy describes it in the "button to nowhere" article. In that approach, "buttons to nowhere" are not just inserted into an early preview/development version, but into the actual product as a continuous part of the development process. This would solve all of the above problems and would give you a much richer set of data over a much longer time span - but from a user perspective that would be the worst thing possible.
- mrmch 13y ago
  This is actually a good point -- the "button to nowhere" product is particularly valuable at an early stage. We're working on a followup story on how we do customer development/feedback with a established product.
DanielBMarkham 13y ago
Relevant similar story: I had this site for years, http://paycheck-stub.com http://paycheck-stub.com Developed it back when I was doing microsites, and I wasn't really sure what to do with it. My main question: why do people visit, anyway? I figured I'd know what to do with it once I figured out why people were there. So I did just what these guys did. I created a list of buttons down the page with all the possible things people visiting a site called paycheck stub might want. Hooked them all into Google Analytics. Paycheck templates? Fake paychecks? Loans? Payroll services? A job? Sample paychecks for school or presentations? Very interesting results. First, about 90% didn't want any of that. God knows why they were there. But out of the other 10%, I got a couple pieces of useful information. They were looking for payroll services. So I put up some banner ads. (I think I took down the Adsense. Can't remember) That, in turn, provided more ad revenue than simply using Adsense on the site. It also provided me with a list of topics to develop and some search terms to try to hit. So I developed those topics and tried to do better on those search terms. At one point I was almost ready to jump into online payroll. Fun times! Then Google killed my search ranking -- I wasn't paying for links or spamming. Beats me why. I got so frustrated with them I gave up. Still get quite a bit of traffic, though. But now, I'm sure, the traffic pattern is completely different. I wonder if, for static sites with a lot of content, this is something you should repeat every year or so? Like these guys, I found it a rewarding experience, both for my understanding of the site and for my readers. The reason I'm posting is that many times we view this as a step on the way to creating SaaS, but it can be just a great way to connect with your readers. I bet it's extremely applicable for a lot of blogs. Wish I could have figured out why those other 90% were there.
- mrmch 13y ago
  Interesting story Daniel -- there should be an "Optimizely for the button to nowhere" product!
  - DanielBMarkham 13y ago
    Hardest part was coming up with the questions. I still feel like I was missing something. Some important reason they were visiting. But when you have a site with traffic, you end up with tons of data and very little insight. The button thing moves the needle a little bit, and that's better than nothing. ADD: One other thing: there's also a minimum amount of traffic you need to make this thing work. 100 visits isn't enough. I'd imagine between 1-5K? Otherwise there's too much noise. Was that your experience as well?
alexeyza 13y ago
Awesome way to design a product. I wonder though, how the users reacted when they saw "dead end features". Were there any frustrated users when they saw it was mainly a "mock-up" web site ?
ForHackernews 13y ago
Does anyone else find this kind of pre-development marketing kind of scummy and deceptive? It's basically lying to potential customers: You're pretending to offer to sell a product that doesn't exist.
- mrmch 13y ago
  Perhaps if we were accepting pre-orders or requiring a credit card. A soft launch like this would be more comparable to a focus group for a traditional product launch.
  - ForHackernews 13y ago
    But you're not being honest and up-front about the fact that it's a focus group. If it was like "Click here to take a survey about what features you most want in an email service" that would be one thing, but that's not the messaging in your case.
    - petervandijck 13y ago
      Effective focus groups aren't up-front or honest either. Effective experiments don't tell the person what the experiment is.
      - ForHackernews 13y ago
        But they at least tell you that it is an experiment. Maybe we need human-subject ethics boards for marketers.
        troymcginnis 13y ago
        Just knowing that it's an experiment defeats the purpose. By getting honest responses and behaviors, it is easier to build a solution that people actually need or would actually use.
        13y ago
deleted 13y ago
[deleted]
allanlundhansen 13y ago
"We knew that transactional email was a problem worth solving" A story worth telling in it's own right I believe, when clamoring on about lean methodologies. Nice article would love the pre-phase lean story leading up to this stage.
- bvanvugt 13y ago
  Absolutely! Prior to this experiment we spent nearly two months talking to customers/partners of all sizes. The details of that period would make a great follow-up blog post.
nobodysfool 13y ago
I did that with my father's computer store website back in 1998. We had computer descriptions, a price, and a 'buy online' button. The button just had a hit counter on the page, with directions to the store. Every hit meant someone wanted to buy a computer through our website and couldn't. I showed him the data that we could double our sales just by making an online store, but he was not convinced. In any case, the cash flows of that business wouldn't have worked out for him. Still, I thought this was a widely known technique.
pbreit 13y ago
So did the API turn out to be a popular feature? I'm wondering if APIs prove interesting on the surface but not always in practice?
- bvanvugt 13y ago
  In our case it did! Our core product today is an API to send templated email.
redguava 13y ago
Are you sure the value you got from the metrics exceeds the bad will generated by annoying your potential customers? To me it's a lack of respect for your users and their time.
- mrmch 13y ago
  Some people seem to be missing that first screenshot; the landing page clearly started with "Start The Demo" (http://sendwithus.wpengine.com/wp-content/uploads/2014/03/Screenshot-2014-03-09-14.27.00-e1394434326899.png http://sendwithus.wpengine.com/wp-content/uploads/2014/03/Sc...)
  - redguava 13y ago
    Demo doesn't imply fake features. For me demo would be not my data, perhaps a sample set in there. I'd have no problem if it said "there's a bunch of features not yet implemented, we have the buttons there to gauge interest and we'll work on the most needed ones".
JimmyL 13y ago
I went though sendwithus zero as an unknowing participant in the experiment, and it sucked. At the time, the company I was working at was looking for a service exactly like this - a service where our app could just fling events, and then we'd do all the logic to control what emails got emitted on the basis of those events on the external service (and A/B test them). One of our "growth hacking" guys found SendWithUs somehow, and introduced us. We had a few calls with them - the service seemed like it had potential, and we got beta access to it...which was really sendwithus zero, but sending basic email. We got most of the way though setting up an integration with them, but every time we asked when the A/B testing framework - which was what sold us on the product - was coming online, we were told "it's coming soon, it's what we're working on right now." So we held on for a few more weeks, and gave them some time...but nothing came of it. I was skeptical at the time that they were actually building it, even moreso having read this. The positive side is that while we were being strung along by sendwithus, we found Vero (https://www.getvero.com/ https://www.getvero.com/). They're not perfect either, but we've been on them for 6-8 months, and they've done well for us. They're responsive to our issues, are willing to Skype or chat when we try and develop some sophisticated campaigns on their platform, and have (or so it seems) rejigged their roadmap a few times to add in small features that we ask for. ExactTarget is still a more full-featured package, but I'm happy with the price/value of Vero. So well done, guys. You've got an interesting story about how you bootstrapped your business, and you're evidently doing OK. I wish you the best, but I can tell you that you won't be getting my business (or my recommendation) - at the time I thought you were small and were actually on the cusp of delivering what you had promised us worked; now I see that we were just one more data point for a feature set that you never planned on having ready when you claimed.
- mrmch 13y ago
  Hey Jimmy, I'm sorry to hear that was your experience. I don't know when you found us, but it did take us time to roll out our A/B testing; we never intended to string anyone along. If you're willing to chat offline about your experience, please get in touch: matt@sendwithus.com
  - thatthatis 13y ago
sonink 13y ago
- Just because some people clicked on features they want, it doesnt mean they will pay for it once you launch. - Just because some people are using your free features, it doesnt mean they will pay for it once it is paid. - Just because you have 5 paying customers, doesnt mean it will be easy to find out 500 more. - Just because you have 500 paying customers, it doesnt mean what you did to acquire these 500 will scale to the next 5000. - Just because you have 5000 paying customers, it doesnt mean that there won't be a new upstart which will dramatically undercut your offering and acquire most of these. There is a name for this validation process - its called a 'startup'.
- troymcginnis 13y ago
  Well said. Everyone has to start somewhere :)
  - assasinine 13y ago
    exactly. startups are essentially always validating, beginning with what sendwithus/others did to get a product and idea, and then going through the steps sonink listed. like sonink is saying, there are no guarantees with startups, so you might as well try to stack the deck in your favour right from the get-go.
bradhe 13y ago
> We knew that transactional email was a problem worth solving and we were determined to build a great solution. I have a problem with this phrase. Seems to me it needs justification maybe just a verb? I don't immediately think of "transactional email" as a problem...
- bvanvugt 13y ago
  Hmm, how about "transactional email management"?
  - bradhe 13y ago
    just as vague. managing transactional email? What's wrong with doing that? Maybe I don't have the problem...I don't get it.
BrechtVds 13y ago
I'm wondering if the A/B testing score was skewed due to it being the first non-working link in the menu. Someone going through the menu to see what's possible will probably take a look at the top item first. A random order for the menu items might have been better. Or maybe not.
joshdance 13y ago
Can't decide how I feel about this. Does anyone feel that this type of testing is kind of dishonest? On one hand, listing a product as available with a price is a great way to gauge interest. On the other hand, telling someone it is "sold out" when it was never available feels wrong somehow. What do you think?
edgecrafter 13y ago
Obviously smart and handy way of getting metrics for a wannabe startup but as some comments states I would feel cheated as no real product are offered, but offers are pretended to be real. Thus wasting my valuable time. I bet this tactic will backfire soon as more and more try to do the same. I would be more positive if some incentive were offered, e.g. Thx for your interest get 3 month for free when we launch ...

120 points by bvanvugt 13y ago

What's the story with using logos for integration (SendGrid, AWS etc.) without actually supporting them? Is it not possible to get into legal issues there as it's quite an effective boost to the site to make it _look_ like you're associated with very reputable sites? Or.. just something to worry about when people come knocking? :)

mrmch 13y ago
if they come knocking... ;)

ultimatedelman 13y ago

this sounds like the most frustrating experience for a user ever

bvanvugt 13y ago
In a way that's a good thing - having a user work through the frustrating UI and still be interested in the product is amazing validation. Early adopters can self select.
- minimaxir 13y ago
  It's an amazing biased validation, though. You won't be able to determine the usage patterns of the average user, and that's what's important.
  - ThomPete 13y ago
    How does that follow? If someone is willing to use it even when it's absurdly cumbersome then they are going to turn into customers. Once you build it you can iterate to find the best patterns.
- mkr-hn 13y ago
  It makes them resent you and long for a competitor.
  - mrmch 13y ago
    Our experience was to the contrary. It's not shown in the blog post, but users were thanked for their feedback immediately. The reaction was overwhelmingly positive; people were excited that we were building something in this space.
  - kmnc 13y ago
    If a good one exists already that is. And if that is the case why are they checking broken demos? Why are they given their email addresses to broken demos? Either way you slice it, if something like this gains interest it certainly illustrates opportunities.
  - teej 13y ago
    I think it's natural to think that people will react this way. It turns out that people react completely differently. In my experience employing this tactic across several different verticals (SaaS, ecommerce, games), I have found people to react positively (yes, please make this!!) or neutral/mildly disappointed (darn, I wanted to use that). Never resentful.
ThomPete 13y ago
which sounds like the most valid form of market validation.

rubiquity 13y ago

I think this approach is very interesting and may give it a try on a product in the future. How did you go about directing people to the demo though?

bvanvugt 13y ago
Excellent question, and probably grounds for another blog post. We tried to use a variety of channels, including twitter, marketer communities, and even some google adwords.

KiwiCoder 13y ago

> We spent the next 24 hours getting as many people as possible to view and interact with the experiment OP - could you elaborate on how you got these 803 unique visitors within 24 hours?

bvanvugt 13y ago
Sure - that's probably worth another blog post too. We used a combination of twitter, adwords, marketing forums, and personal network.
- lzecon 13y ago
  If you're relying heavily on your personal network I'd be worried that those 803 people were not true customers. Don't get me wrong, I think this is a great idea, but I'm wondering how you know that a 803 person sample size is big enough. Also - could/would people click on more than one button once they found out they were non-functional? It could be that some features would be best developed in tandem with others.
  - bvanvugt 13y ago
    We intentionally diversified the traffic as much as possible, exactly for this reason. Only a small portion of the traffic was from personal networks. Repeat clicks are definitely interesting. It makes sense that once a button was clicked and found to be non-functional, that may prevent future interaction. But that first click still holds value. It'd be interesting to pull stats on how many unique visitors generated those click events.

unfunco 13y ago

We knew that transactional email was a problem worth solving Really? Hasn't this been solved already, by many businesses (SendGrid, Mandrill, SES, MailJet) – I'm happy you've managed to build something and are seeing some success, but I don't think you're solving a problem that hasn't yet found a solution.

EC1 13y ago
"but I don't think you're solving a problem that hasn't yet found a solution." - eliminates like 95% of companies.
bvanvugt 13y ago
SendGrid/Mandrill/Mailgun are great! We've built a much needed layer on top of those services, that focuses on email content and customer experience.
davidw 13y ago
Altavista worked ok, but I'm glad Google came along. I think that your comment isn't a bad one, but should go more in depth. Some markets really are unassailable - others aren't. The reasons are many and sometimes complex, so analyzing one is often quite interesting.
- chc 13y ago
  I don't think unfunco is suggesting that the market is unassailable. The point seemed to be that this problem wasn't one that needed to be solved (because there were already solutions), not that there isn't room for another solution-provider.

alexsilver 13y ago

We are going through the same approach with our current beta customers (and will be making our demos available to a wide audience in the future too) and I can attest to the validity of this approach. We make it very clear to our customers that this is a demo and their changes will go away if they refresh the page. We don't have pop-ups integrated since we collect emails on another page. I think event tracking should be sufficient but we'll have to see what the data says in a little while. We actually also use these semi-live demos instead of mock-ups (since we'll be implementing the same pages afterward anyway) since only one of us is proficient with Photoshop but everyone is comfortable with Bootstrap. It serves as a testing ground for various approaches for JS code too.

xg15 13y ago

The apparent origin of the idea, the "button to nowhere" that they linked to in the story is also a good read: https://medium.com/design-ux/77d911517318 https://medium.com/design-ux/77d911517318 From a user perspective, I think there are both good and bad ways how this technique can be used: I actually like the way Sendwithus used it - as part of a very very early "pre-alpha" version where you're not yet interested in providing a reliant service and just want to get a feel for the demand. As a user you're informed early-on that you're looking at a version in development - so you will not be surprised by missing or incomplete features and will maybe even motivated to provide additional feedback. The downside of this approach is of course that the data you're gathering is really only good for that - an early overview. Your test group will likely be very different from the actual people who will later use the service. The usage patterns will be as well. The second way to use the technique is how Nick Kishfy describes it in the "button to nowhere" article. In that approach, "buttons to nowhere" are not just inserted into an early preview/development version, but into the actual product as a continuous part of the development process. This would solve all of the above problems and would give you a much richer set of data over a much longer time span - but from a user perspective that would be the worst thing possible.

mrmch 13y ago
This is actually a good point -- the "button to nowhere" product is particularly valuable at an early stage. We're working on a followup story on how we do customer development/feedback with a established product.

DanielBMarkham 13y ago

Relevant similar story: I had this site for years, http://paycheck-stub.com http://paycheck-stub.com Developed it back when I was doing microsites, and I wasn't really sure what to do with it. My main question: why do people visit, anyway? I figured I'd know what to do with it once I figured out why people were there. So I did just what these guys did. I created a list of buttons down the page with all the possible things people visiting a site called paycheck stub might want. Hooked them all into Google Analytics. Paycheck templates? Fake paychecks? Loans? Payroll services? A job? Sample paychecks for school or presentations? Very interesting results. First, about 90% didn't want any of that. God knows why they were there. But out of the other 10%, I got a couple pieces of useful information. They were looking for payroll services. So I put up some banner ads. (I think I took down the Adsense. Can't remember) That, in turn, provided more ad revenue than simply using Adsense on the site. It also provided me with a list of topics to develop and some search terms to try to hit. So I developed those topics and tried to do better on those search terms. At one point I was almost ready to jump into online payroll. Fun times! Then Google killed my search ranking -- I wasn't paying for links or spamming. Beats me why. I got so frustrated with them I gave up. Still get quite a bit of traffic, though. But now, I'm sure, the traffic pattern is completely different. I wonder if, for static sites with a lot of content, this is something you should repeat every year or so? Like these guys, I found it a rewarding experience, both for my understanding of the site and for my readers. The reason I'm posting is that many times we view this as a step on the way to creating SaaS, but it can be just a great way to connect with your readers. I bet it's extremely applicable for a lot of blogs. Wish I could have figured out why those other 90% were there.

mrmch 13y ago
Interesting story Daniel -- there should be an "Optimizely for the button to nowhere" product!
- DanielBMarkham 13y ago
  Hardest part was coming up with the questions. I still feel like I was missing something. Some important reason they were visiting. But when you have a site with traffic, you end up with tons of data and very little insight. The button thing moves the needle a little bit, and that's better than nothing. ADD: One other thing: there's also a minimum amount of traffic you need to make this thing work. 100 visits isn't enough. I'd imagine between 1-5K? Otherwise there's too much noise. Was that your experience as well?

alexeyza 13y ago

Awesome way to design a product. I wonder though, how the users reacted when they saw "dead end features". Were there any frustrated users when they saw it was mainly a "mock-up" web site ?

ForHackernews 13y ago

Does anyone else find this kind of pre-development marketing kind of scummy and deceptive? It's basically lying to potential customers: You're pretending to offer to sell a product that doesn't exist.

mrmch 13y ago
Perhaps if we were accepting pre-orders or requiring a credit card. A soft launch like this would be more comparable to a focus group for a traditional product launch.
- ForHackernews 13y ago
  But you're not being honest and up-front about the fact that it's a focus group. If it was like "Click here to take a survey about what features you most want in an email service" that would be one thing, but that's not the messaging in your case.
  - petervandijck 13y ago
    Effective focus groups aren't up-front or honest either. Effective experiments don't tell the person what the experiment is.
    - ForHackernews 13y ago
      But they at least tell you that it is an experiment. Maybe we need human-subject ethics boards for marketers.
      - troymcginnis 13y ago
        Just knowing that it's an experiment defeats the purpose. By getting honest responses and behaviors, it is easier to build a solution that people actually need or would actually use.
        13y ago

deleted 13y ago

[deleted]

allanlundhansen 13y ago

"We knew that transactional email was a problem worth solving" A story worth telling in it's own right I believe, when clamoring on about lean methodologies. Nice article would love the pre-phase lean story leading up to this stage.

bvanvugt 13y ago
Absolutely! Prior to this experiment we spent nearly two months talking to customers/partners of all sizes. The details of that period would make a great follow-up blog post.

nobodysfool 13y ago

I did that with my father's computer store website back in 1998. We had computer descriptions, a price, and a 'buy online' button. The button just had a hit counter on the page, with directions to the store. Every hit meant someone wanted to buy a computer through our website and couldn't. I showed him the data that we could double our sales just by making an online store, but he was not convinced. In any case, the cash flows of that business wouldn't have worked out for him. Still, I thought this was a widely known technique.

pbreit 13y ago

So did the API turn out to be a popular feature? I'm wondering if APIs prove interesting on the surface but not always in practice?

bvanvugt 13y ago
In our case it did! Our core product today is an API to send templated email.

redguava 13y ago

Are you sure the value you got from the metrics exceeds the bad will generated by annoying your potential customers? To me it's a lack of respect for your users and their time.

mrmch 13y ago
Some people seem to be missing that first screenshot; the landing page clearly started with "Start The Demo" (http://sendwithus.wpengine.com/wp-content/uploads/2014/03/Screenshot-2014-03-09-14.27.00-e1394434326899.png http://sendwithus.wpengine.com/wp-content/uploads/2014/03/Sc...)
- redguava 13y ago
  Demo doesn't imply fake features. For me demo would be not my data, perhaps a sample set in there. I'd have no problem if it said "there's a bunch of features not yet implemented, we have the buttons there to gauge interest and we'll work on the most needed ones".

JimmyL 13y ago

I went though sendwithus zero as an unknowing participant in the experiment, and it sucked. At the time, the company I was working at was looking for a service exactly like this - a service where our app could just fling events, and then we'd do all the logic to control what emails got emitted on the basis of those events on the external service (and A/B test them). One of our "growth hacking" guys found SendWithUs somehow, and introduced us. We had a few calls with them - the service seemed like it had potential, and we got beta access to it...which was really sendwithus zero, but sending basic email. We got most of the way though setting up an integration with them, but every time we asked when the A/B testing framework - which was what sold us on the product - was coming online, we were told "it's coming soon, it's what we're working on right now." So we held on for a few more weeks, and gave them some time...but nothing came of it. I was skeptical at the time that they were actually building it, even moreso having read this. The positive side is that while we were being strung along by sendwithus, we found Vero (https://www.getvero.com/ https://www.getvero.com/). They're not perfect either, but we've been on them for 6-8 months, and they've done well for us. They're responsive to our issues, are willing to Skype or chat when we try and develop some sophisticated campaigns on their platform, and have (or so it seems) rejigged their roadmap a few times to add in small features that we ask for. ExactTarget is still a more full-featured package, but I'm happy with the price/value of Vero. So well done, guys. You've got an interesting story about how you bootstrapped your business, and you're evidently doing OK. I wish you the best, but I can tell you that you won't be getting my business (or my recommendation) - at the time I thought you were small and were actually on the cusp of delivering what you had promised us worked; now I see that we were just one more data point for a feature set that you never planned on having ready when you claimed.

mrmch 13y ago
Hey Jimmy, I'm sorry to hear that was your experience. I don't know when you found us, but it did take us time to roll out our A/B testing; we never intended to string anyone along. If you're willing to chat offline about your experience, please get in touch: matt@sendwithus.com
- thatthatis 13y ago

sonink 13y ago

- Just because some people clicked on features they want, it doesnt mean they will pay for it once you launch. - Just because some people are using your free features, it doesnt mean they will pay for it once it is paid. - Just because you have 5 paying customers, doesnt mean it will be easy to find out 500 more. - Just because you have 500 paying customers, it doesnt mean what you did to acquire these 500 will scale to the next 5000. - Just because you have 5000 paying customers, it doesnt mean that there won't be a new upstart which will dramatically undercut your offering and acquire most of these. There is a name for this validation process - its called a 'startup'.

troymcginnis 13y ago
Well said. Everyone has to start somewhere :)
- assasinine 13y ago
  exactly. startups are essentially always validating, beginning with what sendwithus/others did to get a product and idea, and then going through the steps sonink listed. like sonink is saying, there are no guarantees with startups, so you might as well try to stack the deck in your favour right from the get-go.

bradhe 13y ago

> We knew that transactional email was a problem worth solving and we were determined to build a great solution. I have a problem with this phrase. Seems to me it needs justification maybe just a verb? I don't immediately think of "transactional email" as a problem...

bvanvugt 13y ago
Hmm, how about "transactional email management"?
- bradhe 13y ago
  just as vague. managing transactional email? What's wrong with doing that? Maybe I don't have the problem...I don't get it.

BrechtVds 13y ago

I'm wondering if the A/B testing score was skewed due to it being the first non-working link in the menu. Someone going through the menu to see what's possible will probably take a look at the top item first. A random order for the menu items might have been better. Or maybe not.

joshdance 13y ago

Can't decide how I feel about this. Does anyone feel that this type of testing is kind of dishonest? On one hand, listing a product as available with a price is a great way to gauge interest. On the other hand, telling someone it is "sold out" when it was never available feels wrong somehow. What do you think?

edgecrafter 13y ago

Obviously smart and handy way of getting metrics for a wannabe startup but as some comments states I would feel cheated as no real product are offered, but offers are pretended to be real. Thus wasting my valuable time. I bet this tactic will backfire soon as more and more try to do the same. I would be more positive if some incentive were offered, e.g. Thx for your interest get 3 month for free when we launch ...