5 ms·

Benefit Layers: How to avoid sales fluff in devtools

68 points by swyx 3y ago

jawns 3y ago
Or A/B test it all against what actually leads to sales, and then use that. It might be the case that we think we want no fluff, but we're actually swayed more by fluff.
- ilrwbwrkhv 3y ago
  You might not get statistic significance if your numbers are low when first starting out.
- swyx 3y ago
  the problem is 1) sample sizes are very low for early startups 2) feedback loops (esp for higher ACV) are long, sometimes >1 year. you still need principles to drive at the end of the day
- alphager 3y ago
  I don't think that A/B tests work for things that enterprises buy. The normal workflow is 1. Dev visits the site, gets convinced that he wants the tool. 2. Dev convinced management to allow the expense. 3. Dev visits the site and buys the tool. You want to test step 1 but only get data for step 3, where the content of the site is completely irrelevant.
swyx 3y ago
i've been doing devtools advising for a little bit, and over 2 years ago I reflected (https://twitter.com/swyx/status/1679258125067767808 https://twitter.com/swyx/status/1679258125067767808) One thing I've consistently had to relearn in developer marketing: "Talk benefits, not features" doesn't work! Developers are sick of vague promises and wary of black boxes. Instead: - Demo - reach Wow! in <10 mins - Explain how it works - Show how real companies use in prod and that really resonated with folks (heard some really good feedback from senior github people). excited to return to this topic with a better framework (https://pbs.twimg.com/media/F03sDA6agAEBf_M?format=png&name=900x900 https://pbs.twimg.com/media/F03sDA6agAEBf_M?format=png&name=...) for when to communicate with benefits vs features
- hsn915 3y ago
  Talking about supposed benefits instead of features is the worst marketing advice I've ever heard that's often repeated by everyone. It obviously doesn't work. Can you imagine a restaurant flyer that just says "Enjoy delicious meals! Have fun time!" It's absurd. You show pictures of the food. Tell people how you source your ingredients. How your chefs have perfected their craftsmanship. Etc etc.
  - wmf 3y ago
    I'm pretty sure it does work for complex enterprise tech being sold to people who don't understand it.
    - hsn915 3y ago
      Doesn't even work there. It only appears to work if you think that kind of advertising is how they got to their market position. Usually it's not. Let's Docker for an example. Docker did not get adoption because of some cliche marketing on their landing page. It got adoption from people doing tech talks and tech demonstrations and several companies writing blog posts about how migrating to Docker saved them from the headaches of deployment they were having before with the ad-hoc systems. Complex enterprise tech gets sold by other means, usually personal relationships with key decision makers, and maybe some amount of bribery, who knows.
      - lmeyerov 3y ago
        That's a funny one wrt enterprise Docker failed to sell its commercial products to enterprises relative to the OSS adoption of its container runtime, at least at the venture scales they committed themselves to Afaict what was being sold: * Artifactory (hub): plenty of companies doing fine.. for product different than what docker did * DC OS (swarm): managed k8s etc doing fine too.. but diff product again They seemed close to PMF for both... But never quite hit it? Selling to enterprises is hard.. I'm not sure what the lesson is here beyond git stars not being revenue, nor guaranteeing that it can turn into revenue. Successfully building one thing but selling another means you have to do 2 hard things not just 1. Maybe a good analogy: Bloomberg has great reporters whose content goes out for free... But that's not what sells the terminal and data feeds
krackers 3y ago
I had to re-skim the article a few times before realizing they're talking about "dev tools" not DevTools (i.e. page inspector). I've seen webpages advertise (usually job openings) on DevTools before, now I wonder how effective _that_ strategy is.
- mcny 3y ago
  > I had to re-skim the article a few times before realizing they're talking about "dev tools" not DevTools (i.e. page inspector). I've seen webpages advertise (usually job openings) on DevTools before, now I wonder how effective _that_ strategy is. Got me a first round at Google. Unfortunately for them, I super suck and therefore failed in the first round. Ain't nobody got time for leet code when they already have a job.
  - intelVISA 3y ago
    Candidate disclosed that runtime complexity was 'irrelevant as he had 3YOE in a random web shop' and that we should 'use real technology like NuxtJs'
    - mcny 3y ago
      > Candidate disclosed that runtime complexity was 'irrelevant as he had 3YOE in a random web shop' and that we should 'use real technology like NuxtJs' In my defense, I was more respectful than that. Also, the "we should use real technology" was my response at Time Magazine, not at Google. I stand by it. Server-side javascript (isomorphic javascript) is stupid. At least try to use typescript or something. The interviewer clearly agreed with me. Problem is it was above their pay grade as well.
laserbeam 3y ago
If you're selling a devtool, and all I'm seeing is features OR benefits in your pitch, and I can't get limitations, tradeoffs, restrictions, drawbacks or any sort of information about the downsides of using that tool... Then it's still a marketing pitch. If it only lists positives and it's not honest about negatives, it's just marketing. Show me your drawbacks and I might consider you are a serious vendor.
- siquick 3y ago
  > Show me your drawbacks and I might consider you are a serious vendor. Can’t remember any other product from any market openly showing what it can’t do, why are devtools any different?
  - swyx 3y ago
    in my original landing page for temporal i was very proud to add "what do we NOT do": https://temporaldotio-a0ktr49oc-temporal.vercel.app/#headlessui-disclosure-button-5 https://temporaldotio-a0ktr49oc-temporal.vercel.app/#headles... most founders refuse to admit that they're not fit for purpose for some things. yes need to promise the world to VCs but also it makes it harder for the developer to assess how much to trust you if you are in denial about TAM
  - laserbeam 3y ago
    To be fair, my argument has nothing to do with devtools, just that the distinction between features and benefits is pointless or is itself a marketing gimmick. I'll only get to drawbacks/limitations once I get to documentation. And once I spend a significant amount of time researching a product. And when I don't I feel like I'm buying the marketing.
  - CJefferson 3y ago
    SQLite has both when to use it, and when to consider something else: https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
  - ben_w 3y ago
    > any other product from any market openly showing what it can’t do ChatGPT, ironically enough given how many times I see people insisting that them saying “we can be dangerously wrong don’t rely on us” is some kind of marketing scheme.
reillyse 3y ago
This is really resonating with me. As a founder who is currently getting frustrated trying to market a devtool ((https://brisktest.com/ https://brisktest.com/) literally a competitor for CircleCI, Github actions etc) it's really hard to talk about the features of the product. It is multiple of times faster than the competitors but nobody believes you when you say that. Developers are so suspicious of something they've never heard about and really reluctant to take a chance even when they are bleeding from the neck. It's a real grind.
- lelanthran 3y ago
  Regardless of the target market, if the target market sees your offering and doesn't show interest, then they are not bleeding from the neck. IOW, the pain you think they are feeling is not the pain they are actually feeling. As someone also trying to bootstrap a product, I feel your pain, but it's not specific to developers.
  - reillyse 3y ago
    I'm not sure. I think specifically with what I'm doing (making tests fast) there is a kind of a Stockholm syndrome. I've talked to people who say things like "It's fine that my tests take 15 minutes to run, I just check my email, or go for a walk". Or they propose that you don't run the tests or only run part of the tests when you merge. I know personally that waiting 15 minutes for a test suite to finish is super frustrating, but people seem to have normalized it. I've also literally been hired to speed up people's tests, so it's not a problem that doesn't exist. Our current customers are also delighted with the speedups so I know it's a problem. But explaining that problem and why the status quo is bad to developers is especially hard. Btw I don't buy the "developers are immune to advertising" spiel, I actually created some of CircleCIs first ads so I have experience doing this. I just think developers are different and marketing devtools is different again (because of it's not a personal purchase and there are a lot of different stakeholders).
    - lelanthran 3y ago
      I hear you, but (devils advocate time) if that 15m delay is not enough for them to open their wallets, then perhaps pivot to a new product. They probably don't notice that delay, because they are doing something else during that time ... Reading the next ticket, some git admin, replying to an email, checking slack, etc. As a single example, I don't consider a 15m delay for a full test suite to be a problem. I've got plenty of 5m tasks to do that I can use to fill in that time. I cannot think of any developer who sits and watches the tests for execute.

68 points by swyx 3y ago

Or A/B test it all against what actually leads to sales, and then use that. It might be the case that we think we want no fluff, but we're actually swayed more by fluff.

ilrwbwrkhv 3y ago
You might not get statistic significance if your numbers are low when first starting out.
swyx 3y ago
the problem is 1) sample sizes are very low for early startups 2) feedback loops (esp for higher ACV) are long, sometimes >1 year. you still need principles to drive at the end of the day
alphager 3y ago
I don't think that A/B tests work for things that enterprises buy. The normal workflow is 1. Dev visits the site, gets convinced that he wants the tool. 2. Dev convinced management to allow the expense. 3. Dev visits the site and buys the tool. You want to test step 1 but only get data for step 3, where the content of the site is completely irrelevant.

swyx 3y ago

i've been doing devtools advising for a little bit, and over 2 years ago I reflected (https://twitter.com/swyx/status/1679258125067767808 https://twitter.com/swyx/status/1679258125067767808) One thing I've consistently had to relearn in developer marketing: "Talk benefits, not features" doesn't work! Developers are sick of vague promises and wary of black boxes. Instead: - Demo - reach Wow! in <10 mins - Explain how it works - Show how real companies use in prod and that really resonated with folks (heard some really good feedback from senior github people). excited to return to this topic with a better framework (https://pbs.twimg.com/media/F03sDA6agAEBf_M?format=png&name=900x900 https://pbs.twimg.com/media/F03sDA6agAEBf_M?format=png&name=...) for when to communicate with benefits vs features

hsn915 3y ago
Talking about supposed benefits instead of features is the worst marketing advice I've ever heard that's often repeated by everyone. It obviously doesn't work. Can you imagine a restaurant flyer that just says "Enjoy delicious meals! Have fun time!" It's absurd. You show pictures of the food. Tell people how you source your ingredients. How your chefs have perfected their craftsmanship. Etc etc.
- wmf 3y ago
  I'm pretty sure it does work for complex enterprise tech being sold to people who don't understand it.
  - hsn915 3y ago
    Doesn't even work there. It only appears to work if you think that kind of advertising is how they got to their market position. Usually it's not. Let's Docker for an example. Docker did not get adoption because of some cliche marketing on their landing page. It got adoption from people doing tech talks and tech demonstrations and several companies writing blog posts about how migrating to Docker saved them from the headaches of deployment they were having before with the ad-hoc systems. Complex enterprise tech gets sold by other means, usually personal relationships with key decision makers, and maybe some amount of bribery, who knows.
    - lmeyerov 3y ago
      That's a funny one wrt enterprise Docker failed to sell its commercial products to enterprises relative to the OSS adoption of its container runtime, at least at the venture scales they committed themselves to Afaict what was being sold: * Artifactory (hub): plenty of companies doing fine.. for product different than what docker did * DC OS (swarm): managed k8s etc doing fine too.. but diff product again They seemed close to PMF for both... But never quite hit it? Selling to enterprises is hard.. I'm not sure what the lesson is here beyond git stars not being revenue, nor guaranteeing that it can turn into revenue. Successfully building one thing but selling another means you have to do 2 hard things not just 1. Maybe a good analogy: Bloomberg has great reporters whose content goes out for free... But that's not what sells the terminal and data feeds

krackers 3y ago

I had to re-skim the article a few times before realizing they're talking about "dev tools" not DevTools (i.e. page inspector). I've seen webpages advertise (usually job openings) on DevTools before, now I wonder how effective _that_ strategy is.

mcny 3y ago
> I had to re-skim the article a few times before realizing they're talking about "dev tools" not DevTools (i.e. page inspector). I've seen webpages advertise (usually job openings) on DevTools before, now I wonder how effective _that_ strategy is. Got me a first round at Google. Unfortunately for them, I super suck and therefore failed in the first round. Ain't nobody got time for leet code when they already have a job.
- intelVISA 3y ago
  Candidate disclosed that runtime complexity was 'irrelevant as he had 3YOE in a random web shop' and that we should 'use real technology like NuxtJs'
  - mcny 3y ago
    > Candidate disclosed that runtime complexity was 'irrelevant as he had 3YOE in a random web shop' and that we should 'use real technology like NuxtJs' In my defense, I was more respectful than that. Also, the "we should use real technology" was my response at Time Magazine, not at Google. I stand by it. Server-side javascript (isomorphic javascript) is stupid. At least try to use typescript or something. The interviewer clearly agreed with me. Problem is it was above their pay grade as well.

laserbeam 3y ago

If you're selling a devtool, and all I'm seeing is features OR benefits in your pitch, and I can't get limitations, tradeoffs, restrictions, drawbacks or any sort of information about the downsides of using that tool... Then it's still a marketing pitch. If it only lists positives and it's not honest about negatives, it's just marketing. Show me your drawbacks and I might consider you are a serious vendor.

siquick 3y ago
> Show me your drawbacks and I might consider you are a serious vendor. Can’t remember any other product from any market openly showing what it can’t do, why are devtools any different?
- swyx 3y ago
  in my original landing page for temporal i was very proud to add "what do we NOT do": https://temporaldotio-a0ktr49oc-temporal.vercel.app/#headlessui-disclosure-button-5 https://temporaldotio-a0ktr49oc-temporal.vercel.app/#headles... most founders refuse to admit that they're not fit for purpose for some things. yes need to promise the world to VCs but also it makes it harder for the developer to assess how much to trust you if you are in denial about TAM
- laserbeam 3y ago
  To be fair, my argument has nothing to do with devtools, just that the distinction between features and benefits is pointless or is itself a marketing gimmick. I'll only get to drawbacks/limitations once I get to documentation. And once I spend a significant amount of time researching a product. And when I don't I feel like I'm buying the marketing.
- CJefferson 3y ago
  SQLite has both when to use it, and when to consider something else: https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html
- ben_w 3y ago
  > any other product from any market openly showing what it can’t do ChatGPT, ironically enough given how many times I see people insisting that them saying “we can be dangerously wrong don’t rely on us” is some kind of marketing scheme.

reillyse 3y ago

This is really resonating with me. As a founder who is currently getting frustrated trying to market a devtool ((https://brisktest.com/ https://brisktest.com/) literally a competitor for CircleCI, Github actions etc) it's really hard to talk about the features of the product. It is multiple of times faster than the competitors but nobody believes you when you say that. Developers are so suspicious of something they've never heard about and really reluctant to take a chance even when they are bleeding from the neck. It's a real grind.

lelanthran 3y ago
Regardless of the target market, if the target market sees your offering and doesn't show interest, then they are not bleeding from the neck. IOW, the pain you think they are feeling is not the pain they are actually feeling. As someone also trying to bootstrap a product, I feel your pain, but it's not specific to developers.
- reillyse 3y ago
  I'm not sure. I think specifically with what I'm doing (making tests fast) there is a kind of a Stockholm syndrome. I've talked to people who say things like "It's fine that my tests take 15 minutes to run, I just check my email, or go for a walk". Or they propose that you don't run the tests or only run part of the tests when you merge. I know personally that waiting 15 minutes for a test suite to finish is super frustrating, but people seem to have normalized it. I've also literally been hired to speed up people's tests, so it's not a problem that doesn't exist. Our current customers are also delighted with the speedups so I know it's a problem. But explaining that problem and why the status quo is bad to developers is especially hard. Btw I don't buy the "developers are immune to advertising" spiel, I actually created some of CircleCIs first ads so I have experience doing this. I just think developers are different and marketing devtools is different again (because of it's not a personal purchase and there are a lot of different stakeholders).
  - lelanthran 3y ago
    I hear you, but (devils advocate time) if that 15m delay is not enough for them to open their wallets, then perhaps pivot to a new product. They probably don't notice that delay, because they are doing something else during that time ... Reading the next ticket, some git admin, replying to an email, checking slack, etc. As a single example, I don't consider a 15m delay for a full test suite to be a problem. I've got plenty of 5m tasks to do that I can use to fill in that time. I cannot think of any developer who sits and watches the tests for execute.