6 ms·

I don't know if this calculator was good or bad, but the rationale sounds superficially ridiculous. > Visitors who didn't see the calculator were 16% more like

I don't know if this calculator was good or bad, but the rationale sounds superficially ridiculous. > Visitors who didn't see the calculator were 16% more likely to sign up and 90% more likely to contact us than those who saw it. There was no increase in support tickets about pricing, which suggests users are overall less confused and happier. Of course if you hide the fact that your product might cost a lot of money from your users, more of them will sign up. Whether they are better off depends on whether they end up getting a bill they are unhappy with later at some unspecified future date, or not. That's not something you will figure out from a short-term A/B test on the signup page. So this seems like totally useless evidence to me. I see this dynamic frequently with A/B tests. For example, one of my coworkers implemented a change that removed information from search result snippets. They then ran an A/B test that showed that after removing the information, people clicked through to the search result page more often. Well, obviously, it makes sense that they might click through more often, if information they wanted which was previously in the snippet, now requires them to click through. The question of which is actually better seemed to have been totally forgotten.

imoverclocked 2y ago
> Whether they are better off depends on whether they end up getting a bill they are unhappy with later at some unspecified future date, or not. How is that a function of the overly-simplified and often-wrong calculator? If the user is never there to be happy/unhappy about it in the first place, then how would you test this anyway? By closing the loop and increasing engagement, you are increasing the chance that you can make the customer happy and properly educated through future interactions.
- afro88 2y ago
  > often-wrong The author was very careful with their words: they didn't say the calculator was wrong. They said it was confusing and sensitive to small adjustments. It's likely that the same confusion and variable sensitivity exists during usage. IMHO they should have bit the bullet and revised the pricing model.
  - imoverclocked 2y ago
    > they didn't say the calculator was wrong Fair point. The author commented elsewhere here and stated that it's not the usage but the understanding of the variables in the calculator which are often wrong by more than 10x. From the response, it seems like the only way to know how much something will cost is to actually run a workload. Edit: if the customer is getting a wrong-answer because of wrong-inputs, IMO, it's still a wrong-answer. > IMHO they should have bit the bullet and revised the pricing model I don't know enough to agree/disagree because they may be offering close to at-cost which might give better overall pricing than competitors. It's a complex-game :)
shubhamjain 2y ago
Absolutely true! An A/B test enthusiast in our team once significantly reduced padding on the pricing page to bring the signup button above the fold and used the increase in signup button clicks as a proof of success of the experiment. Of course, the pricing page became plain ugly, but that didn't matter, because "Signups are increasing!!" In this case, I do agree that the calculator is a bit daunting if you're not used to all the terms, but what should be done with it should have been an intuitive decision ("what can we do to simplify the calculator?") Not a fan of A/B testing culture that everything needs to be statistically analyzed and proved.
- bruce511 2y ago
  >> Of course, the pricing page became plain ugly, but that didn't matter, because "Signups are increasing!!" I'm not sure I'm following you here, so perhaps you'd care to elaborate? The GP critique was that it was perhaps just creating a problem elsewhere later on. I'm not seeing the similarity to your case where the change is cosmetic not functional. The issue of whitespace (padding) is subjective (see the conversation recently between the old and new windows control panels) but "scrolling down" does seem to be something that should potentially be avoided. If sign-ups are increasing is that not the goal of the page? Is there reason to believe that the lack of padding is going to be a problem for those users?
  - strken 2y ago
    I think one problem is that a better design would move the button above the fold without ruining the spacing, and therefore achieve a better result with even higher lift, but someone focused on just the numbers wouldn't understand this. The fact that the A/B test has a big green number next to it doesn't mean you should stop iterating after one improvement.
    - carlmr 2y ago
      It still seems like a valid use-case for AB testing. Ideally, you should maybe redo the design, something you could AB test if it helps. My guess is yes, because consistency in design usually makes people assume better quality.
jessriedel 2y ago
Relatedly, there seemed to be no acknowledgement of the possibility of dark incentives: many businesses have found they can increase sales by removing pricing details so that prospective customers get deeper into the funnel and end up buying because of sunk time costs even though they would have preferred a competitor. Example: car dealerships make it a nightmare to get pricing information online, and instead cajole you to email them or come in in person. In other words, a calculator makes it easier to comparison shop, which many businesses don't like. I have no idea if that's a conscious or unconscious motivation for this business, but even if its not conscious it needs to be considered.
- autonomousErwin 2y ago
  Would you consider doubling your prices so users perceive your product as having higher value a dark pattern?
  - j33zusjuice 2y ago
    Feels like it kinda fits under fake social proof. https://www.deceptive.design/types https://www.deceptive.design/types
  - jessriedel 2y ago
    Only if this were untrue, i.e., I was motivated by the fact that it made my customers believe my product was better than a worse product. For me the principle is based on not exploiting the gap between the consumer and a better informed version of themselves. (“What would they want if they knew?”) There’s a principle of double effect: I don’t have to expend unlimited resources to educate them, but I shouldn’t take active steps to reduce information, and I shouldn’t leave them worse off compared to me not being in the market.
- olejorgenb 2y ago
  To be fair, the pricing is still available in this case: https://www.pinecone.io/pricing/ https://www.pinecone.io/pricing/ (though the "Unlimited XXX" with the actual price below in gray might be considered misleading)
TeMPOraL 2y ago
This is a blind spot for pretty much entire industry, and arguably spreads beyond tech, into industrial design and product engineering in general. Of course being transparent with your users is going to be more confusing - the baseline everyone's measuring against is treating users like dumb cattle that can be nudged to slaughter. Against this standard, any feature that treats the user as a thinking person is going to introduce confusion and compromise conversions.
- worldsayshi 2y ago
  > treating users like dumb cattle that can be nudged Essentially the failure is that we do treat users like this by relying on mass collection of data instead of personal stories. To be human is to communicate face to face with words and emotions. That's how you can get the nuanced conclusions. Data is important but it's far from the whole story.
atoav 2y ago
There are multiple companies on my blacklist that definitely got me to sign up. But as there was a hook that anybody acting as a trustworthy partner would have mentioned, I parted with them — potentially for life. You know, things like "click here to sign up, sacrifice your newborn on a fullmoon night while reciting the last 3 digits of pi to cancel" I don't particular care whether their A/B test captures that potential aspect of customer (dis)satisfaction, but I am not sure how it would.
richardw 2y ago
I designed an internal system that optimises for long term outcomes. We do nothing based on whether you click “upgrade”. We look at the net change over time, including impact to engagement and calls to support months later and whether you leave 6 months after upgrading. Most of the nudges are purely for the customer’s benefit because it’ll improve lifetime value.
- sethammons 2y ago
  You could only be measuring in aggregate, no? Overall signal could be positive but one element happens to be negative while another is overly positive.
  - richardw 2y ago
    Well, adjusting nudges in aggregate but diced in various ways. Measured very much not in aggregate. We’d see positive and negative outcomes roll in over multiple years and want it per identifier (an individual). I’ve heard of companies generating a model per person but we didn’t. A silly amount of work but honestly lots of value. Experimentation optimising for short term goals (eg upgrade) is such a bad version of this, it’s just all that is possible with most datasets.
- bitshiftfaced 2y ago
  That's the only thing I was thinking with their A/B test. The calculator might immunize against unhappy customers later on. I think they could've looked at something like the percentage of customers who leave one or two billing cycles later.
  - fwip 2y ago
    Unfortunately, there's never enough time to run a proper experiment - we want answers now! Who cares if they're the right answers. Short-termism can't wait two months.
otherme123 2y ago
> Of course if you hide the fact that your product might cost a lot of money from your users, more of them will sign up The problem with their calculator was that the users introduced slightly wrong data, or misunderstand what means some metric, and suddenly a 1000x the real price was shown. Their dilemma was "how to fix those cases", and the solution was "get rid of the messy calculator". But they are not hidding a 1000x cost, they are avoiding losing users that get a wrong 1000x quote.
- xorcist 2y ago
  Let's not take a PR piece completely at face value. There's probably a bit of both, at the very least.
  - PaulHoule 2y ago
    In my mind Pinecone is an exemplary example of modern "social media marketing" for a technology company. They started on vector search at a time when RAG in its current form wasn't a thing; there were just a few search products based on vector search (like a document embedding-based search engine for patents that I whipped into shape to get in front of customers) and if you were going to use vector search you'd need to develop your own indexing system in house or just do a primitive full scan (sounds silly but it's a best-case scenario for full scan and vector indexes do not work as well as 1-d indexes) They blogged frequently and consistently about the problem they were working on with heart, which I found fascinating because I'd done a lot of reading about the problem in the mid ought's. Thus Pinecone had a lot of visibility for me, although I don't know if I am really their market. (No budget for a cloud system, full scan is fine for my 5M document collection right now, I'd probably try FAISS if it wasn't.) Today their blog looks more than it used to which makes it a little harder for me to point out how awesome their blog was in the beginning but this post is definitely the kind of post that they made when they were starting out. I'm sure it has been a big help in finding employees, customers and other allies.
    - gk1 2y ago
      Thank you. :) I don’t think of it as social media marketing but more of helping our target audience learn useful things. Yes that requires they actually find the articles which means sharing it on social, being mindful of SEO, and so on. Probably our learning center is what you’re thinking of. https://www.pinecone.io/learn/ https://www.pinecone.io/learn/ … The blog is more of a news ticker for product and company news.
raverbashing 2y ago
> so we dug into it and realized the calculator was far more confusing and sensitive than we thought. One slight misinterpretation and wrong input and you'd get an estimate that's overstated by as much as 1,000x. They should have looked into this to see how to make it more obvious or more reflective of "regular use case" Their sliders there are not too detailed. For example, what are namespaces, how many would a typical use need? Is 100 too much or too little? And if this is one of the variables that is too sensitive they would need to represent this in a different way
scott_w 2y ago
That’s why you need domain experts and clear explanations and hypotheses before you experiment, otherwise you’re throwing shit at a wall to see what sticks. Companies can continue to monitor cohorts to compare retention to check the potential outcomes you highlighted.
hinkley 2y ago
Also clicking through is not a good thing if it doesn’t result in revenue! Why do I want to render eight pages for someone who will never give us money if I can find that out in three?

imoverclocked 2y ago

> Whether they are better off depends on whether they end up getting a bill they are unhappy with later at some unspecified future date, or not. How is that a function of the overly-simplified and often-wrong calculator? If the user is never there to be happy/unhappy about it in the first place, then how would you test this anyway? By closing the loop and increasing engagement, you are increasing the chance that you can make the customer happy and properly educated through future interactions.

afro88 2y ago
> often-wrong The author was very careful with their words: they didn't say the calculator was wrong. They said it was confusing and sensitive to small adjustments. It's likely that the same confusion and variable sensitivity exists during usage. IMHO they should have bit the bullet and revised the pricing model.
- imoverclocked 2y ago
  > they didn't say the calculator was wrong Fair point. The author commented elsewhere here and stated that it's not the usage but the understanding of the variables in the calculator which are often wrong by more than 10x. From the response, it seems like the only way to know how much something will cost is to actually run a workload. Edit: if the customer is getting a wrong-answer because of wrong-inputs, IMO, it's still a wrong-answer. > IMHO they should have bit the bullet and revised the pricing model I don't know enough to agree/disagree because they may be offering close to at-cost which might give better overall pricing than competitors. It's a complex-game :)

shubhamjain 2y ago

Absolutely true! An A/B test enthusiast in our team once significantly reduced padding on the pricing page to bring the signup button above the fold and used the increase in signup button clicks as a proof of success of the experiment. Of course, the pricing page became plain ugly, but that didn't matter, because "Signups are increasing!!" In this case, I do agree that the calculator is a bit daunting if you're not used to all the terms, but what should be done with it should have been an intuitive decision ("what can we do to simplify the calculator?") Not a fan of A/B testing culture that everything needs to be statistically analyzed and proved.

bruce511 2y ago
>> Of course, the pricing page became plain ugly, but that didn't matter, because "Signups are increasing!!" I'm not sure I'm following you here, so perhaps you'd care to elaborate? The GP critique was that it was perhaps just creating a problem elsewhere later on. I'm not seeing the similarity to your case where the change is cosmetic not functional. The issue of whitespace (padding) is subjective (see the conversation recently between the old and new windows control panels) but "scrolling down" does seem to be something that should potentially be avoided. If sign-ups are increasing is that not the goal of the page? Is there reason to believe that the lack of padding is going to be a problem for those users?
- strken 2y ago
  I think one problem is that a better design would move the button above the fold without ruining the spacing, and therefore achieve a better result with even higher lift, but someone focused on just the numbers wouldn't understand this. The fact that the A/B test has a big green number next to it doesn't mean you should stop iterating after one improvement.
  - carlmr 2y ago
    It still seems like a valid use-case for AB testing. Ideally, you should maybe redo the design, something you could AB test if it helps. My guess is yes, because consistency in design usually makes people assume better quality.

jessriedel 2y ago

Relatedly, there seemed to be no acknowledgement of the possibility of dark incentives: many businesses have found they can increase sales by removing pricing details so that prospective customers get deeper into the funnel and end up buying because of sunk time costs even though they would have preferred a competitor. Example: car dealerships make it a nightmare to get pricing information online, and instead cajole you to email them or come in in person. In other words, a calculator makes it easier to comparison shop, which many businesses don't like. I have no idea if that's a conscious or unconscious motivation for this business, but even if its not conscious it needs to be considered.

autonomousErwin 2y ago
Would you consider doubling your prices so users perceive your product as having higher value a dark pattern?
- j33zusjuice 2y ago
  Feels like it kinda fits under fake social proof. https://www.deceptive.design/types https://www.deceptive.design/types
- jessriedel 2y ago
  Only if this were untrue, i.e., I was motivated by the fact that it made my customers believe my product was better than a worse product. For me the principle is based on not exploiting the gap between the consumer and a better informed version of themselves. (“What would they want if they knew?”) There’s a principle of double effect: I don’t have to expend unlimited resources to educate them, but I shouldn’t take active steps to reduce information, and I shouldn’t leave them worse off compared to me not being in the market.
olejorgenb 2y ago
To be fair, the pricing is still available in this case: https://www.pinecone.io/pricing/ https://www.pinecone.io/pricing/ (though the "Unlimited XXX" with the actual price below in gray might be considered misleading)

TeMPOraL 2y ago

This is a blind spot for pretty much entire industry, and arguably spreads beyond tech, into industrial design and product engineering in general. Of course being transparent with your users is going to be more confusing - the baseline everyone's measuring against is treating users like dumb cattle that can be nudged to slaughter. Against this standard, any feature that treats the user as a thinking person is going to introduce confusion and compromise conversions.

worldsayshi 2y ago
> treating users like dumb cattle that can be nudged Essentially the failure is that we do treat users like this by relying on mass collection of data instead of personal stories. To be human is to communicate face to face with words and emotions. That's how you can get the nuanced conclusions. Data is important but it's far from the whole story.

atoav 2y ago

There are multiple companies on my blacklist that definitely got me to sign up. But as there was a hook that anybody acting as a trustworthy partner would have mentioned, I parted with them — potentially for life. You know, things like "click here to sign up, sacrifice your newborn on a fullmoon night while reciting the last 3 digits of pi to cancel" I don't particular care whether their A/B test captures that potential aspect of customer (dis)satisfaction, but I am not sure how it would.

richardw 2y ago

I designed an internal system that optimises for long term outcomes. We do nothing based on whether you click “upgrade”. We look at the net change over time, including impact to engagement and calls to support months later and whether you leave 6 months after upgrading. Most of the nudges are purely for the customer’s benefit because it’ll improve lifetime value.

sethammons 2y ago
You could only be measuring in aggregate, no? Overall signal could be positive but one element happens to be negative while another is overly positive.
- richardw 2y ago
  Well, adjusting nudges in aggregate but diced in various ways. Measured very much not in aggregate. We’d see positive and negative outcomes roll in over multiple years and want it per identifier (an individual). I’ve heard of companies generating a model per person but we didn’t. A silly amount of work but honestly lots of value. Experimentation optimising for short term goals (eg upgrade) is such a bad version of this, it’s just all that is possible with most datasets.
bitshiftfaced 2y ago
That's the only thing I was thinking with their A/B test. The calculator might immunize against unhappy customers later on. I think they could've looked at something like the percentage of customers who leave one or two billing cycles later.
- fwip 2y ago
  Unfortunately, there's never enough time to run a proper experiment - we want answers now! Who cares if they're the right answers. Short-termism can't wait two months.

otherme123 2y ago

> Of course if you hide the fact that your product might cost a lot of money from your users, more of them will sign up The problem with their calculator was that the users introduced slightly wrong data, or misunderstand what means some metric, and suddenly a 1000x the real price was shown. Their dilemma was "how to fix those cases", and the solution was "get rid of the messy calculator". But they are not hidding a 1000x cost, they are avoiding losing users that get a wrong 1000x quote.

xorcist 2y ago
Let's not take a PR piece completely at face value. There's probably a bit of both, at the very least.
- PaulHoule 2y ago
  In my mind Pinecone is an exemplary example of modern "social media marketing" for a technology company. They started on vector search at a time when RAG in its current form wasn't a thing; there were just a few search products based on vector search (like a document embedding-based search engine for patents that I whipped into shape to get in front of customers) and if you were going to use vector search you'd need to develop your own indexing system in house or just do a primitive full scan (sounds silly but it's a best-case scenario for full scan and vector indexes do not work as well as 1-d indexes) They blogged frequently and consistently about the problem they were working on with heart, which I found fascinating because I'd done a lot of reading about the problem in the mid ought's. Thus Pinecone had a lot of visibility for me, although I don't know if I am really their market. (No budget for a cloud system, full scan is fine for my 5M document collection right now, I'd probably try FAISS if it wasn't.) Today their blog looks more than it used to which makes it a little harder for me to point out how awesome their blog was in the beginning but this post is definitely the kind of post that they made when they were starting out. I'm sure it has been a big help in finding employees, customers and other allies.
  - gk1 2y ago
    Thank you. :) I don’t think of it as social media marketing but more of helping our target audience learn useful things. Yes that requires they actually find the articles which means sharing it on social, being mindful of SEO, and so on. Probably our learning center is what you’re thinking of. https://www.pinecone.io/learn/ https://www.pinecone.io/learn/ … The blog is more of a news ticker for product and company news.

raverbashing 2y ago

> so we dug into it and realized the calculator was far more confusing and sensitive than we thought. One slight misinterpretation and wrong input and you'd get an estimate that's overstated by as much as 1,000x. They should have looked into this to see how to make it more obvious or more reflective of "regular use case" Their sliders there are not too detailed. For example, what are namespaces, how many would a typical use need? Is 100 too much or too little? And if this is one of the variables that is too sensitive they would need to represent this in a different way

scott_w 2y ago

That’s why you need domain experts and clear explanations and hypotheses before you experiment, otherwise you’re throwing shit at a wall to see what sticks. Companies can continue to monitor cohorts to compare retention to check the potential outcomes you highlighted.

hinkley 2y ago

Also clicking through is not a good thing if it doesn’t result in revenue! Why do I want to render eight pages for someone who will never give us money if I can find that out in three?