6 ms·

Why we’re pausing our Pay Later program

83 points by leeny 4y ago

WheatMillington 4y ago
I've never heard of this organisation, and maybe it's because I don't work in tech but I don't really understand why they need to exist. But in any case, I'm entirely unsurprised to see the customer response to these changes. It comes off as incredibly predatory - AB testing the perfect, most "frictionless" experience which is obviously going to lead people to signing up without understanding they're committing to hundreds of dollars of future expenses. Gross.
- js4 4y ago
  Optimizing conversion rate is a best practice. Dont hate the player. Hate the game.
  - WheatMillington 4y ago
    [flagged]
  - elil17 4y ago
    Due diligence is a best practice for lenders.
  - remh 4y ago
    https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
- dahdum 4y ago
  There are many dark patterns out there but what they've described and the UI screenshots don't scream all that predatory to me. It looks like a good faith effort that went awry. They're certainly handling it well afterward. I could have made the same mistake. Their customers are highly technical engineers applying to FAANGs and other top companies, I'd personally expect a greater understanding of terms and payment details from that cohort.
  - WheatMillington 4y ago
    >Their customers are highly technical engineers applying to FAANGs and other top companies They don't, as far as I can tell, vet their customers. They deliberately cast as wide a net as possible, so it was entirely foreseeable they were going to hook some less sophisticated users.
    - joecot 4y ago
      Right, the issue is that they're going to hook people who are aspirationally looking to work at a FAANG and will not by any stretch of the imagination pass their interviewing process. They picked Pay Later because they figured they could pay for it later if they got the job, or given the funnel hacking discussed in the post, they never considered that they would have to pay for it later even if they didn't get the job. This is a lower scale and lower cost version of the for profit bootcamps and IT schools. Yes, if you go in with the tools you need to succeed, you will probably gain something and get a good start at a FAANG or similar. The programs cannot work miracles and have a vested interest in casting as wide a net as possible even though that's a disservice to their customers. This service figured out they need to reign in that net to confirm people can actually pay their cost.
    - bombcar 4y ago
      Exactly - this is a HUGE common problem at startups, describing their ideal customer who doesn't really exist and completely ignoring their actual customer base and what it is made up of. The vast majority of the people signing up are going to be wannabe engineers - what percentage are actually highly technical needs to be determined.
  - ianbutler 4y ago
    I wouldn’t describe most of the interviewing body in software as “highly technical” that’s why there is such an issue of finding qualified talent. I’d personally bet a good deal of people would not and did not understand what they were getting themselves into. Tech has become popular because of the salaries, the amount of people interviewing has heavily increased, but the talent levels are quite diluted. Especially because any engineer worth their salt is off the market in under a month right now, if not shorter.
- dmkirwan 4y ago
  It shouldn't need to exist but unfortunately interviewing is still a separate skill that engineers need to learn. People can get so caught up in the numbers on a dashboard or AB test that they forget that there's a human on the other end that just got duped into a 4 figure expense. Gross indeed!
sbierwagen 4y ago
>We launched our Pay Later Program at the end of December 2021. As we usually do with new features, we A/B tested a few different variants, with the intent of increasing enrollment as much as possible each time. Another example to add to the annals of A/B horror stories. Sample size too small, test length time too short, extremely relevant KPIs like chargebacks or fraud complaints not included. A/B the background color of the home page all you want, but A/B testing legal disclaimers is going to be, uh, risky. And this is a very tired and boring thing to say, but startups are not AAAMM. If you're on a product team at Meta you can do ten A/B tests a week because tens of millions of people are using your product. If you're an early stage startup looking for market fit, a A/B test really will authentically take weeks or months to get enough data. If you're on a short runway I could imagine it would be painful and unpleasant to just sit on your hands the whole time, but otherwise you wander down blind alleys like the above.
- emeraldd 4y ago
  Would it be safe to say that A/B testing should only be used for things that can "afford to fail/be wrong"? I'm not saying don't test in other cases, but that this particular technique is not a good fit for situations where the consequences of a mistake are high. Some other kind of validation (rigorous or not) is necessary in those instances.
  - akiselev 4y ago
    IMO it's only safe to A/B test things where the product vision doesn't matter one bit. Stuff like where to place the button on the screen to drive conversions on a landing page or what copy to use to target X market segment. Edit: A better way to put that might be - only use A/B testing to make tactical decisions, but never use it to decide strategy or logistics.
  - uoaei 4y ago
    A/B tests are statistical tests. So you should pre-register an appropriate confidence interval and stick to that. The width of your confidence interval should be relevant to the risks you are taking.
  - 4y ago
JamesBarney 4y ago
I appreciate the candor that Aline has shown in this write up. >Practically speaking, I’ve learned that creating some amount of friction is necessary when you’re asking people to promise to pay you ~$1000 in the future. Removing that friction can create short-term wins but may hurt you (and disappoint your users) in the long run. But people are going to be confused from a "click this button and you owe us a grand" UI seems blindingly obvious. I suspect they got greedy and thought it wouldn't going to be as big of a deal as it turned out to be.
- librish 4y ago
  I think that's an uncharitable way of reading this. Looking at the UI the TLDR is pretty clear, plus there's the friction of putting down a credit card.
  - treis 4y ago
    It depends on what the screen before this one looked like. I'd definitely blow pass the TCs they showed. So it would have to have been very explicit in the screen before that you were agreeing to pay something.
    - librish 4y ago
      You would put down your credit card on a website without reading anything on the page? Why?
      - treis 4y ago
        Because most T&Cs are boilerplate nonsense Because I have CC disputes as a backup Because I'm lazy
        librish 4y ago
        OK but this wasn't a 40 page T&C, this was a text box with three statements. What would you think you were getting when putting in your credit card?
jacques_chester 4y ago
Never mind Charles. Cash is king again.
bluepnume 4y ago
The language really does sound like "pay only if/when you get a job".
kurtreed 4y ago
The fact that a service exists to make you better at interviews implies that interviews don't evaluate competency. If interviews evaluated competency then the way to get better at interviews would be the same as the way to get better at software development (books, projects, etc.). I feel like improving my competency is worthwhile but improving my interviewing is rent seeking.
- dmkirwan 4y ago
  I couldn't agree more. Most interviews assess your ability to interview well, not do the job well. Stanford even run a class on the software engineering interview! I'm currently working on a product to help companies to build realistic interviews, and I'm seeing some other startups focus on this too, so hopefully it will become a thing of the past. The Stanford course, for the curious: https://web.stanford.edu/class/cs9/ https://web.stanford.edu/class/cs9/ The product we're working on: https://devscreen.io https://devscreen.io
  - donretag 4y ago
    Great implementation for solving a problem that is often discussed, but never executed.!I have always thought that paired programming works better than whiteboarding. Do you have any reference clients? Would love to interview at such companies.
    - piva00 4y ago
      Pair programming has been my most successful and least stressful way of interviewing. I can display all my skills in a short amount of time and, most important to me, in a realistic and fun way. I don't feel so pressured when I interview through pair programming which helps a lot to showcase my normal self. It's easy to display your technical skills (and any gaps in it), soft skills (such as communication, perseverance, thought process, etc.), problem-solving skills and so on. I'd much rather spend a 5-hour interview just pairing with different people than going through rounds and rounds of whiteboarding. Whiteboarding makes me feel like I'm a lab subject, being prodded in different directions while the experimenters take notes on their clipboards. It's extremely artificial to evaluate any real-life scenario.
neilv 4y ago
The screenshots of the deal are in gray-on-white text, which looks like the writer doesn't want the reader to read it. Additionally, they wrap up the gray/small print with the final word, on what the reader is supposed to think, in boldface, and delimited by triple asterisks: "94% of our users find a job within 4 months of starting to practice, and of those, the majority get offers from FAANG". There seems to be a conscious component of persuasion in that, so I think no surprise if some customers later felt they'd been misled or manipulated.
happyopossum 4y ago
> Then hiring basically froze when COVID-19 happened, and to survive, we started charging engineers Err, this doesn’t line up with any of my experience, nor that of any of my peers or the multiple employers I worked for throughout the pandemic. Maybe things slowed down for a month or so, but there was so much job mobility and choice it was bonkers!
londons_explore 4y ago
I wonder how many of these users thought "It's just a web page, it's not going to send around a debt collector, I won't ever need to pay". Those types of users tend to be put off by PDF documents you have to print, sign and return, because then they feel it's more likely that they will get a court summons for not paying...
londons_explore 4y ago
I'm really amazed they got a 90% collection rate... If you'd have asked me to guess, I would reckon only 30% will pay. A chunk will have used credit cards with no balance, or cards about to expire. A chunk will dispute charges. A chunk will never see your emails because they land in spam.
londons_explore 4y ago
The sign-up form at first glance really does look like "we are giving you a $1024 free credit to get started on our platform"

83 points by leeny 4y ago

I've never heard of this organisation, and maybe it's because I don't work in tech but I don't really understand why they need to exist. But in any case, I'm entirely unsurprised to see the customer response to these changes. It comes off as incredibly predatory - AB testing the perfect, most "frictionless" experience which is obviously going to lead people to signing up without understanding they're committing to hundreds of dollars of future expenses. Gross.

js4 4y ago
Optimizing conversion rate is a best practice. Dont hate the player. Hate the game.
- WheatMillington 4y ago
  [flagged]
- elil17 4y ago
  Due diligence is a best practice for lenders.
- remh 4y ago
  https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
dahdum 4y ago
There are many dark patterns out there but what they've described and the UI screenshots don't scream all that predatory to me. It looks like a good faith effort that went awry. They're certainly handling it well afterward. I could have made the same mistake. Their customers are highly technical engineers applying to FAANGs and other top companies, I'd personally expect a greater understanding of terms and payment details from that cohort.
- WheatMillington 4y ago
  >Their customers are highly technical engineers applying to FAANGs and other top companies They don't, as far as I can tell, vet their customers. They deliberately cast as wide a net as possible, so it was entirely foreseeable they were going to hook some less sophisticated users.
  - joecot 4y ago
    Right, the issue is that they're going to hook people who are aspirationally looking to work at a FAANG and will not by any stretch of the imagination pass their interviewing process. They picked Pay Later because they figured they could pay for it later if they got the job, or given the funnel hacking discussed in the post, they never considered that they would have to pay for it later even if they didn't get the job. This is a lower scale and lower cost version of the for profit bootcamps and IT schools. Yes, if you go in with the tools you need to succeed, you will probably gain something and get a good start at a FAANG or similar. The programs cannot work miracles and have a vested interest in casting as wide a net as possible even though that's a disservice to their customers. This service figured out they need to reign in that net to confirm people can actually pay their cost.
  - bombcar 4y ago
    Exactly - this is a HUGE common problem at startups, describing their ideal customer who doesn't really exist and completely ignoring their actual customer base and what it is made up of. The vast majority of the people signing up are going to be wannabe engineers - what percentage are actually highly technical needs to be determined.
- ianbutler 4y ago
  I wouldn’t describe most of the interviewing body in software as “highly technical” that’s why there is such an issue of finding qualified talent. I’d personally bet a good deal of people would not and did not understand what they were getting themselves into. Tech has become popular because of the salaries, the amount of people interviewing has heavily increased, but the talent levels are quite diluted. Especially because any engineer worth their salt is off the market in under a month right now, if not shorter.
dmkirwan 4y ago
It shouldn't need to exist but unfortunately interviewing is still a separate skill that engineers need to learn. People can get so caught up in the numbers on a dashboard or AB test that they forget that there's a human on the other end that just got duped into a 4 figure expense. Gross indeed!

sbierwagen 4y ago

>We launched our Pay Later Program at the end of December 2021. As we usually do with new features, we A/B tested a few different variants, with the intent of increasing enrollment as much as possible each time. Another example to add to the annals of A/B horror stories. Sample size too small, test length time too short, extremely relevant KPIs like chargebacks or fraud complaints not included. A/B the background color of the home page all you want, but A/B testing legal disclaimers is going to be, uh, risky. And this is a very tired and boring thing to say, but startups are not AAAMM. If you're on a product team at Meta you can do ten A/B tests a week because tens of millions of people are using your product. If you're an early stage startup looking for market fit, a A/B test really will authentically take weeks or months to get enough data. If you're on a short runway I could imagine it would be painful and unpleasant to just sit on your hands the whole time, but otherwise you wander down blind alleys like the above.

emeraldd 4y ago
Would it be safe to say that A/B testing should only be used for things that can "afford to fail/be wrong"? I'm not saying don't test in other cases, but that this particular technique is not a good fit for situations where the consequences of a mistake are high. Some other kind of validation (rigorous or not) is necessary in those instances.
- akiselev 4y ago
  IMO it's only safe to A/B test things where the product vision doesn't matter one bit. Stuff like where to place the button on the screen to drive conversions on a landing page or what copy to use to target X market segment. Edit: A better way to put that might be - only use A/B testing to make tactical decisions, but never use it to decide strategy or logistics.
- uoaei 4y ago
  A/B tests are statistical tests. So you should pre-register an appropriate confidence interval and stick to that. The width of your confidence interval should be relevant to the risks you are taking.
- 4y ago

JamesBarney 4y ago

I appreciate the candor that Aline has shown in this write up. >Practically speaking, I’ve learned that creating some amount of friction is necessary when you’re asking people to promise to pay you ~$1000 in the future. Removing that friction can create short-term wins but may hurt you (and disappoint your users) in the long run. But people are going to be confused from a "click this button and you owe us a grand" UI seems blindingly obvious. I suspect they got greedy and thought it wouldn't going to be as big of a deal as it turned out to be.

librish 4y ago
I think that's an uncharitable way of reading this. Looking at the UI the TLDR is pretty clear, plus there's the friction of putting down a credit card.
- treis 4y ago
  It depends on what the screen before this one looked like. I'd definitely blow pass the TCs they showed. So it would have to have been very explicit in the screen before that you were agreeing to pay something.
  - librish 4y ago
    You would put down your credit card on a website without reading anything on the page? Why?
    - treis 4y ago
      Because most T&Cs are boilerplate nonsense Because I have CC disputes as a backup Because I'm lazy
      - librish 4y ago
        OK but this wasn't a 40 page T&C, this was a text box with three statements. What would you think you were getting when putting in your credit card?

jacques_chester 4y ago

Never mind Charles. Cash is king again.

bluepnume 4y ago

The language really does sound like "pay only if/when you get a job".

kurtreed 4y ago

The fact that a service exists to make you better at interviews implies that interviews don't evaluate competency. If interviews evaluated competency then the way to get better at interviews would be the same as the way to get better at software development (books, projects, etc.). I feel like improving my competency is worthwhile but improving my interviewing is rent seeking.

dmkirwan 4y ago
I couldn't agree more. Most interviews assess your ability to interview well, not do the job well. Stanford even run a class on the software engineering interview! I'm currently working on a product to help companies to build realistic interviews, and I'm seeing some other startups focus on this too, so hopefully it will become a thing of the past. The Stanford course, for the curious: https://web.stanford.edu/class/cs9/ https://web.stanford.edu/class/cs9/ The product we're working on: https://devscreen.io https://devscreen.io
- donretag 4y ago
  Great implementation for solving a problem that is often discussed, but never executed.!I have always thought that paired programming works better than whiteboarding. Do you have any reference clients? Would love to interview at such companies.
  - piva00 4y ago
    Pair programming has been my most successful and least stressful way of interviewing. I can display all my skills in a short amount of time and, most important to me, in a realistic and fun way. I don't feel so pressured when I interview through pair programming which helps a lot to showcase my normal self. It's easy to display your technical skills (and any gaps in it), soft skills (such as communication, perseverance, thought process, etc.), problem-solving skills and so on. I'd much rather spend a 5-hour interview just pairing with different people than going through rounds and rounds of whiteboarding. Whiteboarding makes me feel like I'm a lab subject, being prodded in different directions while the experimenters take notes on their clipboards. It's extremely artificial to evaluate any real-life scenario.

neilv 4y ago

The screenshots of the deal are in gray-on-white text, which looks like the writer doesn't want the reader to read it. Additionally, they wrap up the gray/small print with the final word, on what the reader is supposed to think, in boldface, and delimited by triple asterisks: "94% of our users find a job within 4 months of starting to practice, and of those, the majority get offers from FAANG". There seems to be a conscious component of persuasion in that, so I think no surprise if some customers later felt they'd been misled or manipulated.

happyopossum 4y ago

> Then hiring basically froze when COVID-19 happened, and to survive, we started charging engineers Err, this doesn’t line up with any of my experience, nor that of any of my peers or the multiple employers I worked for throughout the pandemic. Maybe things slowed down for a month or so, but there was so much job mobility and choice it was bonkers!

londons_explore 4y ago

I wonder how many of these users thought "It's just a web page, it's not going to send around a debt collector, I won't ever need to pay". Those types of users tend to be put off by PDF documents you have to print, sign and return, because then they feel it's more likely that they will get a court summons for not paying...

londons_explore 4y ago

I'm really amazed they got a 90% collection rate... If you'd have asked me to guess, I would reckon only 30% will pay. A chunk will have used credit cards with no balance, or cards about to expire. A chunk will dispute charges. A chunk will never see your emails because they land in spam.

londons_explore 4y ago

The sign-up form at first glance really does look like "we are giving you a $1024 free credit to get started on our platform"