6 ms·

Isn't it what's everybody does in the industry?! Every single place that I ever worked at in a past 20 years tests payments using real cards and real API endpo

Isn't it what's everybody does in the industry?! Every single place that I ever worked at in a past 20 years tests payments using real cards and real API endpoints. Yes, refunds cost a few pennies and sometimes can't be automated, but most payment providers simply do not offer testing APIs of a sufficient quality. Situations when a testing endpoint has one set of bugs not found on production and vice versa used to be so ubiquitous in mid-2000 to mid-2010s, that many teams make a choice agains using testing endpoints altogether - it's too much work to work around bugs unique to the environment that no real customers actually hit. And now the whole generation of developers grew in a world of bad testing APIs of PayPal, Authorize.net, BrainTree, BalancedPayments (remember them?), early Stripe, etc. So, now it became an institutional knowledge: "do not use testing endpoints for payments". To be exact, people often start using testing endpoints for early stages of development when you don't have any payment code at all, but before the product launch things get switched to production endpoints and from that point on testing endpoints aren't used at all. Even for local development people usually use corporate cards if necessary. I have a suspicion that things may be different in the US, with many payment providers' testing environments simulate a typical domestic US scenario: credit cards and not debit, no 3d-secure, no strict payment jurisdiction restrictions, etc.

weinzierl 2y ago
"Isn't it what's everybody does in the industry?!" Everybody, some do it manually, some let their QA people use their private credit cards - or so I've heard.
boesboes 2y ago
Eh, no? I've never tested payment code using real payments. Ever. The idea of doing it with real payments is pretty out there in my book even :) Then again, every payment provider/bank I've integrated with, had decent testing end-points and we often even support them in production. i.e, you can select a staging/testing env of you provider to test order flow or whatever.
- willcipriano 2y ago
  Then you just had the customer test it with a real payment. That's pretty out there in my book.
  - BonoboIO 2y ago
    Amen. If you don’t test it, you will test it with your customer and will eventually fail.
hypeatei 2y ago
Is doing a smoke test in prod with corporate cards bad practice? We are rolling out subscriptions with Stripe and an internal business unit is will actually be using the service so they put it on a company card. Basically they're our first live customer to test all the prod systems. No refunds or anything.
- mickeyp 2y ago
  No, it is not bad practice. Only developers who don't care about money actually coming through the door -- the same ones that get caught up in a local maxima trying to perfect the imperfect -- say that it's bad and that you should not do it.
  - kbolino 2y ago
    Regardless of what developers think, the payment providers generally forbid it. For example, Stripe says: > Don’t use real card details. The Stripe Services Agreement prohibits testing in live mode using real payment method details. Use your test API keys and the card numbers below. https://docs.stripe.com/testing https://docs.stripe.com/testing
    - mjr00 2y ago
      I think you're misunderstanding here; people are talking about a smoke "test" using a real credit card against the real production payment system, using production API keys/authentication/etc, with real money moving around. No payment provider forbids that.
      - anonymoushn 2y ago
        That's precisely what they all forbid?
        mjr00 2y ago
        They forbid using a valid credit card to make a purchase on a production system?
Rygian 2y ago
How do those places ever pass a PCI audit? One of the first things the auditor asks is "please show me proof that your testing is never done with real credit cards" (Unless they're getting their test environments PCI certified, which sounds like a waste of money.)
- hermanradtke 2y ago
  Don’t do it in staging/test environment. As a sibling commented stated: smoke test in production with corporate cards.
  - deleted 2y ago
    [deleted]
- bdcravens 2y ago
  Most companies are never PCI audited because they're using a provider who already has been (like Stripe)
  - jabart 2y ago
    At volume you can no longer self-certify to be PCI SAQ-D. There are limits based on transaction count or volume.
    - bdcravens 2y ago
      Makes sense. What are those volume limits like? I suspect the lion's share of companies like Stripe are those under those limits (since the largest companies are willing to trade simplicity for better rates) edit: according to a quick search, it looks like it's 6M transactions/year to require an audit vs self-assessment
- deleted 2y ago
  [deleted]
- closeparen 2y ago
  >Every single place that I ever worked at in a past 20 years tests payments using real cards and real API endpoints. Yes, refunds cost a few pennies and sometimes can't be automated, but most payment providers simply do not offer testing APIs of a sufficient quality. I think this means their own real-money credit cards, that they spend a few bucks on for testing purposes. Not customer credit-card data.
hnthrow289570 2y ago
>Isn't it what's everybody does in the industry?! We built against Stripe's sandbox and never had to test in production, so I never had to use a real credit card. It may have happened when going live for the first time, but that would have been two/three charges with one time payments and recurring payments (hardly what you'd call robust testing). Issues observed in production can usually be reproduced in the sandbox. There's some other caveats between the environments though that I'm forgetting but I don't think we ran into those. We also had an ACH payment provider (add your bank account, verify it, we deduct from it, etc.) that also had a sandbox and had no issues there either.
- arkh 2y ago
  > We built against Stripe's sandbox and never had to test in production, so I never had to use a real credit card. Did it, got bit in the ass when some workflow was disabled in production and not in their sandbox. I don't recall the exact thing but always fun to push into production all sure of yourself to have to rollback fast and ponder why you get some fun message in your logs. At least the error message was clear about what we were doing being available only on the sandbox. It was some years ago so I don't remember if it was in Stripe Connect or during the mandatory 2FA rollout.
  - ambicapter 2y ago
    > got bit in the ass when some workflow was disabled in production and not in their sandbox What Stripe workflow was this? Or was it specific to your codebase?
    - danielheath 2y ago
      Stripe is the best I've used, but it has a ton of issues to this day: A) Account settings are specified separately in prod / staging B) Differences between those environments are not automatically reported in a useful way C) Only one staging env per customer. Want to check what a new setting will do? Every developer is getting that setting turned on.
ghosty141 2y ago
Standard practice is to use testing api for development and the real api for verification.
nunez 2y ago
Absolutely not. Production is still an impenetrable fortress at a lot of places, or at least it's perceived to be.
zitterbewegung 2y ago
From any engineering post I have seen about Uber it seems like some deceptive marketing / hiring tactic when it is analyzed . They seem to repurpose standard practices into a blog post (this one might sounds like it has been derived from one).
Tade0 2y ago
> but most payment providers simply do not offer testing APIs of a sufficient quality. Moreover, sometimes they vehemently oppose testing via real payments and reserve the right to cancel the contract should this happen. To this day I have a distaste for working with payments.
ukd1 2y ago
My last company, Rainforest QA, developed issuing virtual credit-card numbers for just this purpose - kinda like the ones used for privacy.com. Customers use them for testing in prod. Simple, effective - and testing the exact same flow as customers. Before this, we found a lot of teams using either their own corp cards, or pre-paid visa type things. All, a pain to manage balance wise. Seemingly, the biggest problem left with doing this is production metrics; these transactions in prod tend to affect the main metrics - either your own, or payment related things. I can't find anything current, but - https://www.businesswire.com/news/home/20180417005414/en/Rainforest-QA-Brings-Simple-and-Secure-Online-Payment-Testing-to-Ecommerce-Industry-with-Virtual-Credit-Cards https://www.businesswire.com/news/home/20180417005414/en/Rai... covers it.
rileymichael 2y ago
> Situations when a testing endpoint has one set of bugs not found on production and vice versa used to be so ubiquitous in mid-2000 to mid-2010s Honestly that's still the case, at least with Adyen. At $pastJob we had a pretty robust regression suite that'd regularly run into breaking changes in their test environment (against _old_ versioned api endpoints!). We seriously questioned whether or not anyone else used them since they were never aware until we submitted tickets. This also falls apart as soon as you use any of their "special" features that require an account rep to enable, they just don't work in the sandbox. Another pain point is that the test payment methods offered are static. You can't setup cards for specific scenarios, e.g. tokenize, successful payment, then it expires -- you can only test an expired card as a one-off.
- whstl 2y ago
  I just ran a couple hours ago into such a bug with Adyen, a couple payment methods have completely different behaviours in test vs production, and we ended up having to test in prod anyway.
netdevnet 2y ago
Nooope! Services like stripe let you test the payment workflow in their test environment which works exactly like the production one but without the payments going outside Stripe. Let us not normalise bad practices
serial_dev 2y ago
I've worked on adding Google Pay / Apple Pay to the mobile app of a large European ecommerce company, and that's more of less how we went about it. Start with the sandbox / test environments, once you get reasonable responses end to end, release the thing behind a feature flag. Backend moves slowly, so you add stuff to the mobile app that really belongs to the backend, but f it, because it's the only way you meet the deadline, you will convince someone on the backend later (aka never) that it's backend responsibility. Ask (pressure) the developers into buying some cheap stuff from your shop with their own credit cards, because the company is a behemoth and approving real credit cards for testing would just take ages and you want to release yesterday. It annoys you, but you realize that 5x5 euros is worth getting this done rather than start fighting a losing battle against company processes. Cancelation is possible, but it will take a couple of days. If there's any issue, you debug it across a bunch of teams and/or companies. Things start to work most of the time, time to release the stuff to x percent of your users. Check analytics and error logs frequently. Some production users got their payments through, increase rollout percentage. You discover more and more undocumented error codes, you improve the error messages to the users so that they don't retry 10 times with a card without sufficient funds. After a couple weeks, things start to stabilize, you move on to a new feature... The test environments were so complicated and had so many caveats that whenever I had to do something, I had to re-read the docs and our notes to know all the "traps" we already discovered. For the people who didn't work on this payment feature from start to finish, testing in the officially recommended test channels were hopeless.
- CapstanRoller 2y ago
  >Ask (pressure) the developers into buying some cheap stuff from your shop with their own credit cards This is illegal. I've always refused such "requests" and asked for a company expense card.
  - ceejayoz 2y ago
    Illegal in what jurisdiction?
    - 2y ago
singleshot_ 2y ago
I seem to remember about 4111 1111 1111 1111 times that I tested a payment system with a card that wasn’t real, although I acknowledge that when I was done convincing myself that I was a good programmer, I would almost always be disavowed of this notion after using a real number.
hot_gril 2y ago
This is also true for cryptocurrency. There's always a testnet, you use it whenever possible cause xact fees are high, but it never works the same as real.
hoffs 2y ago
Doing penny testing yourself is different from letting a chunk of your user base test it
ComputerGuru 2y ago
Stripe considers using a real card to test a zero-tolerance fireable offence. I, as the name on record for an organization with Stripe, made an actual, legitimate payment to said organization (and did not refund it). Stripe's automated system caught the payment and terminated my account automatically by the time I woke up the next morning; fortunately I was able to reach a real human and explain that in addition to working for the non-profit, I was also donating to it, and they begrudgingly restored the account.
- loremaster 2y ago
  Stripe will take pretty much any excuse to terminate an account and pocket the money. At this point it might as well be part of their business model.
- aceofspades19 2y ago
  My company has hundreds/thousands of ecommerce clients, of which a huge portion use Stripe and we do real card testing every time we deploy code that could affect payments and its never been an issue.
  - ComputerGuru 2y ago
    It’s pretty clearly stipulated in their TOS, for what it’s worth.
    - aceofspades19 2y ago
      I don't see anything referring to testing in https://stripe.com/en-ca/legal/ssa https://stripe.com/en-ca/legal/ssa Other than that you are provided test keys and live keys.
- pheatherlite 2y ago
  But why? Why testing with real payment methods is a bad idea for stripe?
firesteelrain 2y ago
PA DSS/PCI DSS standards say you aren’t supposed to test with real cards. It’s a certification criteria. I had to make my own PCI DSS (Payment Card Industry Data Security Standard) Requirement 6.4.3 of the PCI DSS states: "Production data (live PANs) are not used for testing or development." This requirement is aimed at ensuring that real cardholder data (Primary Account Numbers or PANs) is not used in non-production environments, such as testing or development environments, to minimize the risk of exposure and unauthorized access. PA DSS (Payment Application Data Security Standard) Requirement 6.3.4 of the PA DSS states: "Production data (real PANs, track data, or other real cardholder data) is not used for testing or development."
sebastiennight 2y ago
I built software for the self-service automats of a very large European rail company, and I remember spending whole afternoons testing every possible scenario listed in our testing book. Those required manually purchasing tickets to many different cities, using the credit card terminal on the machine. They even had fake Discover, VISA, Mastercards you were to use to make those test purchases (to this day I don't know who came up with those fake cards and how that part of the system worked). But no, not everybody tests payment in production. It was a while ago, but I don't think they'd have switched the entire philosophy by that much since then.
TeeWEE 2y ago
Testing against the test environment works well for us. Even for terminal in person payments. Even makes it much easier to simulate edge cases.

weinzierl 2y ago

"Isn't it what's everybody does in the industry?!" Everybody, some do it manually, some let their QA people use their private credit cards - or so I've heard.

boesboes 2y ago

Eh, no? I've never tested payment code using real payments. Ever. The idea of doing it with real payments is pretty out there in my book even :) Then again, every payment provider/bank I've integrated with, had decent testing end-points and we often even support them in production. i.e, you can select a staging/testing env of you provider to test order flow or whatever.

willcipriano 2y ago
Then you just had the customer test it with a real payment. That's pretty out there in my book.
- BonoboIO 2y ago
  Amen. If you don’t test it, you will test it with your customer and will eventually fail.

hypeatei 2y ago

Is doing a smoke test in prod with corporate cards bad practice? We are rolling out subscriptions with Stripe and an internal business unit is will actually be using the service so they put it on a company card. Basically they're our first live customer to test all the prod systems. No refunds or anything.

mickeyp 2y ago
No, it is not bad practice. Only developers who don't care about money actually coming through the door -- the same ones that get caught up in a local maxima trying to perfect the imperfect -- say that it's bad and that you should not do it.
- kbolino 2y ago
  Regardless of what developers think, the payment providers generally forbid it. For example, Stripe says: > Don’t use real card details. The Stripe Services Agreement prohibits testing in live mode using real payment method details. Use your test API keys and the card numbers below. https://docs.stripe.com/testing https://docs.stripe.com/testing
  - mjr00 2y ago
    I think you're misunderstanding here; people are talking about a smoke "test" using a real credit card against the real production payment system, using production API keys/authentication/etc, with real money moving around. No payment provider forbids that.
    - anonymoushn 2y ago
      That's precisely what they all forbid?
      - mjr00 2y ago
        They forbid using a valid credit card to make a purchase on a production system?

Rygian 2y ago

How do those places ever pass a PCI audit? One of the first things the auditor asks is "please show me proof that your testing is never done with real credit cards" (Unless they're getting their test environments PCI certified, which sounds like a waste of money.)

hermanradtke 2y ago
Don’t do it in staging/test environment. As a sibling commented stated: smoke test in production with corporate cards.
- deleted 2y ago
  [deleted]
bdcravens 2y ago
Most companies are never PCI audited because they're using a provider who already has been (like Stripe)
- jabart 2y ago
  At volume you can no longer self-certify to be PCI SAQ-D. There are limits based on transaction count or volume.
  - bdcravens 2y ago
    Makes sense. What are those volume limits like? I suspect the lion's share of companies like Stripe are those under those limits (since the largest companies are willing to trade simplicity for better rates) edit: according to a quick search, it looks like it's 6M transactions/year to require an audit vs self-assessment
deleted 2y ago
[deleted]
closeparen 2y ago
>Every single place that I ever worked at in a past 20 years tests payments using real cards and real API endpoints. Yes, refunds cost a few pennies and sometimes can't be automated, but most payment providers simply do not offer testing APIs of a sufficient quality. I think this means their own real-money credit cards, that they spend a few bucks on for testing purposes. Not customer credit-card data.

hnthrow289570 2y ago

>Isn't it what's everybody does in the industry?! We built against Stripe's sandbox and never had to test in production, so I never had to use a real credit card. It may have happened when going live for the first time, but that would have been two/three charges with one time payments and recurring payments (hardly what you'd call robust testing). Issues observed in production can usually be reproduced in the sandbox. There's some other caveats between the environments though that I'm forgetting but I don't think we ran into those. We also had an ACH payment provider (add your bank account, verify it, we deduct from it, etc.) that also had a sandbox and had no issues there either.

arkh 2y ago
> We built against Stripe's sandbox and never had to test in production, so I never had to use a real credit card. Did it, got bit in the ass when some workflow was disabled in production and not in their sandbox. I don't recall the exact thing but always fun to push into production all sure of yourself to have to rollback fast and ponder why you get some fun message in your logs. At least the error message was clear about what we were doing being available only on the sandbox. It was some years ago so I don't remember if it was in Stripe Connect or during the mandatory 2FA rollout.
- ambicapter 2y ago
  > got bit in the ass when some workflow was disabled in production and not in their sandbox What Stripe workflow was this? Or was it specific to your codebase?
  - danielheath 2y ago
    Stripe is the best I've used, but it has a ton of issues to this day: A) Account settings are specified separately in prod / staging B) Differences between those environments are not automatically reported in a useful way C) Only one staging env per customer. Want to check what a new setting will do? Every developer is getting that setting turned on.

ghosty141 2y ago

Standard practice is to use testing api for development and the real api for verification.

nunez 2y ago

Absolutely not. Production is still an impenetrable fortress at a lot of places, or at least it's perceived to be.

zitterbewegung 2y ago

From any engineering post I have seen about Uber it seems like some deceptive marketing / hiring tactic when it is analyzed . They seem to repurpose standard practices into a blog post (this one might sounds like it has been derived from one).

Tade0 2y ago

> but most payment providers simply do not offer testing APIs of a sufficient quality. Moreover, sometimes they vehemently oppose testing via real payments and reserve the right to cancel the contract should this happen. To this day I have a distaste for working with payments.

ukd1 2y ago

My last company, Rainforest QA, developed issuing virtual credit-card numbers for just this purpose - kinda like the ones used for privacy.com. Customers use them for testing in prod. Simple, effective - and testing the exact same flow as customers. Before this, we found a lot of teams using either their own corp cards, or pre-paid visa type things. All, a pain to manage balance wise. Seemingly, the biggest problem left with doing this is production metrics; these transactions in prod tend to affect the main metrics - either your own, or payment related things. I can't find anything current, but - https://www.businesswire.com/news/home/20180417005414/en/Rainforest-QA-Brings-Simple-and-Secure-Online-Payment-Testing-to-Ecommerce-Industry-with-Virtual-Credit-Cards https://www.businesswire.com/news/home/20180417005414/en/Rai... covers it.

rileymichael 2y ago

> Situations when a testing endpoint has one set of bugs not found on production and vice versa used to be so ubiquitous in mid-2000 to mid-2010s Honestly that's still the case, at least with Adyen. At $pastJob we had a pretty robust regression suite that'd regularly run into breaking changes in their test environment (against _old_ versioned api endpoints!). We seriously questioned whether or not anyone else used them since they were never aware until we submitted tickets. This also falls apart as soon as you use any of their "special" features that require an account rep to enable, they just don't work in the sandbox. Another pain point is that the test payment methods offered are static. You can't setup cards for specific scenarios, e.g. tokenize, successful payment, then it expires -- you can only test an expired card as a one-off.

whstl 2y ago
I just ran a couple hours ago into such a bug with Adyen, a couple payment methods have completely different behaviours in test vs production, and we ended up having to test in prod anyway.

netdevnet 2y ago

Nooope! Services like stripe let you test the payment workflow in their test environment which works exactly like the production one but without the payments going outside Stripe. Let us not normalise bad practices

serial_dev 2y ago

I've worked on adding Google Pay / Apple Pay to the mobile app of a large European ecommerce company, and that's more of less how we went about it. Start with the sandbox / test environments, once you get reasonable responses end to end, release the thing behind a feature flag. Backend moves slowly, so you add stuff to the mobile app that really belongs to the backend, but f it, because it's the only way you meet the deadline, you will convince someone on the backend later (aka never) that it's backend responsibility. Ask (pressure) the developers into buying some cheap stuff from your shop with their own credit cards, because the company is a behemoth and approving real credit cards for testing would just take ages and you want to release yesterday. It annoys you, but you realize that 5x5 euros is worth getting this done rather than start fighting a losing battle against company processes. Cancelation is possible, but it will take a couple of days. If there's any issue, you debug it across a bunch of teams and/or companies. Things start to work most of the time, time to release the stuff to x percent of your users. Check analytics and error logs frequently. Some production users got their payments through, increase rollout percentage. You discover more and more undocumented error codes, you improve the error messages to the users so that they don't retry 10 times with a card without sufficient funds. After a couple weeks, things start to stabilize, you move on to a new feature... The test environments were so complicated and had so many caveats that whenever I had to do something, I had to re-read the docs and our notes to know all the "traps" we already discovered. For the people who didn't work on this payment feature from start to finish, testing in the officially recommended test channels were hopeless.

CapstanRoller 2y ago
>Ask (pressure) the developers into buying some cheap stuff from your shop with their own credit cards This is illegal. I've always refused such "requests" and asked for a company expense card.
- ceejayoz 2y ago
  Illegal in what jurisdiction?
  - 2y ago

singleshot_ 2y ago

I seem to remember about 4111 1111 1111 1111 times that I tested a payment system with a card that wasn’t real, although I acknowledge that when I was done convincing myself that I was a good programmer, I would almost always be disavowed of this notion after using a real number.

hot_gril 2y ago

This is also true for cryptocurrency. There's always a testnet, you use it whenever possible cause xact fees are high, but it never works the same as real.

hoffs 2y ago

Doing penny testing yourself is different from letting a chunk of your user base test it

ComputerGuru 2y ago

Stripe considers using a real card to test a zero-tolerance fireable offence. I, as the name on record for an organization with Stripe, made an actual, legitimate payment to said organization (and did not refund it). Stripe's automated system caught the payment and terminated my account automatically by the time I woke up the next morning; fortunately I was able to reach a real human and explain that in addition to working for the non-profit, I was also donating to it, and they begrudgingly restored the account.

loremaster 2y ago
Stripe will take pretty much any excuse to terminate an account and pocket the money. At this point it might as well be part of their business model.
aceofspades19 2y ago
My company has hundreds/thousands of ecommerce clients, of which a huge portion use Stripe and we do real card testing every time we deploy code that could affect payments and its never been an issue.
- ComputerGuru 2y ago
  It’s pretty clearly stipulated in their TOS, for what it’s worth.
  - aceofspades19 2y ago
    I don't see anything referring to testing in https://stripe.com/en-ca/legal/ssa https://stripe.com/en-ca/legal/ssa Other than that you are provided test keys and live keys.
pheatherlite 2y ago
But why? Why testing with real payment methods is a bad idea for stripe?

firesteelrain 2y ago

PA DSS/PCI DSS standards say you aren’t supposed to test with real cards. It’s a certification criteria. I had to make my own PCI DSS (Payment Card Industry Data Security Standard) Requirement 6.4.3 of the PCI DSS states: "Production data (live PANs) are not used for testing or development." This requirement is aimed at ensuring that real cardholder data (Primary Account Numbers or PANs) is not used in non-production environments, such as testing or development environments, to minimize the risk of exposure and unauthorized access. PA DSS (Payment Application Data Security Standard) Requirement 6.3.4 of the PA DSS states: "Production data (real PANs, track data, or other real cardholder data) is not used for testing or development."

sebastiennight 2y ago

I built software for the self-service automats of a very large European rail company, and I remember spending whole afternoons testing every possible scenario listed in our testing book. Those required manually purchasing tickets to many different cities, using the credit card terminal on the machine. They even had fake Discover, VISA, Mastercards you were to use to make those test purchases (to this day I don't know who came up with those fake cards and how that part of the system worked). But no, not everybody tests payment in production. It was a while ago, but I don't think they'd have switched the entire philosophy by that much since then.

TeeWEE 2y ago

Testing against the test environment works well for us. Even for terminal in person payments. Even makes it much easier to simulate edge cases.