7 ms·

Typical implementations of payment processing and related security issues

211 points by solarist 4y ago

cersa8 4y ago
I have a much different experience. I use Mollie, probably very similar to Stripe or any other modern payment service provider. I have the occasional failed SEPA transaction due to an invalid reference id but otherwise this is very smooth sailing. There is almost zero friction in the payment process. I might be a small fish with say 40 transactions a day but I guess my situation is not much different from bigger players. Maybe this is because most transactions are in the Euro zone and things go south fast when dealing with international payments.
anonymousisme 4y ago
Should also include the political/policy problems related to processors preventing transactions for "unpopular" product/service providers.
ravirajx7 4y ago
I think this is absolutely correct time to ask something to the people who have worked on this kinda integration. Currently we have a PSP which gives a webhook on successful payment. The challenge here is once I receive webhook (which is a stateless POST request to one of our server) we want to attach it to a browser with an open websocket connection. I have to make a redirect now what do you think is an ideal way to make it? Currently I am posting it directly to the notify page which is waiting for success/failure response. But the problem here is I don't have any mechanism through which I could make a post request directly with the browser context as well as redirect without explicit post(which ofcourse displays payload - think success/fail response). Can you provide a better way for handling the redirect here with form post? To make things more clear we have our client which wants to display a QR hosted on a different service where it comes via http redirect on requesting one of the server API - then this service gets the webhook & it has to redirect back to the client again with the payment status/webhook response.
- btilly 4y ago
  Set up a Redis pub-sub: https://redis.io/docs/manual/pubsub/ https://redis.io/docs/manual/pubsub/ The code for the websocket subscribes to Redis. The webhook will publish to Redis. The sequence of actions is now that you send the client to the PSP, which will hit the webhook, that publishes to Redis, that the server side of the websocket is receiving, that realizes it should send the message to the client, that receives the message from the websocket, that then tells the client to redirect.
  - Mavvie 4y ago
    I was about to talk about issues with scaling Redis pubsub but apparently they just recently published a newer version (7) with proper sharding support in cluster mode!
    - btilly 4y ago
      You do have to be aware of Redis' limits. But for most sites, "people paying at the moment" is well within them. And Redis is the easiest solution to implement from scratch. That said, if you're deployed in a cloud, look up cloud specific messaging services. Specifically AWS SNS, Google PubSub and Azure Web PubSub. But the pattern remains the same. The webhook publishes to the PubSub which finds the other end of the websocket which sends information to the client that does something with it.
- Nextgrid 4y ago
  I think my solution in another comment here would work for you: https://news.ycombinator.com/item?id=32138881 https://news.ycombinator.com/item?id=32138881
waiwai933 4y ago
What's the right way to make sure you've implemented a payment service integration correctly? Do payment processors provide services (or checklists to go down) to make sure you haven't made any of the common mistakes? Or is the answer a professional pentest? In an ideal world I feel like you'd want two layers of defense in case one team miss something...
rafaelturk 4y ago
I'm impressed how Brazil Pix solved much of those problems. IMO is currently the best payment method in the World. Mainly because it broke apart from Visa/Mastercard rules and implemented a new protocol from scratch. http://openpix.com.br/ http://openpix.com.br/
- birracerveza 4y ago
  Situation: there are 14 competing standards https://xkcd.com/927/ https://xkcd.com/927/
- kyrra 4y ago
  The website you link is not the Pix specification put out by the Brazilian government. Rather that is a payment integrator building services on top of the government spec. Their specs are published here: https://openbankingbrasil.atlassian.net/wiki/spaces/OB/overview?homepageId=262146 https://openbankingbrasil.atlassian.net/wiki/spaces/OB/overv..., with the main landing page for the program is here: https://www.bcb.gov.br/en/financialstability/pix_en https://www.bcb.gov.br/en/financialstability/pix_en
- dustypotato 4y ago
  Also see UPI in India. Very convenient and even street side vendors to Tech giants like Google accept payments through it
  - Tijdreiziger 4y ago
    Most EU countries have something like this (iDEAL, Bancontact, Swish). Works great but breaks down when you need to do a cross-border payment (e.g. you are German and want to order from a Dutch shop).
_vvhw 4y ago
I had the experience of being part of a team in 2020 doing performance and safety analysis on Mojaloop [1], the open-source payments switch. The emphasis was mostly on identifying performance bottlenecks. For example, graphing waterfalls of database queries, estimating expected vs actual concurrency, digging into latency spikes, surprising timeout interactions, and missed group commit opportunities. However, on the safety front, one of the most challenging issues was guaranteeing the rollback of funds in the event of failure as part of the two-phase commit protocol for moving money—coordinating this across multiple SQL queries, database transactions, Kafka queues, even multiple code repositories, especially as different systems experience clock drift or as disks or machines fail. You want to ensure that the money either moves, or doesn't move, that it doesn't get lost somewhere in between. Yet most payment systems re-implement all this business logic on an ad hoc basis, again and again. We therefore extracted these primitives from Mojaloop “once and for all” to create TigerBeetle [2], an open-source financial accounting database, that provides multi-AZ replication, automated leader election, and two-phase payments out-of-the-box. Moving this business logic to the financial database layer opened up three orders of magnitude more performance, with TigerBeetle able to process a million transactions a second, all with an extremely high safety standard. For example, we do FoundationDB-style deterministic simulation testing, but with automated storage fault injection, such as 30% corruption on all replicas including the leader. Design docs, and links to talks are all in the GitHub repo [2]. Happy to answer questions from our experience, or chat if you're working on similar systems! [1] https://mojaloop.io https://mojaloop.io [2] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle (Zig)
- jasfi 4y ago
  > Moving this business logic to the financial database layer opened up three orders of magnitude more performance This is because there is an IO cost in sending a request to the DB and then receiving a response. If the DB is on another machine the cost is even higher. Do this enough times and the wait times add up a lot. By shifting business logic to stored procedures you avoid this. That's also why SQLite is very fast, as it runs in your application's memory as a library. But then your data is tied to the same limitations as the machine the application is on.
kyrra 4y ago
Googler opinions are my own. As someone who has been integrating with payment processors for the past 7 years at Google, this is a great article. Many of the things in here are things we test when we integrate with processors. This post spends a lot of time on redirect style payments, where the user lands on some other website to enter credentials or confirm payment. This form of payment is fraught with corner cases, because the merchant and the processor are using the user's browser as the transport mechanism (ie: these are can MITM part of this flow). If you are not a super extra careful, there are so many ways to attack it. If the payloads aren't signed or encrypted, you have to worry about users messing with the content. If they are signed/encrypted, you still have to worry about replay attacks. (I've spent a lot of time thinking about redirects the last 3 years, especially with writing of our redirect spec: https://developers.google.com/pay/redirect-fop-v1?hl=en https://developers.google.com/pay/redirect-fop-v1?hl=en )
- Nextgrid 4y ago
  Ideally you shouldn't be using the browser as a transport mechanism. The browser should only be a means of data entry - redirect the user to the payment processor so they can enter their details, but then the redirect back should just go to a "payment status" page while your system on the backend waits for a notification (webhook, etc) from the processor in the background which would then update the record on your side (at which point the read-only status page will pick it up on its next poll and then redirect the user to the next step of whatever process they were doing). This also allows you to recover the flow should the user drop off after they've paid - you can email them a link to that "payment status" page so they can "rescue" the flow without having to start from scratch.
  - afiori 4y ago
    We should essentially use the Oauth PKCE flow everywhere it is applicable
- pbreit 4y ago
  What if you rely on lookups or webhooks to verify payment details?
Linda703 4y ago
[dead]
hardwaresofton 4y ago
Does anyone else find it weird that this site absolutely fails to mention Stripe? The word is not in the article, and the logo is not included. Not alleging anything here but a bit curious -- I can't imagine having a discussion of payment providers without mentioning Stripe. I wouldn't be surprised, but are they just about as close to perfection as it gets on the fraud/security front?
- bigtones 4y ago
  Yeah I had just one thought after skimming it, use Stripe.
- nibbleshifter 4y ago
  Article mostly covers payment methods used in CIS (Russia, etc). Stripe isn't available in those countries.
- hollowturtle 4y ago
  Noticed the same thing and went back to comments just to check if I was alone or not :) Pretty odd for sure…
- Tomdarkness 4y ago
  Yeah, on the list on the article the only "modern major payment service" I actually recognised was PayPal and Alipay. I noticed that Rubles are mentioned multiple times in the article, perhaps the author is Russian. Is Stripe even available in Russia? However saying that I thought PayPal had suspended operations in Russia.
  - Tijdreiziger 4y ago
    > I noticed that Rubles are mentioned multiple times in the article, perhaps the author is Russian. It looks like the article is available in both Russian and English (language selector in the top right).
  - nibbleshifter 4y ago
    Stripe doesn't operate in Russia.
mritchie712 4y ago
The spaghetti of systems involved in this post alone is why I think crypto will ultimately work. This[0] is a long thread, but it makes a strong case for payments in crypto. Not sure how you beat a cheap, open source, global payments system as Visa, a bank or other competitor. 0 - https://twitter.com/SBF_FTX/status/1548292844640616451 https://twitter.com/SBF_FTX/status/1548292844640616451
- birracerveza 4y ago
  Pretty much. There won't be an anarchist wild west scenario as the maxis hope, but crypto will inevitably take over legacy systems due to how unmaintainable these have become. The anti-crypto crowd tends to forget that even though the vast majority of blockchain transactions can be attributed to scams, rugpulls or whatever else, the fact remains that an insane amount of value, bogus or not, has been transacted with little to no issues and downtime on the most popular blockchains (except Solana, that is). Once regulation is in place you'll have a technology that is vastly superior to the duct-taped monster we have today.
- XorNot 4y ago
  That thread makes crypto look better by just plain ignoring any problems. It ignores the existence of exchanges for crypto, or that exchange rate fluctuates widely, assumes transactions fees in crypto are zero, assumes exchanges don't charge fees. It ignores that in the event of any problem in the crypto process there is no refund or unwind possible. It assumes crypto payments are instant, which they are not, and definitely not at scale. It also manages to confuse "credit card" with "debit card" because you can't get "surprise batch transactions" with a credit card, that's the whole point of them. And finally its incredibly US centric, because if it weren't it might have to acknowledge that most of those problems are due to the US banking system, and other civilized countries solve them.
  - zinekeller 4y ago
    > And finally its incredibly US centric, because if it weren't it might have to acknowledge that most of those problems are due to the US banking system, and other civilized countries solve them. US banking is probably the most frustrating experience, at least in developed world (to the point that some developing countries have better systems). Probably explains why PayPal and Venmo (which are private enterprises) are popular for consumers while businesses are still relying on checks.

211 points by solarist 4y ago

I have a much different experience. I use Mollie, probably very similar to Stripe or any other modern payment service provider. I have the occasional failed SEPA transaction due to an invalid reference id but otherwise this is very smooth sailing. There is almost zero friction in the payment process. I might be a small fish with say 40 transactions a day but I guess my situation is not much different from bigger players. Maybe this is because most transactions are in the Euro zone and things go south fast when dealing with international payments.

anonymousisme 4y ago

Should also include the political/policy problems related to processors preventing transactions for "unpopular" product/service providers.

ravirajx7 4y ago

I think this is absolutely correct time to ask something to the people who have worked on this kinda integration. Currently we have a PSP which gives a webhook on successful payment. The challenge here is once I receive webhook (which is a stateless POST request to one of our server) we want to attach it to a browser with an open websocket connection. I have to make a redirect now what do you think is an ideal way to make it? Currently I am posting it directly to the notify page which is waiting for success/failure response. But the problem here is I don't have any mechanism through which I could make a post request directly with the browser context as well as redirect without explicit post(which ofcourse displays payload - think success/fail response). Can you provide a better way for handling the redirect here with form post? To make things more clear we have our client which wants to display a QR hosted on a different service where it comes via http redirect on requesting one of the server API - then this service gets the webhook & it has to redirect back to the client again with the payment status/webhook response.

btilly 4y ago
Set up a Redis pub-sub: https://redis.io/docs/manual/pubsub/ https://redis.io/docs/manual/pubsub/ The code for the websocket subscribes to Redis. The webhook will publish to Redis. The sequence of actions is now that you send the client to the PSP, which will hit the webhook, that publishes to Redis, that the server side of the websocket is receiving, that realizes it should send the message to the client, that receives the message from the websocket, that then tells the client to redirect.
- Mavvie 4y ago
  I was about to talk about issues with scaling Redis pubsub but apparently they just recently published a newer version (7) with proper sharding support in cluster mode!
  - btilly 4y ago
    You do have to be aware of Redis' limits. But for most sites, "people paying at the moment" is well within them. And Redis is the easiest solution to implement from scratch. That said, if you're deployed in a cloud, look up cloud specific messaging services. Specifically AWS SNS, Google PubSub and Azure Web PubSub. But the pattern remains the same. The webhook publishes to the PubSub which finds the other end of the websocket which sends information to the client that does something with it.
Nextgrid 4y ago
I think my solution in another comment here would work for you: https://news.ycombinator.com/item?id=32138881 https://news.ycombinator.com/item?id=32138881

waiwai933 4y ago

What's the right way to make sure you've implemented a payment service integration correctly? Do payment processors provide services (or checklists to go down) to make sure you haven't made any of the common mistakes? Or is the answer a professional pentest? In an ideal world I feel like you'd want two layers of defense in case one team miss something...

rafaelturk 4y ago

I'm impressed how Brazil Pix solved much of those problems. IMO is currently the best payment method in the World. Mainly because it broke apart from Visa/Mastercard rules and implemented a new protocol from scratch. http://openpix.com.br/ http://openpix.com.br/

birracerveza 4y ago
Situation: there are 14 competing standards https://xkcd.com/927/ https://xkcd.com/927/
kyrra 4y ago
The website you link is not the Pix specification put out by the Brazilian government. Rather that is a payment integrator building services on top of the government spec. Their specs are published here: https://openbankingbrasil.atlassian.net/wiki/spaces/OB/overview?homepageId=262146 https://openbankingbrasil.atlassian.net/wiki/spaces/OB/overv..., with the main landing page for the program is here: https://www.bcb.gov.br/en/financialstability/pix_en https://www.bcb.gov.br/en/financialstability/pix_en
dustypotato 4y ago
Also see UPI in India. Very convenient and even street side vendors to Tech giants like Google accept payments through it
- Tijdreiziger 4y ago
  Most EU countries have something like this (iDEAL, Bancontact, Swish). Works great but breaks down when you need to do a cross-border payment (e.g. you are German and want to order from a Dutch shop).

_vvhw 4y ago

I had the experience of being part of a team in 2020 doing performance and safety analysis on Mojaloop [1], the open-source payments switch. The emphasis was mostly on identifying performance bottlenecks. For example, graphing waterfalls of database queries, estimating expected vs actual concurrency, digging into latency spikes, surprising timeout interactions, and missed group commit opportunities. However, on the safety front, one of the most challenging issues was guaranteeing the rollback of funds in the event of failure as part of the two-phase commit protocol for moving money—coordinating this across multiple SQL queries, database transactions, Kafka queues, even multiple code repositories, especially as different systems experience clock drift or as disks or machines fail. You want to ensure that the money either moves, or doesn't move, that it doesn't get lost somewhere in between. Yet most payment systems re-implement all this business logic on an ad hoc basis, again and again. We therefore extracted these primitives from Mojaloop “once and for all” to create TigerBeetle [2], an open-source financial accounting database, that provides multi-AZ replication, automated leader election, and two-phase payments out-of-the-box. Moving this business logic to the financial database layer opened up three orders of magnitude more performance, with TigerBeetle able to process a million transactions a second, all with an extremely high safety standard. For example, we do FoundationDB-style deterministic simulation testing, but with automated storage fault injection, such as 30% corruption on all replicas including the leader. Design docs, and links to talks are all in the GitHub repo [2]. Happy to answer questions from our experience, or chat if you're working on similar systems! [1] https://mojaloop.io https://mojaloop.io [2] https://github.com/coilhq/tigerbeetle https://github.com/coilhq/tigerbeetle (Zig)

jasfi 4y ago
> Moving this business logic to the financial database layer opened up three orders of magnitude more performance This is because there is an IO cost in sending a request to the DB and then receiving a response. If the DB is on another machine the cost is even higher. Do this enough times and the wait times add up a lot. By shifting business logic to stored procedures you avoid this. That's also why SQLite is very fast, as it runs in your application's memory as a library. But then your data is tied to the same limitations as the machine the application is on.

kyrra 4y ago

Googler opinions are my own. As someone who has been integrating with payment processors for the past 7 years at Google, this is a great article. Many of the things in here are things we test when we integrate with processors. This post spends a lot of time on redirect style payments, where the user lands on some other website to enter credentials or confirm payment. This form of payment is fraught with corner cases, because the merchant and the processor are using the user's browser as the transport mechanism (ie: these are can MITM part of this flow). If you are not a super extra careful, there are so many ways to attack it. If the payloads aren't signed or encrypted, you have to worry about users messing with the content. If they are signed/encrypted, you still have to worry about replay attacks. (I've spent a lot of time thinking about redirects the last 3 years, especially with writing of our redirect spec: https://developers.google.com/pay/redirect-fop-v1?hl=en https://developers.google.com/pay/redirect-fop-v1?hl=en )

Nextgrid 4y ago
Ideally you shouldn't be using the browser as a transport mechanism. The browser should only be a means of data entry - redirect the user to the payment processor so they can enter their details, but then the redirect back should just go to a "payment status" page while your system on the backend waits for a notification (webhook, etc) from the processor in the background which would then update the record on your side (at which point the read-only status page will pick it up on its next poll and then redirect the user to the next step of whatever process they were doing). This also allows you to recover the flow should the user drop off after they've paid - you can email them a link to that "payment status" page so they can "rescue" the flow without having to start from scratch.
- afiori 4y ago
  We should essentially use the Oauth PKCE flow everywhere it is applicable
pbreit 4y ago
What if you rely on lookups or webhooks to verify payment details?

Linda703 4y ago

[dead]

hardwaresofton 4y ago

Does anyone else find it weird that this site absolutely fails to mention Stripe? The word is not in the article, and the logo is not included. Not alleging anything here but a bit curious -- I can't imagine having a discussion of payment providers without mentioning Stripe. I wouldn't be surprised, but are they just about as close to perfection as it gets on the fraud/security front?

bigtones 4y ago
Yeah I had just one thought after skimming it, use Stripe.
nibbleshifter 4y ago
Article mostly covers payment methods used in CIS (Russia, etc). Stripe isn't available in those countries.
hollowturtle 4y ago
Noticed the same thing and went back to comments just to check if I was alone or not :) Pretty odd for sure…
Tomdarkness 4y ago
Yeah, on the list on the article the only "modern major payment service" I actually recognised was PayPal and Alipay. I noticed that Rubles are mentioned multiple times in the article, perhaps the author is Russian. Is Stripe even available in Russia? However saying that I thought PayPal had suspended operations in Russia.
- Tijdreiziger 4y ago
  > I noticed that Rubles are mentioned multiple times in the article, perhaps the author is Russian. It looks like the article is available in both Russian and English (language selector in the top right).
- nibbleshifter 4y ago
  Stripe doesn't operate in Russia.

mritchie712 4y ago

The spaghetti of systems involved in this post alone is why I think crypto will ultimately work. This[0] is a long thread, but it makes a strong case for payments in crypto. Not sure how you beat a cheap, open source, global payments system as Visa, a bank or other competitor. 0 - https://twitter.com/SBF_FTX/status/1548292844640616451 https://twitter.com/SBF_FTX/status/1548292844640616451

birracerveza 4y ago
Pretty much. There won't be an anarchist wild west scenario as the maxis hope, but crypto will inevitably take over legacy systems due to how unmaintainable these have become. The anti-crypto crowd tends to forget that even though the vast majority of blockchain transactions can be attributed to scams, rugpulls or whatever else, the fact remains that an insane amount of value, bogus or not, has been transacted with little to no issues and downtime on the most popular blockchains (except Solana, that is). Once regulation is in place you'll have a technology that is vastly superior to the duct-taped monster we have today.
XorNot 4y ago
That thread makes crypto look better by just plain ignoring any problems. It ignores the existence of exchanges for crypto, or that exchange rate fluctuates widely, assumes transactions fees in crypto are zero, assumes exchanges don't charge fees. It ignores that in the event of any problem in the crypto process there is no refund or unwind possible. It assumes crypto payments are instant, which they are not, and definitely not at scale. It also manages to confuse "credit card" with "debit card" because you can't get "surprise batch transactions" with a credit card, that's the whole point of them. And finally its incredibly US centric, because if it weren't it might have to acknowledge that most of those problems are due to the US banking system, and other civilized countries solve them.
- zinekeller 4y ago
  > And finally its incredibly US centric, because if it weren't it might have to acknowledge that most of those problems are due to the US banking system, and other civilized countries solve them. US banking is probably the most frustrating experience, at least in developed world (to the point that some developing countries have better systems). Probably explains why PayPal and Venmo (which are private enterprises) are popular for consumers while businesses are still relying on checks.