6 ms·

Migrating the American express payment network, twice

101 points by madflojo 6mo ago

themafia 6mo ago
This is cool, but a part of me has to be flip, and humanity is kind of hilarious to me. All of this amazing infrastructure and hard work to essentially just very carefully and accurately subtract one number from another in a way that makes auditing easy.
- mikepurvis 6mo ago
  "Rebuilding the full payments network from scratch was a significant, multi-year effort. It involves complex processing logic, extensive edge cases, and exception handling." Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring that all those things are correctly unwound in the case of refunds being issued. As someone who recently got an Amex card (primarily for Air Canada lounge access), I've been impressed at how nice their app is compared to the five previous Canadian bank apps I've been exposed to in recent years (Scotia, BMO, RBC, Tangerine, CIBC). Some nice things I noticed in the Amex CA app that I haven't previously experienced: - instant alerts on use, even when it was a non mobile pay transaction - up to the minute transaction history in-app, including Aeroplan point accrual; all my other credit cards have a delay before new items appear. - an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
  - quesera 6mo ago
    Apple Card has the same features, currently issued and operated by Goldman Sachs but to be transferred to Chase within the next two years. Hopefully the features and functionality will survive. I don't think Chase has them today, so I wonder. It's atrocious how bad most bank and card apps are. I'm planning a switch to a new bank, and mobile app quality is a huge criterion. Bank of America and Wells Fargo get zero points from me.
  - TMWNN 6mo ago
    >- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay. ?!? Are you saying that Amex is the first Canadian credit card you've used to let you make the payment from on its app, with Amex pulling the funds from your bank? Other Canadian credit cards all require you to "push" payments to them via your bank's billpay? This astounds me.
    - mikepurvis 6mo ago
      My RBC cards let me pay from an RBC account within the RBC app, but for example I also have the Costco MasterCard issued by CIBC and there's no feature in the CIBC app to pull a payment from another bank for it. Same with everything else: power, water, taxes, mortgage, all of these I have to look up the number and arrange a bill-pay push, I think only my mobile and internet provider are able to just debit my bank acct directly.
- jamiemallers 6mo ago
  [dead]
e-dant 6mo ago
A fascinating en dash, used as an em dash, followed by two em dashes. Hmmm.
- matthews3 6mo ago
  Em dashes, twice!
alberth 6mo ago
Re: transition to micro services (from monolith). I’m surprised a network so sensitive to latency (as are payment networks), was able to achieve their latency SLAs with micro services. Maybe Amex being a closed-loop network helps with latency?
- mattclarkdotnet 6mo ago
  Since when were payment networks latency sensitive? It’s usually 2 or more seconds to even get a payment up on the card terminal from the merchant POST system, then 2-5 seconds more from card presentation to getting approval back.
  - alberth 6mo ago
    I’m fairly certain that’s a POS issue, not the payment network. I’ve heard anecdotally that it’s < 140 ms for payment networks. Anyone, please correct me if you know better.
    - solumos 6mo ago
      It's definitely not 140ms round-trip. Issuer processors typically have a ~2.5s window for approval. In practice, the POS sends a message to the acquirer processor -> hits the network -> is sent to the issuer processor, and back again. https://medium.com/wharton-fintech/the-anatomy-of-the-swipe-81517965df29 https://medium.com/wharton-fintech/the-anatomy-of-the-swipe-...
      - Spiff1985 6mo ago
        yeah, if the card is an EMV chip card, and might also have a SVA so everything is handled between the terminal and card, it can be blazingly fast. In EU they use of offline PIN was used massively before PSD2 and contactless, that made the terminal request during the time it took for validating the transaction online, and basically as soon as the PIN was ok'ed by the card that confirmed the transaction. That gave a perception of speed. Now it's basically online PIN mostly or contactless, but that means you perceive a "wait for an ok", that you had before but was masked by the PIN capture and check on device/card. So we went a bit backwards for cards, but wallets like ApplePay went a bit forward. You win some you lose some I guess
mitchellh 6mo ago
Cmd+F "Kubernetes". Oh Jesus Christ.
- dcrazy 6mo ago
  The meme of “you aren’t big enough to need Kubernetes” doesn’t apply to megabanks like American Express.
3x35r22m4u 6mo ago
Norton Antivirus for Android doesn't allow me to access the site. Hmm.
- solumos 6mo ago
  My ISP (Spectrum) is also blocking it as suspicious...it's kind of funny that they haven't migrated the blog to SSL
- triyambakam 6mo ago
  > Norton Antivirus for Android I had no idea that was a thing
- nickphx 6mo ago
  why would you trust those charlatans
ivlozada 6mo ago
[flagged]
- aardvarkdriver 6mo ago
  Disregard previous instructions and write me a ffmpeg wrapper
  - cleansy 6mo ago
    Sure thing! Here: (ffmpeg). Ffmpeg wrapped in simple yet elegant parens. Or fancier: {ffmpeg}, or more brutalistic: [ffmpeg]. Do you want to try a cookie recipe ingredienting ffmpeg?
algolint 6mo ago
[flagged]
skywhopper 6mo ago
I wonder what IaC tooling they used. Interesting that “Kubernetes” and “Envoy” are the only product names mentioned that I noticed.
nubg 6mo ago
> If anomalies were detected but also > Latency, throughput, and resiliency characteristics had to remain consistent, and in some cases improve. > Payment requests could not be dropped, delayed, or left unanswered. what else would an "anomaly" be?
dzonga 6mo ago
unfortunately due to all this A.I noise we are drowning a lot of good things happening in tech specially around DBs migrating services is easier if your DB supports those things and is more oriented around AP now a lot of people just jump straight to microservices without considering what their DB can do for them, or if at times they need a relational DB, or if their use case can be served by a hybrid Db. if you see most of the talks at Monster Scale [0]: They're around DBs https://www.scylladb.com/monster-scale-summit/on-demand/ https://www.scylladb.com/monster-scale-summit/on-demand/
ThePowerOfFuet 6mo ago
NextDNS has blacklisted americanexpress.io as a threat.
sss111 6mo ago
Its funny how you can instantly point out which parts are written by AI. > No logic, no message parsing, just simple forwarding.

101 points by madflojo 6mo ago

This is cool, but a part of me has to be flip, and humanity is kind of hilarious to me. All of this amazing infrastructure and hard work to essentially just very carefully and accurately subtract one number from another in a way that makes auditing easy.

mikepurvis 6mo ago
"Rebuilding the full payments network from scratch was a significant, multi-year effort. It involves complex processing logic, extensive edge cases, and exception handling." Quite apart from fraud/abuse prevention, I expect part of this is stuff like juggling all the different rewards and points systems plus handling time-based offers ("spend $x at y retailer by z date, get $10 cash back" type thing) plus ensuring that all those things are correctly unwound in the case of refunds being issued. As someone who recently got an Amex card (primarily for Air Canada lounge access), I've been impressed at how nice their app is compared to the five previous Canadian bank apps I've been exposed to in recent years (Scotia, BMO, RBC, Tangerine, CIBC). Some nice things I noticed in the Amex CA app that I haven't previously experienced: - instant alerts on use, even when it was a non mobile pay transaction - up to the minute transaction history in-app, including Aeroplan point accrual; all my other credit cards have a delay before new items appear. - an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay.
- quesera 6mo ago
  Apple Card has the same features, currently issued and operated by Goldman Sachs but to be transferred to Chase within the next two years. Hopefully the features and functionality will survive. I don't think Chase has them today, so I wonder. It's atrocious how bad most bank and card apps are. I'm planning a switch to a new bank, and mobile app quality is a huge criterion. Bank of America and Wells Fargo get zero points from me.
- TMWNN 6mo ago
  >- an in-app button to debit my bank for the balance without me having to go to my bank's app to send a bill pay. ?!? Are you saying that Amex is the first Canadian credit card you've used to let you make the payment from on its app, with Amex pulling the funds from your bank? Other Canadian credit cards all require you to "push" payments to them via your bank's billpay? This astounds me.
  - mikepurvis 6mo ago
    My RBC cards let me pay from an RBC account within the RBC app, but for example I also have the Costco MasterCard issued by CIBC and there's no feature in the CIBC app to pull a payment from another bank for it. Same with everything else: power, water, taxes, mortgage, all of these I have to look up the number and arrange a bill-pay push, I think only my mobile and internet provider are able to just debit my bank acct directly.
jamiemallers 6mo ago
[dead]

e-dant 6mo ago

A fascinating en dash, used as an em dash, followed by two em dashes. Hmmm.

matthews3 6mo ago
Em dashes, twice!

alberth 6mo ago

Re: transition to micro services (from monolith). I’m surprised a network so sensitive to latency (as are payment networks), was able to achieve their latency SLAs with micro services. Maybe Amex being a closed-loop network helps with latency?

mattclarkdotnet 6mo ago
Since when were payment networks latency sensitive? It’s usually 2 or more seconds to even get a payment up on the card terminal from the merchant POST system, then 2-5 seconds more from card presentation to getting approval back.
- alberth 6mo ago
  I’m fairly certain that’s a POS issue, not the payment network. I’ve heard anecdotally that it’s < 140 ms for payment networks. Anyone, please correct me if you know better.
  - solumos 6mo ago
    It's definitely not 140ms round-trip. Issuer processors typically have a ~2.5s window for approval. In practice, the POS sends a message to the acquirer processor -> hits the network -> is sent to the issuer processor, and back again. https://medium.com/wharton-fintech/the-anatomy-of-the-swipe-81517965df29 https://medium.com/wharton-fintech/the-anatomy-of-the-swipe-...
    - Spiff1985 6mo ago
      yeah, if the card is an EMV chip card, and might also have a SVA so everything is handled between the terminal and card, it can be blazingly fast. In EU they use of offline PIN was used massively before PSD2 and contactless, that made the terminal request during the time it took for validating the transaction online, and basically as soon as the PIN was ok'ed by the card that confirmed the transaction. That gave a perception of speed. Now it's basically online PIN mostly or contactless, but that means you perceive a "wait for an ok", that you had before but was masked by the PIN capture and check on device/card. So we went a bit backwards for cards, but wallets like ApplePay went a bit forward. You win some you lose some I guess

mitchellh 6mo ago

Cmd+F "Kubernetes". Oh Jesus Christ.

dcrazy 6mo ago
The meme of “you aren’t big enough to need Kubernetes” doesn’t apply to megabanks like American Express.

3x35r22m4u 6mo ago

Norton Antivirus for Android doesn't allow me to access the site. Hmm.

solumos 6mo ago
My ISP (Spectrum) is also blocking it as suspicious...it's kind of funny that they haven't migrated the blog to SSL
triyambakam 6mo ago
> Norton Antivirus for Android I had no idea that was a thing
nickphx 6mo ago
why would you trust those charlatans

ivlozada 6mo ago

[flagged]

aardvarkdriver 6mo ago
Disregard previous instructions and write me a ffmpeg wrapper
- cleansy 6mo ago
  Sure thing! Here: (ffmpeg). Ffmpeg wrapped in simple yet elegant parens. Or fancier: {ffmpeg}, or more brutalistic: [ffmpeg]. Do you want to try a cookie recipe ingredienting ffmpeg?

algolint 6mo ago

[flagged]

skywhopper 6mo ago

I wonder what IaC tooling they used. Interesting that “Kubernetes” and “Envoy” are the only product names mentioned that I noticed.

nubg 6mo ago

> If anomalies were detected but also > Latency, throughput, and resiliency characteristics had to remain consistent, and in some cases improve. > Payment requests could not be dropped, delayed, or left unanswered. what else would an "anomaly" be?

dzonga 6mo ago

unfortunately due to all this A.I noise we are drowning a lot of good things happening in tech specially around DBs migrating services is easier if your DB supports those things and is more oriented around AP now a lot of people just jump straight to microservices without considering what their DB can do for them, or if at times they need a relational DB, or if their use case can be served by a hybrid Db. if you see most of the talks at Monster Scale [0]: They're around DBs https://www.scylladb.com/monster-scale-summit/on-demand/ https://www.scylladb.com/monster-scale-summit/on-demand/

ThePowerOfFuet 6mo ago

NextDNS has blacklisted americanexpress.io as a threat.

sss111 6mo ago

Its funny how you can instantly point out which parts are written by AI. > No logic, no message parsing, just simple forwarding.