The speed difference isn't in not phone vs. credit card, it's credit card vs. stored value card. Stored value cards are optimized for speed and cost, and as a

The speed difference isn't in not phone vs. credit card, it's credit card vs. stored value card. Stored value cards are optimized for speed and cost, and as a result have shared symmetric keys available in both card and reader. That enables extremely fast transactions. EMV cards are (also) optimized (at least in the offline case, which transit gates use – the latency to talk to the bank backend would be too high) for security. The security model doesn't trust readers to keep keys secret, which means you need asymmetric cryptography for card authentication. The specs are old, which means RSA – which is very slow to run in cheap ICs embedded in these cards produced at scale.