The OSI model accounts for that in layer 4. However, the IP based protocols didn't implement that piece, but let the protocols (TCP/UDP etc.) just use the laye

The OSI model accounts for that in layer 4. However, the IP based protocols didn't implement that piece, but let the protocols (TCP/UDP etc.) just use the layer 3 addresses instead of having their own layer 4 addresses. I'll try to find some references, but back when TCP was conceived, the idea was to have layer 4 addresses too, but this was dismissed for simplicity.