And what do you do when two people want to go to youtube at the same time? since the packets coming back could be to the same port from the same remote host yo

And what do you do when two people want to go to youtube at the same time? since the packets coming back could be to the same port from the same remote host you can't say which is which. You might be able to get around this by using the tcp window field but the more you have to examine the packets to do this the more expensive it becomes to route packets. And even then you aren't sure that you won't have the same things happening with the same window parameter. there's all kinds of complicated multiplexing and routing you can do but eventually it just becomes cheaper to switch to ipv6 and not have to deal with these issues at all.