I think you're hitting on the biggest contributor to this feeling of it being one city. In pretty much every city on the planet you have the different districts

I think you're hitting on the biggest contributor to this feeling of it being one city. In pretty much every city on the planet you have the different districts that are known for different things, predominantly the wealth of that area, but New York is somewhat unique in how intertwined and connected all of those distinct districts are in that city. As others point out, the same trains all go through the same areas, and everybody rides them. It's hard to develop the "little people" attitude when you spend so much time around said little people.