This is just untrue. I can say city X is "my city" because I was born there. I can say city Y is "my city" because I grew up there. I can say city Z is "my city

This is just untrue. I can say city X is "my city" because I was born there. I can say city Y is "my city" because I grew up there. I can say city Z is "my city" because my family has roots there. "My city" in no way implies only that you currently live in a city; it implies a connection to, and in this case entitlement to and right to live in, a city. This right does not belong just to the people living there now. So, the implication that he must "own" a city to call it "my city" and restrict who can come and go is also idiomatically correct.