Simple question, why is it not possible to have unrestricted immigration between nations period? Please do not give an answer that requires respecting privilege

Simple question, why is it not possible to have unrestricted immigration between nations period? Please do not give an answer that requires respecting privilege of native born inhabitants; by doing this you are implicitly stating that these people are more deserving than someone coming from some other place by right of birth. You may as well agitate for the divine right of kings and the reinstatement of the monarchy if you view this as an acceptable path to tread. Also, it seems to me that actually making immigration from and to anywhere on the planet would pretty much force governments to actually compete with each other, a country in which nobody wants to live is not a country that will stick around for long if everyone is free to leave at their choosing. Wouldn't this be a net positive, really?