Well, it is complicated. There is a recognized right to travel as implicit, not in the Bill of Rights, but in the structure of the Constitution. The idea is t

Well, it is complicated. There is a recognized right to travel as implicit, not in the Bill of Rights, but in the structure of the Constitution. The idea is that we are all citizens of the United States and we have a right to travel about the United States. See Crandall v. Nevada. That poses some very interesting questions. If there is no right to a specific mode of travel, but some places can only be reached effectively by plane (say Hawaii) then does placing an American on a no-fly list impact that right to go to Hawaii? Does it matter if that individual is a resident of Hawaii and is thus effectively barred from interstate travel by this? Internationally things get more complicated. I am not aware of any cases which hold squarely a right for Americans to travel about the world as they please. Certainly bans on spending money travelling to Cuba have been upheld. However what the government clearly cannot do is to take a protected criteria like speech, religion, or political viewpoint and use that as a basis for travel. If an individual were to say that policy was to dissuade Muslims from travelling, the harm would be that Muslims would get together and sue the government in order to determine the scope of this misbehavior and put a stop to it. There really isn't that much harm when compared to the likely other uses of such a power (silencing discussions, for example, of what Islam means for politics in the US). On the whole disadvantaged groups are protected by bans on hate speech, given what the government has done in the past.