As zzleeper says in practice it can be quite difficult even if possible in theory. If there are other faculty that are in your field they may not want to offend

As zzleeper says in practice it can be quite difficult even if possible in theory. If there are other faculty that are in your field they may not want to offend a colleague who they'll be dealing with forever for a PhD student who will be leaving in a couple years.