Is it racist to be more or less attracted to certain physical attributes including skin colour? I'm gay but I don't think that makes me sexist, I prefer certain

Is it racist to be more or less attracted to certain physical attributes including skin colour? I'm gay but I don't think that makes me sexist, I prefer certain colours of hair but I don't think that makes me.. whatever the bad thing there is... personally I've rarely been physically attracted to anybody black, does "never having dated a black person" equal "treating another person worse because of their race"? (I do agree that everyone is capable of racism even if they aren't going round lynching people, just not that racial preference when it comes to sex is always about racism.)