Of course it can be the basis of prejudice. The only difference between “I will not kill except in self defense no matter how angry I am” and “I will not serve

Of course it can be the basis of prejudice. The only difference between “I will not kill except in self defense no matter how angry I am” and “I will not serve any black people no matter how desperate for cash I am” is the goodness of the choice being made. It’s a tool. Like any tool, you can use it for good or for evil.