Now that I think about it, my (white) ancestors have been enslaved by people with darker skin color, and they never enslaved people with a darker skin color. Do

Now that I think about it, my (white) ancestors have been enslaved by people with darker skin color, and they never enslaved people with a darker skin color. Does that mean I can call black people racist if they point out my skin color?